Hi,
I'm trying to get decent performance with ZFS on my Blackbird system, running Ubuntu 20.04 with 128 GB RAM. So far, I failed...
I have tried to make things as simple as possible, and using a sinngle SSD (Samsung 840 Pro, 512 GB), tested with bonnie++, here is what I get :
franck@blackbird:~$ bonnie++ -d data -c 8 -m zfs-sdb -s 270000 -r 130688
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
zfs-sdb 270000M::8 85k 99 475m 89 152m 47 185k 99 499m 53 4070 115
Latency 98772us 3795us 18291us 50503us 16191us 7025us
Version 1.98 ------Sequential Create------ --------Random Create--------
zfs-sdc -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 685584420 99 +++++ +++ 1143148205 14 -1321910944 98 +++++ +++ -382954332 99
Latency 4631us 2108us 5575ms 4596us 140us 320us
1.98,1.98,zfs-sdb,8,1588801810,270000M,,8192,5,85,99,486661,89,155677,47,185,99,510981,53,4070,115,16,,,,,19138,99,+++++,+++,2146,14,19113,98,+++++,+++,14365,99,98772us,3795us,18291us,50503us,16191us,7025us,4631us,2108us,5575ms,4596us,140us,320us
Notice the huge latency (5545 ms) on sequential create / delete, and the poor sequential input block performance.
Under ext4, the same disk gives me:
franck@blackbird:~$ bonnie++ -d data -c 8 -m ext4-sdb -s 270000 -r 130688
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
ext4-sdb 270000M::8 331k 99 481m 31 169m 15 1585k 99 465m 12 12131 87
Latency 27081us 71048us 54910us 9400us 23853us 1330us
Version 1.98 ------Sequential Create------ --------Random Create--------
ext4-sdb -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 927us 566us 720us 122us 4us 44us
1.98,1.98,ext4-sdb,8,1588605414,270000M,,8192,5,331,99,492198,31,173283,15,1585,99,476038,12,12131,87,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,27081us,71048us,54910us,9400us,23853us,1330us,927us,566us,720us,122us,4us,44us
I also tested it with pgbench, and ZFS gives me 30% less TPS than ext4.
So, here is my question: should I keep fighting, or is ZFS on Linux known to be badly supported/tested on ppc64 ?
Do you have any experience regarding ZFS on Blackbird ?
Thanks in advance,
Franck
