Trivial HugeTLB Benchmark
Hey,
Out of curiosity I benchmarked PostgreSQL 8.2.3 using huge pages for shared memory. Oracle claims fairly significant speedups with huge pages but I couldn't find any information on PostgreSQL.
I used the attached patch to enable huge pages on Linux. The test hardware is a dual Nocona Xeon 3.2Ghz with 4GB of RAM and two 15K 73GB Ultra320 disks in a software RAID-1. The box is running CentOS 4.4 for x86-64 and the vendor's stock 2.6.9 kernel. The relevant postgresql.conf settings are:
shared_buffers=160MB
work_mem=8MB
fsync=off
full_page_writes=off
effective_cache_size=3GB
I ran each pgbench after a fresh reboot. I used 85 huge pages reserved at boot for the huge page test, and none for the normal shared memory test.
Normal shared memory:
-bash-3.00$ pgbench -c 5 -t 10000
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
number of clients: 5
number of transactions per client: 10000
number of transactions actually processed: 50000/50000
tps = 1669.009344 (including connections establishing)
tps = 1669.941756 (excluding connections establishing)
Huge pages:
-bash-3.00$ pgbench -c 5 -t 10000
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
number of clients: 5
number of transactions per client: 10000
number of transactions actually processed: 50000/50000
tps = 1678.392138 (including connections establishing)
tps = 1679.268344 (excluding connections establishing)
Assuming that this is a representative benchmark, it looks like huge pages are a very slight (0.5%) performance win. I'm guessing that PostgreSQL doesn't benefit as much as Oracle due to its much less ridiculous shared memory size. That performance boost is almost certainly not worth the platform-specific code or administration overhead of hugetlb on Linux.
-Ryan
This electronic mail transmission and any accompanying attachments contain confidential information intended only for the use of the individual or entity named above. Any dissemination, distribution, copying or action taken in reliance on the contents of this communication by anyone other than the intended recipient is strictly prohibited. If you have received this communication in error please immediately delete the e-mail and either notify the sender at the above e-mail address or by telephone at 250.386.5323.
Attachments:
hugetlb-test.difftext/x-patch; name=hugetlb-test.diffDownload
diff -u postgresql-8.2.3/src/backend/port/sysv_shmem.c /usr/src/redhat/BUILD/postgresql-8.2.3/src/backend/port/sysv_shmem.c
--- postgresql-8.2.3/src/backend/port/sysv_shmem.c 2006-07-13 22:28:28.000000000 -0700
+++ /usr/src/redhat/BUILD/postgresql-8.2.3/src/backend/port/sysv_shmem.c 2007-03-03 17:05:37.000000000 -0800
@@ -46,6 +46,7 @@
#define PG_SHMAT_FLAGS 0
#endif
+#define PG_LARGE_PAGE_SIZE 2048 * 1024
unsigned long UsedShmemSegID = 0;
void *UsedShmemSegAddr = NULL;
@@ -75,7 +76,8 @@
IpcMemoryId shmid;
void *memAddress;
- shmid = shmget(memKey, size, IPC_CREAT | IPC_EXCL | IPCProtection);
+ size = ((size - 1) & ~(PG_LARGE_PAGE_SIZE - 1)) + PG_LARGE_PAGE_SIZE;
+ shmid = shmget(memKey, size, IPC_CREAT | IPC_EXCL | IPCProtection | SHM_HUGETLB);
if (shmid < 0)
{
Ryan Cumming <ryan.cumming@neverbluemedia.com> writes:
I ran each pgbench after a fresh reboot. I used 85 huge pages reserved at boot for the huge page test, and none for the normal shared memory test.
Normal shared memory:
-bash-3.00$ pgbench -c 5 -t 10000
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
number of clients: 5
number of transactions per client: 10000
number of transactions actually processed: 50000/50000
tps = 1669.009344 (including connections establishing)
tps = 1669.941756 (excluding connections establishing)
If you did this only once, the results are not really trustworthy;
you need to average several similar runs before you can have much
confidence. pgbench's inter-run variation is usually upwards of 10%,
so trying to draw conclusions about half-percentage-point differences
without averaging is a waste of time.
Also, if scaling factor < number of clients then what you're mostly
measuring is update-contention behavior. Try it with -s 10 and -c 5;
and don't forget to reinitialize the database for each run of tests
to be sure it's fair.
regards, tom lane
On Sun, 2007-03-04 at 10:14 -0800, Tom Lane wrote:
If you did this only once, the results are not really trustworthy;
you need to average several similar runs before you can have much
confidence. pgbench's inter-run variation is usually upwards of 10%,
so trying to draw conclusions about half-percentage-point differences
without averaging is a waste of time.
Good point, thanks
Also, if scaling factor < number of clients then what you're mostly
measuring is update-contention behavior. Try it with -s 10 and -c 5;
and don't forget to reinitialize the database for each run of tests
to be sure it's fair.
I did another 18 runs, 9 each for huge pages and normal shared memory.
The database was reinitialized before every third run with "pgbench -i
-s 10". The runs themselves were done with "pgbench -s 10 -c 5 -t 10000"
Normal shared memory:
tps = 1835.929043 (including connections establishing)
tps = 1697.455165 (including connections establishing)
tps = 1378.393001 (including connections establishing)
tps = 1834.802729 (including connections establishing)
tps = 1630.100895 (including connections establishing)
tps = 1415.504943 (including connections establishing)
tps = 1864.908838 (including connections establishing)
tps = 1726.295622 (including connections establishing)
tps = 1323.679649 (including connections establishing)
Average: 1634.19 tps
Huge pages:
tps = 1867.400381 (including connections establishing)
tps = 1715.269338 (including connections establishing)
tps = 1259.314139 (including connections establishing)
tps = 1880.803336 (including connections establishing)
tps = 1885.351404 (including connections establishing)
tps = 1603.302855 (including connections establishing)
tps = 1884.888431 (including connections establishing)
tps = 1563.452093 (including connections establishing)
tps = 1361.896887 (including connections establishing)
Average: 1669.08
That works out to approximately a 2.1% performance boost for huge pages.
It still doesn't seem very compelling but I could try to put together a
patch for inclusion if people were interested in such a thing.
-Ryan
This electronic mail transmission and any accompanying attachments contain confidential information intended only for the use of the individual or entity named above. Any dissemination, distribution, copying or action taken in reliance on the contents of this communication by anyone other than the intended recipient is strictly prohibited. If you have received this communication in error please immediately delete the e-mail and either notify the sender at the above e-mail address or by telephone at 250.386.5323.
On Mar 4, 2007, at 3:33 PM, Ryan Cumming wrote:
I did another 18 runs, 9 each for huge pages and normal shared memory.
The database was reinitialized before every third run with "pgbench -i
-s 10". The runs themselves were done with "pgbench -s 10 -c 5 -t
10000"
Rather than doing that, I think you'd be much better off just running
a very long benchmark and turning on autovaccum. That would at least
be closer to real-world usage.
--
Jim Nasby jim@nasby.net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)