Shared memory estimation for postgres

Do a web search on setting shared memory the hard way, and I think you'll see what you really need to do.
--
Jay

Sent from my iPad

On Nov 10, 2016, at 5:57 PM, leoaaryan <leoaaryan@gmail.com> wrote:

I am a newbie to databases and Postgres and I am trying to analyze the shared
memory being calculated and allocated by Postgres in the method
"CreateSharedMemoryAndSemaphores" for different major versions for different
postgres.conf file

My idea was to create a utility in Postgres and calll out the methods like
BufferShmemSize(), LockShmemSize() etc being used in the
CreateSharedMemoryAndSemaphores() to find the value.

Till now what I have done: I have created a new utility and am trying to
link the src/backend code to it but I'm not able to get it working
correctly.

Is there any other interest idea / way where I use a postgres.conf file to
calculate the above mentioned shared memory value ?

I have found this discussion thread from the past but haven't made much
sense to me in terms of where to look.
Old discussion thread:
http://postgresql.nabble.com/postgresql-conf-basic-analysis-tool-td1948070.html

--
View this message in context: http://postgresql.nabble.com/Shared-memory-estimation-for-postgres-tp5929868.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Hi Jay,

If you are talking about
http://evol-monkey.blogspot.com/2013/08/setting-sharedbuffers-hard-way.html
and the "pg_buffercache" extensions then yes I have gone through it.

The easiest way to find the value for the shared memory computation is to
change the logging level to DEBUG3 and start postgres DB engine and it will
give the calculated value in the log file.

I believe postgres as a DB needs to be running for any extension to run (I
may be wrong here) and I dont want to start the database for this analysis.

Please correct me if I'm wrong in my concepts or if I've not understood
anything.

-leoaaryan.

--
View this message in context: http://postgresql.nabble.com/Shared-memory-estimation-for-postgres-tp5929868p5929872.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Nov 11, 2016 at 8:40 AM, leoaaryan <leoaaryan@gmail.com> wrote:

The easiest way to find the value for the shared memory computation is to
change the logging level to DEBUG3 and start postgres DB engine and it will
give the calculated value in the log file.

I believe postgres as a DB needs to be running for any extension to run (I
may be wrong here) and I dont want to start the database for this analysis.

Please correct me if I'm wrong in my concepts or if I've not understood
anything.

Some time ago a patch has been proposed to project to a system view
the shared memory allocations currently happening on an instance:
/messages/by-id/20140504114417.GM12715@awork2.anarazel.de
This could be plugged into its own extension rather easily. Note
though that DSM segments (dynamic shared memory) do not have names
associated to them which would be helpful if you'd like to track that
as well. But perhaps you don't care much.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 11 November 2016 at 06:57, leoaaryan <leoaaryan@gmail.com> wrote:

I am a newbie to databases and Postgres and I am trying to analyze the shared
memory being calculated and allocated by Postgres in the method
"CreateSharedMemoryAndSemaphores" for different major versions for different
postgres.conf file

Note that this follows on from a number of other posts, including

http://stackoverflow.com/questions/39607940/is-it-possible-to-know-the-memory-being-allocated-by-the-method-createsharedmem

http://stackoverflow.com/questions/40433784/is-it-possible-to-call-a-postgres-internal-method-from-a-util

Please, PLEASE link to related prior discussion when you post
somewhere. It's really annoying having to look it up or wasting
people's time duplicating discussion that's already happened.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Hi Michael,

I think the method "pg_get_shmem_allocations" mentioned in the patch will
give the allocated shared memory when the postgres db server is running. I'm
trying to get the same without running the server if possible.

Please correct me if I have failed to understand the discussion thread
contents.

--
View this message in context: http://postgresql.nabble.com/Shared-memory-estimation-for-postgres-tp5929868p5929886.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Hi Craig,

Sorry for the multiple point of contact for the same question. I'll keep in
mind to attach similar corresponding links in future if any.

http://stackoverflow.com/questions/39607940/is-it-possible-to-know-the-memory-being-allocated-by-the-method-createsharedmem
and
http://stackoverflow.com/questions/40433784/is-it-possible-to-call-a-postgres-internal-method-from-a-util
are the other two posts from my side on stackoverflow to understand more
about the feasibility of the nature of task I'm trying to accomplish.

--
View this message in context: http://postgresql.nabble.com/Shared-memory-estimation-for-postgres-tp5929868p5929887.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Nov 11, 2016 at 1:26 PM, leoaaryan <leoaaryan@gmail.com> wrote:

I think the method "pg_get_shmem_allocations" mentioned in the patch will
give the allocated shared memory when the postgres db server is running. I'm
trying to get the same without running the server if possible.

That's up to "read the code and create a formula based on the system
parameter to calculate the amount" then.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Hi Michael,

Thanks for all the help and time. I have already developed a code where I
can exactly calculate the to be allocated shared memory value based on the
Postgres 9.5.4 code (i went through the code, found out the sizes and offset
of all the structures used in the memory calculation process and then use
the values from postgres.conf file to calculate the required value).

But the problem is if there is any change in the structures or anything is
newly added in the next major version, I need to look at the code again and
see what changed and then modify the hardcoded values of the structure size.
I'm trying to avoid that.

-leoaaryan

--
View this message in context: http://postgresql.nabble.com/Shared-memory-estimation-for-postgres-tp5929868p5929891.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 11 Nov. 2016 13:00, "leoaaryan" <leoaaryan@gmail.com> wrote:

Hi Michael,

Thanks for all the help and time. I have already developed a code where I
can exactly calculate the to be allocated shared memory value based on the
Postgres 9.5.4 code (i went through the code, found out the sizes and

offset

of all the structures used in the memory calculation process and then use
the values from postgres.conf file to calculate the required value).

But the problem is if there is any change in the structures or anything is
newly added in the next major version, I need to look at the code again

and

see what changed and then modify the hardcoded values of the structure

size.

I'm trying to avoid that.

Earlier I suggested adding a command line flag to the backend that, like
--version, prints the desired info and exits.

It's still most unclear to me what the underlying problem you're trying to
solve here is. Why you want this info and why you're so keen to avoid
starting a backend to find it.

On 2016-11-11 09:56:21 +0900, Michael Paquier wrote:

On Fri, Nov 11, 2016 at 8:40 AM, leoaaryan <leoaaryan@gmail.com> wrote:

The easiest way to find the value for the shared memory computation is to
change the logging level to DEBUG3 and start postgres DB engine and it will
give the calculated value in the log file.

I believe postgres as a DB needs to be running for any extension to run (I
may be wrong here) and I dont want to start the database for this analysis.

Please correct me if I'm wrong in my concepts or if I've not understood
anything.

Some time ago a patch has been proposed to project to a system view
the shared memory allocations currently happening on an instance:
/messages/by-id/20140504114417.GM12715@awork2.anarazel.de
This could be plugged into its own extension rather easily. Note
though that DSM segments (dynamic shared memory) do not have names
associated to them which would be helpful if you'd like to track that
as well. But perhaps you don't care much.

FWIW, there's a number of purposes where it'd be more useful to have the
total amount of shared memory output without starting postgres. E.g. to
compute the amount of hugepages to configure as root... I wondered if we
could have a readonly guc that's output by something like -C
total_shared_memory or such (which doesn't start a server...).

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers