Database health check/auditing

On Thu, Feb 15, 2018 at 9:22 PM, Tim Cross <theophilusx@gmail.com> wrote:

Hi All,

I was wondering if anyone has some pointers to
sites/repositories/resources for scripts to perform basic database
audits and health checks.

situation: I have just commenced a DBA and developer role for an
organisation with a number of Postgres databases (9.4 and 9.6
versions). There has been no dedicated DBA and a number of the databases
were setup by people with little to know Postgres or database
experience. I need to get an overview on what I'm dealing with and start
prioritising what to address first.

It has been some years since I've done any real work with Postgres. Most
of my technical work over the last 10 years has been with Oracle. I
prefer to use scripts over GUI tools like pgAdmin and suspect that there
is probably some good resources out there with existing scripts I can
use as a starting point.

Any pointers greatly appreciated.

thanks,

Tim

--
Tim Cross

*Tim, Good luck to you. I have been in your situation a few times.I have
attached the following scripts which I use as a starterto get a general
idea of database status. All of the should run on both 9.4 & 9.4, but
sometimes catalog changes may fudge things up, so you may have to tweak a
bit. I have manymore general queries, so if you have any specific need,
letme know and I'll be glad to send if I have one that fits theneed.The
names should be self descriptive as to what the dobut except for
get_trans_min_cnt.sql (Transaction per minute) none have any DDL.
database_sizes.sqlbad_idx.sqlcache_hit_ratio.sqlget_trans_min_cnt.sqlget_version_num.sqlpg_runtime.sqlpg_stat_all_indexes.sqlpg_stat_all_tables.sqltable_sizes.sqltable_stats.sqluseless_indexes2.sqlPlease
also note I have bash script versions of the same, butas you did not state
the O/S, I felt the sql was best.*--
*Melvin Davidson*
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

Hi Mel,

thanks a lot. The databases are all running on Red Hat (well OUL to be
specific). The SQL is fine and I can always wrap them in a bash script if
needed.

Having these scripts is a real help. My biggest challenge at the moment is
just turning off my Oracle habits and getting back Postgres ones! Reading
these scripts really helps drag out old forgotten stuff. Luckily, there
doesn't seem to be too much really nasty or weird. For the most part,
doesn't look like anyone has made weird configuration changes and there are
no obscure triggers doing hidden things. Most of the user defined
functions seem pretty reasonable, though some seem to be doing some pretty
inefficient SQL and unnecessary type casting etc. Privileges and roles are
a mess - seems to be a bit of the 'make everyone a super user' approach
rather than work out what is required, but I expected that. There are a
couple of databases with considerable size, but many of them are quite
small (I will likely be consolidating a number of servers as their size and
load is low and it will be easier to manage fewer servers). All in all, it
isn't as bad as it could be or as bad as I've seen before, so it shouldn't
bee too bad. Establishing some standards and change control will help.

thanks again,

tim

On 16 February 2018 at 14:11, Melvin Davidson <melvin6925@gmail.com> wrote:

On Thu, Feb 15, 2018 at 9:22 PM, Tim Cross <theophilusx@gmail.com> wrote:

Hi All,

I was wondering if anyone has some pointers to
sites/repositories/resources for scripts to perform basic database
audits and health checks.

situation: I have just commenced a DBA and developer role for an
organisation with a number of Postgres databases (9.4 and 9.6
versions). There has been no dedicated DBA and a number of the databases
were setup by people with little to know Postgres or database
experience. I need to get an overview on what I'm dealing with and start
prioritising what to address first.

It has been some years since I've done any real work with Postgres. Most
of my technical work over the last 10 years has been with Oracle. I
prefer to use scripts over GUI tools like pgAdmin and suspect that there
is probably some good resources out there with existing scripts I can
use as a starting point.

Any pointers greatly appreciated.

thanks,

Tim

--
Tim Cross

*Tim, Good luck to you. I have been in your situation a few times.I have
attached the following scripts which I use as a starterto get a general
idea of database status. All of the should run on both 9.4 & 9.4, but
sometimes catalog changes may fudge things up, so you may have to tweak a
bit. I have manymore general queries, so if you have any specific need,
letme know and I'll be glad to send if I have one that fits theneed.The
names should be self descriptive as to what the dobut except for
get_trans_min_cnt.sql (Transaction per minute) none have any DDL.
database_sizes.sqlbad_idx.sqlcache_hit_ratio.sqlget_trans_min_cnt.sqlget_version_num.sqlpg_runtime.sqlpg_stat_all_indexes.sqlpg_stat_all_tables.sqltable_sizes.sqltable_stats.sqluseless_indexes2.sqlPlease
also note I have bash script versions of the same, butas you did not state
the O/S, I felt the sql was best.*--
*Melvin Davidson*
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

--
regards,

Tim

--
Tim Cross

On Thu, Feb 15, 2018 at 11:03 PM, Tim Cross <theophilusx@gmail.com> wrote:

Hi Mel,

thanks a lot. The databases are all running on Red Hat (well OUL to be
specific). The SQL is fine and I can always wrap them in a bash script if
needed.

Having these scripts is a real help. My biggest challenge at the moment is
just turning off my Oracle habits and getting back Postgres ones! Reading
these scripts really helps drag out old forgotten stuff. Luckily, there
doesn't seem to be too much really nasty or weird. For the most part,
doesn't look like anyone has made weird configuration changes and there are
no obscure triggers doing hidden things. Most of the user defined
functions seem pretty reasonable, though some seem to be doing some pretty
inefficient SQL and unnecessary type casting etc. Privileges and roles are
a mess - seems to be a bit of the 'make everyone a super user' approach
rather than work out what is required, but I expected that. There are a
couple of databases with considerable size, but many of them are quite
small (I will likely be consolidating a number of servers as their size and
load is low and it will be easier to manage fewer servers). All in all, it
isn't as bad as it could be or as bad as I've seen before, so it shouldn't
bee too bad. Establishing some standards and change control will help.

thanks again,

tim

On 16 February 2018 at 14:11, Melvin Davidson <melvin6925@gmail.com>
wrote:

On Thu, Feb 15, 2018 at 9:22 PM, Tim Cross <theophilusx@gmail.com> wrote:

Hi All,

I was wondering if anyone has some pointers to
sites/repositories/resources for scripts to perform basic database
audits and health checks.

situation: I have just commenced a DBA and developer role for an
organisation with a number of Postgres databases (9.4 and 9.6
versions). There has been no dedicated DBA and a number of the databases
were setup by people with little to know Postgres or database
experience. I need to get an overview on what I'm dealing with and start
prioritising what to address first.

It has been some years since I've done any real work with Postgres. Most
of my technical work over the last 10 years has been with Oracle. I
prefer to use scripts over GUI tools like pgAdmin and suspect that there
is probably some good resources out there with existing scripts I can
use as a starting point.

Any pointers greatly appreciated.

thanks,

Tim

--
Tim Cross

*Tim, Good luck to you. I have been in your situation a few times.I have
attached the following scripts which I use as a starterto get a general
idea of database status. All of the should run on both 9.4 & 9.4, but
sometimes catalog changes may fudge things up, so you may have to tweak a
bit. I have manymore general queries, so if you have any specific need,
letme know and I'll be glad to send if I have one that fits theneed.The
names should be self descriptive as to what the dobut except for
get_trans_min_cnt.sql (Transaction per minute) none have any DDL.
database_sizes.sqlbad_idx.sqlcache_hit_ratio.sqlget_trans_min_cnt.sqlget_version_num.sqlpg_runtime.sqlpg_stat_all_indexes.sqlpg_stat_all_tables.sqltable_sizes.sqltable_stats.sqluseless_indexes2.sqlPlease
also note I have bash script versions of the same, butas you did not state
the O/S, I felt the sql was best.*--
*Melvin Davidson*
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

--
regards,

Tim

--
Tim Cross

Tim,

FYI, the policy in this list is to avoid top posting and bottom post
instead.

Before you do anything else, * check the postgresql.conf* for *shared_buffers,
work_mem & maintenance_work_mem* values.
Quite often the developers have no clue on how to tune a database.

Roles and privileges can always be fixed. At least you don't have to deal
with the case of where they created indexes for every column
in every table, then copied the schema for each client....yeech!

Here's a couple more you may find helpful.

current_queries.sql
triggers.sql

--
*Melvin Davidson*
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

Melvin Davidson schrieb am 16.02.2018 um 05:26:

Tim,

FYI, the policy in this list is to avoid top posting and bottom post instead.

Plus: trimming the original content, so that not the whole email thread is repeated in the quote.

Thomas

Melvin,

Thanks for posting these. I haven’t even looked at them yet and just grabbed them based on the names. :c)

On Feb 15, 2018, at 9:11 PM, Melvin Davidson <melvin6925@gmail.com<mailto:melvin6925@gmail.com>> wrote:

On Thu, Feb 15, 2018 at 9:22 PM, Tim Cross <theophilusx@gmail.com<mailto:theophilusx@gmail.com>> wrote:
Hi All,

I was wondering if anyone has some pointers to

--
Tim Cross

Tim,
Good luck to you. I have been in your situation a few times.

What about bottom and top posting. Just joking sort of. I really like to reply to questions inline myself on a lot of occasions.

Seems like a good Mail tool lets you know what’s what with top,vs bottom posting . . . .

bobb

"The doer alone learneth. “
- Friedrich Nietzsche

On Fri, Feb 16, 2018 at 10:46 AM, Basques, Bob (CI-StPaul) <
bob.basques@ci.stpaul.mn.us> wrote:

Melvin,

Thanks for posting these. I haven’t even looked at them yet and just
grabbed them based on the names. :c)

On Feb 15, 2018, at 9:11 PM, Melvin Davidson <melvin6925@gmail.com> wrote:

On Thu, Feb 15, 2018 at 9:22 PM, Tim Cross <theophilusx@gmail.com> wrote:

Hi All,

I was wondering if anyone has some pointers to

--
Tim Cross

*Tim, Good luck to you. I have been in your situation a few times. *

What about bottom *and* top posting. Just joking sort of. I really like
to reply to questions inline myself on a lot of occasions.

Seems like a good Mail tool lets you know what’s what with top,vs bottom
posting . . . .

bobb

"The doer alone learneth. “
- Friedrich Nietzsche

*Bob,> Thanks for posting these. *

*You are welcome. I have a bunch more I've written through the years, and
also bash counterparts. Let me know if you have *

*any particular need and I'll be glad to send them.*

*As for bottom or top posting, that is simply the convention of this
support list and I don't know of any email programs that *

*understand that policy. At least not until AI becomes more sophisicated.*

--
*Melvin Davidson*
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

Hi Tim,
There are good continuously running apps to monitor postgres.
IMHO, the most successful one is pgcenter.
Here is the link. https://github.com/lesovsky/pgcenter

Regards
İbrahim

On 16 Feb 2018 5:22 am, "Tim Cross" <theophilusx@gmail.com> wrote:

situation: I have just commenced a DBA and developer role for an
organisation with a number of Postgres databases (9.4 and 9.6
versions). There has been no dedicated DBA and a number of the databases
were setup by people with little to know Postgres or database
experience. I need to get an overview on what I'm dealing with and start
prioritising what to address first.

It has been some years since I've done any real work with Postgres. Most
of my technical work over the last 10 years has been with Oracle. I
prefer to use scripts over GUI tools like pgAdmin and suspect that there
is probably some good resources out there with existing scripts I can
use as a starting point.

You might want to have a look at OPM: http://opm.io/

All,

Not really an app, but we have a Apache script that logs all the activity to our Postgres/PostGIS services and inserts the log entries directly into Postgres. Works great for reporting our Postgres web traffic stuff. My dev guy was skeptical about it being effective and not binding up at some point, but it’s been running for a couple of years now with no ill effects.

I have a bunch of canned and automated reports being generated from it.

bobb

On Feb 16, 2018, at 10:35 AM, Ibrahim Edib Kokdemir <kokdemir@gmail.com<mailto:kokdemir@gmail.com>> wrote:

Hi Tim,
There are good continuously running apps to monitor postgres.
IMHO, the most successful one is pgcenter.
Here is the link. https://github.com/lesovsky/pgcenter

Regards
İbrahim

On 16 Feb 2018 5:22 am, "Tim Cross" <theophilusx@gmail.com<mailto:theophilusx@gmail.com>> wrote:
Hi All,

I was wondering if anyone has some pointers to
sites/repositories/resources for scripts to perform basic database
audits and health checks.

situation: I have just commenced a DBA and developer role for an
organisation with a number of Postgres databases (9.4 and 9.6
versions). There has been no dedicated DBA and a number of the databases
were setup by people with little to know Postgres or database
experience. I need to get an overview on what I'm dealing with and start
prioritising what to address first.

It has been some years since I've done any real work with Postgres. Most
of my technical work over the last 10 years has been with Oracle. I
prefer to use scripts over GUI tools like pgAdmin and suspect that there
is probably some good resources out there with existing scripts I can
use as a starting point.

Any pointers greatly appreciated.

thanks,

Tim

--
Tim Cross

"Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.”
- Albert Einstein

On Fri, Feb 16, 2018 at 2:40 PM, Basques, Bob (CI-StPaul) <
bob.basques@ci.stpaul.mn.us> wrote:

All,

Not really an app, but we have a Apache script that logs all the activity
to our Postgres/PostGIS services and inserts the log entries directly into
Postgres. Works great for reporting our Postgres web traffic stuff. My
dev guy was skeptical about it being effective and not binding up at some
point, but it’s been running for a couple of years now with no ill effects.

I have a bunch of canned and automated reports being generated from it.

bobb

On Feb 16, 2018, at 10:35 AM, Ibrahim Edib Kokdemir <kokdemir@gmail.com>
wrote:

Hi Tim,
There are good continuously running apps to monitor postgres.
IMHO, the most successful one is pgcenter.
Here is the link. https://github.com/lesovsky/pgcenter

Regards
İbrahim

On 16 Feb 2018 5:22 am, "Tim Cross" <theophilusx@gmail.com> wrote:

Hi All,

I was wondering if anyone has some pointers to
sites/repositories/resources for scripts to perform basic database
audits and health checks.

situation: I have just commenced a DBA and developer role for an
organisation with a number of Postgres databases (9.4 and 9.6
versions). There has been no dedicated DBA and a number of the databases
were setup by people with little to know Postgres or database
experience. I need to get an overview on what I'm dealing with and start
prioritising what to address first.

It has been some years since I've done any real work with Postgres. Most
of my technical work over the last 10 years has been with Oracle. I
prefer to use scripts over GUI tools like pgAdmin and suspect that there
is probably some good resources out there with existing scripts I can
use as a starting point.

Any pointers greatly appreciated.

thanks,

Tim

--
Tim Cross

"Only two things are infinite, the universe and human stupidity, and I'm
not sure about the former.”
- Albert Einstein

*> resources for scripts to perform basic database audits and health
checks. *

*PgBadger is free and an excellent log analyzer which is useful for
tracking query execution times and transaction
rates.https://sourceforge.net/projects/pgbadger/
<https://sourceforge.net/projects/pgbadger/&gt;*

--
*Melvin Davidson*
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

Ibrahim Edib Kokdemir <kokdemir@gmail.com> writes:

Hi Tim,
There are good continuously running apps to monitor postgres.
IMHO, the most successful one is pgcenter.
Here is the link. https://github.com/lesovsky/pgcenter

Thanks, a useful link.

Tim

--
Tim Cross

Thomas Kellerer <spam_eater@gmx.net> writes:

situation: I have just commenced a DBA and developer role for an
organisation with a number of Postgres databases (9.4 and 9.6
versions). There has been no dedicated DBA and a number of the databases
were setup by people with little to know Postgres or database
experience. I need to get an overview on what I'm dealing with and start
prioritising what to address first.

You might want to have a look at OPM: http://opm.io/

Looks interesting. Nagios is what is used here for monitoring and
although rusty, once upon a time, I use to cut quite a lot of perl, so
reasonable skill set match. Will check it out.

Although on-going monitoring of the systems is definitely required, the
first step is to just get an overall handle on the situation. There is
no documentation and configuration, maintenance etc has been patchy.

Tim

--
Tim Cross

Melvin Davidson <melvin6925@gmail.com> writes:

On Fri, Feb 16, 2018 at 10:46 AM, Basques, Bob (CI-StPaul) <
bob.basques@ci.stpaul.mn.us> wrote:

What about bottom *and* top posting. Just joking sort of. I really like
to reply to questions inline myself on a lot of occasions.

Seems like a good Mail tool lets you know what’s what with top,vs bottom
posting . . . .

*As for bottom or top posting, that is simply the convention of this
support list and I don't know of any email programs that *

*understand that policy. At least not until AI becomes more sophisicated.*

I will try to comply with the list policy/conventions. My original reply
was from within the Gmail web interface, which is OK as web based mail
goes, but it certainly 'encourages' top posting. My preferred MUA is
more 'old school' (mu4e), where dealing with such preferences is much
easier, but our wise network admins ban access internally (but you can
still access the web interface...).

Top v bottom posting issues are a blast from the past - I've not even
seen a reference to it since the old Newsgroups days! ;-)

Tim
--
Tim Cross

Thomas Kellerer <spam_eater@gmx.net> writes:

Melvin Davidson schrieb am 16.02.2018 um 05:26:

Tim,

FYI, the policy in this list is to avoid top posting and bottom post instead.

Plus: trimming the original content, so that not the whole email thread is repeated in the quote.

Thomas

While I'm happy to comply, I disagree with trimming/editing the
thread. Certainly made sense when networks were slower and MUAs were
less sophisticated, but these days I prefer using a MUA which is able to
fold the original content and keep all the content in the message. This
reduces confusion, misunderstanding and incorrect attribution arising
from poor editing and makes it easy to reference the full discussion in
one spot rather than having to go back through messages putting it all
back together. YMMV.

Tim

--
Tim Cross

What Tim said!! :c)

I think the bigger deal is setting a new message Subject at the approriate point in a thread/conversation.

bobb

On Feb 16, 2018, at 2:23 PM, Tim Cross <theophilusx@gmail.com<mailto:theophilusx@gmail.com>> wrote:

Thomas Kellerer <spam_eater@gmx.net<mailto:spam_eater@gmx.net>> writes:

Melvin Davidson schrieb am 16.02.2018 um 05:26:
Tim,

FYI, the policy in this list is to avoid top posting and bottom post instead.

Plus: trimming the original content, so that not the whole email thread is repeated in the quote.

Thomas

While I'm happy to comply, I disagree with trimming/editing the
thread. Certainly made sense when networks were slower and MUAs were
less sophisticated, but these days I prefer using a MUA which is able to
fold the original content and keep all the content in the message. This
reduces confusion, misunderstanding and incorrect attribution arising
from poor editing and makes it easy to reference the full discussion in
one spot rather than having to go back through messages putting it all
back together. YMMV.

Tim

--
Tim Cross

The third-rate mind is only happy when it is thinking with the majority. The second-rate mind is only happy when it is thinking with the minority. The first-rate mind is only happy when it is thinking.
—A. A. Milne

On Sat, 17 Feb 2018 07:23:32 +1100, Tim Cross <theophilusx@gmail.com>
wrote:

Thomas Kellerer <spam_eater@gmx.net> writes:

Plus: trimming the original content, so that not the whole email
thread is repeated in the quote.

While I'm happy to comply, I disagree with trimming/editing the
thread. Certainly made sense when networks were slower and MUAs were
less sophisticated, but these days I prefer using a MUA which is able to
fold the original content and keep all the content in the message. This
reduces confusion, misunderstanding and incorrect attribution arising
from poor editing and makes it easy to reference the full discussion in
one spot rather than having to go back through messages putting it all
back together. YMMV.

Tim

The problem is that conversations rarely remained confined to a single
topic [this thread for example 8-)].

The list servers do have message size limits (independent of any email
server limits). It isn't a problem for simple text messages, but a
lot of people now are sending MIME messages that include all the text
twice (or 3 times) plus embedded HTML+CSS formatting.

On some lists, I have seen MIME messages (mostly sent from Google
Groups users) that contain - not kidding - 5,000+ lines of hidden
HTML+CSS surrounding a few lines of comment.

I agree with you that messages should stand alone wrt to the subject
they address. It's just that, too often, messages that contain a lot
of quoted material have wandered considerably and so are mostly noise.

YMMV,
George

George Neuner <gneuner2@comcast.net> writes:

On Sat, 17 Feb 2018 07:23:32 +1100, Tim Cross <theophilusx@gmail.com>
wrote:

Thomas Kellerer <spam_eater@gmx.net> writes:

Plus: trimming the original content, so that not the whole email
thread is repeated in the quote.

While I'm happy to comply, I disagree with trimming/editing the
thread. Certainly made sense when networks were slower and MUAs were
less sophisticated, but these days I prefer using a MUA which is able to
fold the original content and keep all the content in the message. This
reduces confusion, misunderstanding and incorrect attribution arising
from poor editing and makes it easy to reference the full discussion in
one spot rather than having to go back through messages putting it all
back together. YMMV.

Tim

The problem is that conversations rarely remained confined to a single
topic [this thread for example 8-)].

The list servers do have message size limits (independent of any email
server limits). It isn't a problem for simple text messages, but a
lot of people now are sending MIME messages that include all the text
twice (or 3 times) plus embedded HTML+CSS formatting.

On some lists, I have seen MIME messages (mostly sent from Google
Groups users) that contain - not kidding - 5,000+ lines of hidden
HTML+CSS surrounding a few lines of comment.

I agree with you that messages should stand alone wrt to the subject
they address. It's just that, too often, messages that contain a lot
of quoted material have wandered considerably and so are mostly noise.

YMMV,
George

I certainly agree with the comments regarding MIME and bloody HTML
messages. Extending email to use HTML is something I've always
considered to have been a big mistake. In addition to the wasted
bandwidth, it has created far too many security problems. The whole
situation has been made worse by the appalling HTML generation of many
clients (I"m looking at you Microsoft!). Only this week yet another
exploit was discovered which uses MUA preview facilities and MIME/HTML
integration. As someone who is legally considered to be blind, I can
state categorically that email has become LESS accessible as a result of
these 'enhancements'.

However, I think the key takeaway from your comments is the need for all
of us to be a little more proactive when it comes to the subject
title. I agree 100% we should try to ensure we change the title when the
thread wanders off topic.

Tim
--
Tim Cross

George Neuner <gneuner2@comcast.net> writes:

On Sat, 17 Feb 2018 07:23:32 +1100, Tim Cross <theophilusx@gmail.com>
wrote:

Thomas Kellerer <spam_eater@gmx.net> writes:

Plus: trimming the original content, so that not the whole email
thread is repeated in the quote.

While I'm happy to comply, I disagree with trimming/editing the
thread.

The problem is that conversations rarely remained confined to a single
topic [this thread for example 8-)].
The list servers do have message size limits (independent of any email
server limits).

I don't think this rule is about any technical limits. It's about having
minimal courtesy to your readers by not forcing them to re-read stuff they
just read. There are hundreds or thousands of people reading this list;
if you take thirty seconds to save each reader a third of a second, it's
a good tradeoff. If you won't spend that time, you're being discourteous.

Personally, if I get two or three screenfuls down into a message without
seeing a single line of new content, I stop reading.

Conversely, if you figure you can avoid spending any of your precious time
in quote-clipping by top-posting a couple of lines of commentary, you're
also being discourteous to readers who may come across your post sometime
later, who won't have just seen the whole thread. They won't know what it
is you're responding to, and you shouldn't make them dig through the
entire thread history in order to figure out what your one or two
sentences are in response to.

Gmail have done their level best to destroy email in general, and mailing
lists in particular, as a useful communication medium. Don't help them
out by being a jerk about your quoting habits.

regards, tom lane