Postgresql, PSN hack and table limits

On Sun, May 1, 2011 at 07:50, Mark Morgan Lloyd
<markMLl.pgsql-general@telemetry.co.uk> wrote:

Somebody is making a very specific claim that Postgres can support a limited
number of rows:

"INPS (a data forensics team) said that there is 7 main Databases all hosted
at different data centers but linked over a type of 'cloud' Each database
uses PostGRESSQL which would mean the most amount of data each database
could hold with no stability issues is aproximitely equal to that of
10,348,439 Rows" http://pastebin.com/MtX1MDdh

Does anybody have any idea where they've got hold of this figure?

PostgreSQL, of course, has no such ridiculous limits.

Whether a specific application running on top of PostgreSQL would have
a limitation like that, is of course a different question - that might
certainly be possible, even though the limit mentioned is a really
weird number.

I find it really hard to parse the text of that post to even
understand what they mean, but it's rather obviously filled with other
completely incorrect technical statements, so I wouldn't pay any
attention to this one in particular. (e.g. since when did you need
port 25 to download an email attachment? and suddenly they mention
oracle metasploits, which obviously wouldn't work on postgres)

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 04/30/11 10:50 PM, Mark Morgan Lloyd wrote:

Somebody is making a very specific claim that Postgres can support a
limited number of rows:

"INPS (a data forensics team) said that there is 7 main Databases all
hosted at different data centers but linked over a type of 'cloud'
Each database uses PostGRESSQL which would mean the most amount of
data each database could hold with no stability issues is aproximitely
equal to that of 10,348,439 Rows" http://pastebin.com/MtX1MDdh

Does anybody have any idea where they've got hold of this figure?

a pastebin is the 'source' for this rant full of random WTF ?

Magnus Hagander wrote:

On Sun, May 1, 2011 at 07:50, Mark Morgan Lloyd
<markMLl.pgsql-general@telemetry.co.uk> wrote:

Somebody is making a very specific claim that Postgres can support a limited
number of rows:

"INPS (a data forensics team) said that there is 7 main Databases all hosted
at different data centers but linked over a type of 'cloud' Each database
uses PostGRESSQL which would mean the most amount of data each database
could hold with no stability issues is aproximitely equal to that of
10,348,439 Rows" http://pastebin.com/MtX1MDdh

Does anybody have any idea where they've got hold of this figure?

PostgreSQL, of course, has no such ridiculous limits.

Whether a specific application running on top of PostgreSQL would have
a limitation like that, is of course a different question - that might
certainly be possible, even though the limit mentioned is a really
weird number.

I find it really hard to parse the text of that post to even
understand what they mean, but it's rather obviously filled with other
completely incorrect technical statements, so I wouldn't pay any
attention to this one in particular. (e.g. since when did you need
port 25 to download an email attachment? and suddenly they mention
oracle metasploits, which obviously wouldn't work on postgres)

I'm not paying attention because of the amount of rubbish in that text,
but other people are likely to if Google makes the connection between
the current unpleasantness and this project- which is why I'm avoiding
mentioning certain names. I don't know for certain who these INPS people
are or why they think they're qualified to pontificate.

The Oracle angle could be related to older SQL injection attacks (2008?)
or somebody could be assuming that a compatibility layer introduces
vulnerabilities.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]

On 2011-05-01, Mark Morgan Lloyd <markMLl.pgsql-general@telemetry.co.uk> wrote:

Somebody is making a very specific claim that Postgres can support a
limited number of rows:

"INPS (a data forensics team) said that there is 7 main Databases all
hosted at different data centers but linked over a type of 'cloud' Each
database uses PostGRESSQL which would mean the most amount of data each
database could hold with no stability issues is aproximitely equal to
that of 10,348,439 Rows" http://pastebin.com/MtX1MDdh

Does anybody have any idea where they've got hold of this figure?

the figure is within 1% of the maximun size for data stored in text
(or bytea) column. It doesn't seem close to any other PG limit that
I know of.

--
⚂⚃ 100% natural

Jasen Betts <jasen@xnet.co.nz> writes:

On 2011-05-01, Mark Morgan Lloyd <markMLl.pgsql-general@telemetry.co.uk> wrote:

Somebody is making a very specific claim that Postgres can support a
limited number of rows:

"INPS (a data forensics team) said that there is 7 main Databases all
hosted at different data centers but linked over a type of 'cloud' Each
database uses PostGRESSQL which would mean the most amount of data each
database could hold with no stability issues is aproximitely equal to
that of 10,348,439 Rows" http://pastebin.com/MtX1MDdh

Does anybody have any idea where they've got hold of this figure?

the figure is within 1% of the maximun size for data stored in text
(or bytea) column.

No it isn't; the max size per field is 1GB. Although actually
manipulating such field values will probably not work very well unless
you have a 64-bit machine, else you'll hit address-space issues.

I could believe that a specific application using specific fields in
a specific way in a 32-bit machine might start to hit "out of memory"
errors for field widths somewhere in the tens-of-MB range. But the
stated claim is about number of rows, not row width, and the exactness
and breadth of the claim is, well, ridiculous on its face.

I think INPS's level of knowledge about PG must be about as good as
their ability to spell it :-(

BTW, there *is* a hard limit of 32TB per table, arising from the limited
size of BlockNumber. But it's hard to believe that INPS's claim has
anything to do with that.

regards, tom lane

Tom Lane wrote:

Jasen Betts <jasen@xnet.co.nz> writes:

On 2011-05-01, Mark Morgan Lloyd <markMLl.pgsql-general@telemetry.co.uk> wrote:

Somebody is making a very specific claim that Postgres can support a
limited number of rows:

"INPS (a data forensics team) said that there is 7 main Databases all
hosted at different data centers but linked over a type of 'cloud' Each
database uses PostGRESSQL which would mean the most amount of data each
database could hold with no stability issues is aproximitely equal to
that of 10,348,439 Rows" http://pastebin.com/MtX1MDdh

Does anybody have any idea where they've got hold of this figure?

the figure is within 1% of the maximun size for data stored in text
(or bytea) column.

No it isn't; the max size per field is 1GB. Although actually
manipulating such field values will probably not work very well unless
you have a 64-bit machine, else you'll hit address-space issues.

I could believe that a specific application using specific fields in
a specific way in a 32-bit machine might start to hit "out of memory"
errors for field widths somewhere in the tens-of-MB range. But the
stated claim is about number of rows, not row width, and the exactness
and breadth of the claim is, well, ridiculous on its face.

I think INPS's level of knowledge about PG must be about as good as
their ability to spell it :-(

BTW, there *is* a hard limit of 32TB per table, arising from the limited
size of BlockNumber. But it's hard to believe that INPS's claim has
anything to do with that.

I suspect that INPS is something to do with the French police, so might
have been briefed by somebody who purports to know what he's talking
about. But I'm assuming that whoever put those ramblings on Pastebin is
unreliable, the only reason I mentioned it was that specific claims were
being made which might become widely repeated.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]

On 05/01/2011 01:50 AM, Mark Morgan Lloyd wrote:

Somebody is making a very specific claim that Postgres can support a
limited number of rows

Did you find this via
http://www.reversecurity.com/2011/04/new-details-from-psn-hack.html ?
That was the only Google-indexed source leading to it I found. I just
left a note there about the silliness of these claims. I could run more
than a 10M row PostgreSQL instance on my phone. Unless there's a new
16-bit only Vic 20 port of PostgreSQL available or something, it's seems
unlikely the data had to be partitioned due to any hard limit.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
"PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books

Greg Smith wrote:

On 05/01/2011 01:50 AM, Mark Morgan Lloyd wrote:

Somebody is making a very specific claim that Postgres can support a
limited number of rows

Did you find this via
http://www.reversecurity.com/2011/04/new-details-from-psn-hack.html ?
That was the only Google-indexed source leading to it I found. I just
left a note there about the silliness of these claims. I could run more
than a 10M row PostgreSQL instance on my phone. Unless there's a new
16-bit only Vic 20 port of PostgreSQL available or something, it's seems
unlikely the data had to be partitioned due to any hard limit.

Yes, via Google. I was digging around for any information about what
sort of access and APIs the network made available to end-user (or
ersatz developer) systems.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]