custom function for converting human readable sizes to bytes

Started by Pavel Stehuleover 10 years ago94 messageshackers
Jump to latest
#1Pavel Stehule
pavel.stehule@gmail.com

Hi

I have to write some filters, and filtering by size is "unfriendly" due
calculation in bytes.

I propose inversion function to pg_size_pretty function - like
pg_human_size.

Usage:

SELECT * FROM pg_class
WHERE pg_table_size(oid) > pg_human_size('2GB');

Ideas, comments?

Regards

Pavel

#2Jim Nasby
Jim.Nasby@BlueTreble.com
In reply to: Pavel Stehule (#1)
Re: custom function for converting human readable sizes to bytes

On 11/21/15 12:49 AM, Pavel Stehule wrote:

I propose inversion function to pg_size_pretty function - like
pg_human_size.

Usage:

SELECT * FROM pg_class
WHERE pg_table_size(oid) > pg_human_size('2GB');

I'm not a big fan of the name, but +1 on the general idea.

Maybe pg_size_pretty(text)?
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#3Pavel Stehule
pavel.stehule@gmail.com
In reply to: Jim Nasby (#2)
Re: custom function for converting human readable sizes to bytes

2015-11-22 18:30 GMT+01:00 Jim Nasby <Jim.Nasby@bluetreble.com>:

On 11/21/15 12:49 AM, Pavel Stehule wrote:

I propose inversion function to pg_size_pretty function - like
pg_human_size.

Usage:

SELECT * FROM pg_class
WHERE pg_table_size(oid) > pg_human_size('2GB');

I'm not a big fan of the name, but +1 on the general idea.

I am for any other good name

Maybe pg_size_pretty(text)?

I understand to your idea, but it can be too strange - function and
inversion function share same name.

What about pg_size(text), pg_size(value bigint, unit text) ?

Regards

Pavel

Show quoted text

--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

#4Marko Tiikkaja
marko@joh.to
In reply to: Jim Nasby (#2)
Re: custom function for converting human readable sizes to bytes

On 2015-11-22 18:30, Jim Nasby wrote:

On 11/21/15 12:49 AM, Pavel Stehule wrote:

I propose inversion function to pg_size_pretty function - like
pg_human_size.

Usage:

SELECT * FROM pg_class
WHERE pg_table_size(oid) > pg_human_size('2GB');

I'm not a big fan of the name, but +1 on the general idea.

Maybe pg_size_pretty(text)?

pg_ytterp_ezis(text)

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#5Jim Nasby
Jim.Nasby@BlueTreble.com
In reply to: Pavel Stehule (#3)
Re: custom function for converting human readable sizes to bytes

On 11/22/15 2:11 PM, Pavel Stehule wrote:

What about pg_size(text), pg_size(value bigint, unit text) ?

I like, though I'd make it numeric or float. pg_size(3.5, 'GB')
certainly seems like a reasonable use case...
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#6Pavel Stehule
pavel.stehule@gmail.com
In reply to: Jim Nasby (#5)
Re: custom function for converting human readable sizes to bytes

2015-11-22 21:19 GMT+01:00 Jim Nasby <Jim.Nasby@bluetreble.com>:

On 11/22/15 2:11 PM, Pavel Stehule wrote:

What about pg_size(text), pg_size(value bigint, unit text) ?

I like, though I'd make it numeric or float. pg_size(3.5, 'GB') certainly
seems like a reasonable use case...

yes, good note.

Regards

Pavel

Show quoted text

--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

#7Guillaume Lelarge
guillaume@lelarge.info
In reply to: Pavel Stehule (#6)
Re: custom function for converting human readable sizes to bytes

Le 22 nov. 2015 21:29, "Pavel Stehule" <pavel.stehule@gmail.com> a écrit :

2015-11-22 21:19 GMT+01:00 Jim Nasby <Jim.Nasby@bluetreble.com>:

On 11/22/15 2:11 PM, Pavel Stehule wrote:

What about pg_size(text), pg_size(value bigint, unit text) ?

I like, though I'd make it numeric or float. pg_size(3.5, 'GB')

certainly seems like a reasonable use case...

yes, good note.

What about pg_size_unpretty()?

#8Corey Huinker
corey.huinker@gmail.com
In reply to: Guillaume Lelarge (#7)
Re: custom function for converting human readable sizes to bytes

What about pg_size_unpretty()?

I was going to suggest pg_size_ugly(), but unpretty does emphasize the
inverse (rather than opposite) nature of the function.

#9Pavel Stehule
pavel.stehule@gmail.com
In reply to: Corey Huinker (#8)
Re: custom function for converting human readable sizes to bytes

2015-11-22 23:54 GMT+01:00 Corey Huinker <corey.huinker@gmail.com>:

What about pg_size_unpretty()?

I was going to suggest pg_size_ugly(), but unpretty does emphasize the
inverse (rather than opposite) nature of the function.

"unpretty", "ugly" aren't good names

maybe pg_size_bytes or different approach

we can introduce data type - bytesize - default input/output is human
readable text - and conversion to bigint is implicit

Some like

select .. where pg_table_size(oid) > bytesize '3.5GB'

and instead pg_size_pretty(pg_table_size(oid)) we can write
pg_table_size(oid)::bytesize

Regards

Pavel

#10Corey Huinker
corey.huinker@gmail.com
In reply to: Pavel Stehule (#9)
Re: custom function for converting human readable sizes to bytes

On Sun, Nov 22, 2015 at 11:24 PM, Pavel Stehule <pavel.stehule@gmail.com>
wrote:

2015-11-22 23:54 GMT+01:00 Corey Huinker <corey.huinker@gmail.com>:

What about pg_size_unpretty()?

I was going to suggest pg_size_ugly(), but unpretty does emphasize the
inverse (rather than opposite) nature of the function.

"unpretty", "ugly" aren't good names

maybe pg_size_bytes or different approach

we can introduce data type - bytesize - default input/output is human
readable text - and conversion to bigint is implicit

Some like

select .. where pg_table_size(oid) > bytesize '3.5GB'

and instead pg_size_pretty(pg_table_size(oid)) we can write
pg_table_size(oid)::bytesize

Regards

Pavel

+1 to both pg_size_bytes() and ::bytesize. Both contribute to making the
statements more self-documenting.

#11Jim Nasby
Jim.Nasby@BlueTreble.com
In reply to: Corey Huinker (#10)
Re: custom function for converting human readable sizes to bytes

On 11/23/15 3:11 AM, Corey Huinker wrote:

+1 to both pg_size_bytes() and ::bytesize. Both contribute to making the
statements more self-documenting.

The function seems like overkill to me if we have the type. Just my
opinion though. I'm thinking the type could just be called 'size' too
(or prettysize?). No reason it has to be tied to bytes (in particular
this would work for bits too).

If we're going to add this, I suppose it should support the 'i prefixes'
too (GiB, MiB, etc).
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#12Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jim Nasby (#11)
Re: custom function for converting human readable sizes to bytes

Jim Nasby <Jim.Nasby@BlueTreble.com> writes:

On 11/23/15 3:11 AM, Corey Huinker wrote:

+1 to both pg_size_bytes() and ::bytesize. Both contribute to making the
statements more self-documenting.

The function seems like overkill to me if we have the type. Just my
opinion though. I'm thinking the type could just be called 'size' too
(or prettysize?). No reason it has to be tied to bytes (in particular
this would work for bits too).

Please, no. That's *way* too generic a name.

I do not actually agree with making a type for this anyway. I can
tolerate a function, but adding a datatype is overkill; and it will
introduce far more definitional issues than it's worth. (eg, which
other types should have casts to/from it, and at what level)

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#13Pavel Stehule
pavel.stehule@gmail.com
In reply to: Tom Lane (#12)
Re: custom function for converting human readable sizes to bytes

2015-11-23 18:04 GMT+01:00 Tom Lane <tgl@sss.pgh.pa.us>:

Jim Nasby <Jim.Nasby@BlueTreble.com> writes:

On 11/23/15 3:11 AM, Corey Huinker wrote:

+1 to both pg_size_bytes() and ::bytesize. Both contribute to making the
statements more self-documenting.

The function seems like overkill to me if we have the type. Just my
opinion though. I'm thinking the type could just be called 'size' too
(or prettysize?). No reason it has to be tied to bytes (in particular
this would work for bits too).

Please, no. That's *way* too generic a name.

I do not actually agree with making a type for this anyway. I can
tolerate a function, but adding a datatype is overkill; and it will
introduce far more definitional issues than it's worth. (eg, which
other types should have casts to/from it, and at what level)

so pg_size_bytes is good enough for everybody?

Regards

Pavel

Show quoted text

regards, tom lane

#14Gavin Flower
GavinFlower@archidevsys.co.nz
In reply to: Pavel Stehule (#13)
Re: custom function for converting human readable sizes to bytes

On 24/11/15 06:31, Pavel Stehule wrote:

2015-11-23 18:04 GMT+01:00 Tom Lane <tgl@sss.pgh.pa.us
<mailto:tgl@sss.pgh.pa.us>>:

Jim Nasby <Jim.Nasby@BlueTreble.com> writes:

On 11/23/15 3:11 AM, Corey Huinker wrote:

+1 to both pg_size_bytes() and ::bytesize. Both contribute to

making the

statements more self-documenting.

The function seems like overkill to me if we have the type. Just my
opinion though. I'm thinking the type could just be called

'size' too

(or prettysize?). No reason it has to be tied to bytes (in

particular

this would work for bits too).

Please, no. That's *way* too generic a name.

I do not actually agree with making a type for this anyway. I can
tolerate a function, but adding a datatype is overkill; and it will
introduce far more definitional issues than it's worth. (eg, which
other types should have casts to/from it, and at what level)

so pg_size_bytes is good enough for everybody?

Regards

Pavel

regards, tom lane

perhaps pg_size_bites for those people who want: KiB, MiB, GiB, TiB,
PiB, ,.. ??? :-)

Cheers,
Gavin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#15Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Pavel Stehule (#13)
Re: custom function for converting human readable sizes to bytes

Pavel Stehule wrote:

so pg_size_bytes is good enough for everybody?

That seems good enough to me.

I would have it accept GiB and GB and have both transform to base 2, and
have an optional boolean flag whose non-default value turns the GB
interpretation into base 10, leaving the GiB interpretation unaffected.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#16Robert Haas
robertmhaas@gmail.com
In reply to: Alvaro Herrera (#15)
Re: custom function for converting human readable sizes to bytes

On Mon, Nov 23, 2015 at 1:47 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Pavel Stehule wrote:

so pg_size_bytes is good enough for everybody?

That seems good enough to me.

I would have it accept GiB and GB and have both transform to base 2, and
have an optional boolean flag whose non-default value turns the GB
interpretation into base 10, leaving the GiB interpretation unaffected.

I think it should be orange.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#17Pavel Stehule
pavel.stehule@gmail.com
In reply to: Alvaro Herrera (#15)
Re: custom function for converting human readable sizes to bytes

Hi

2015-11-23 19:47 GMT+01:00 Alvaro Herrera <alvherre@2ndquadrant.com>:

Pavel Stehule wrote:

so pg_size_bytes is good enough for everybody?

That seems good enough to me.

I would have it accept GiB and GB and have both transform to base 2, and
have an optional boolean flag whose non-default value turns the GB
interpretation into base 10, leaving the GiB interpretation unaffected.

attached proof concept based on parser "parse_int" from guc.c

It works well to 1TB what is enough for memory setting, but too low for
proposed target.

There are two ways

1. enhance the "parse_int"

2. using independent implementation - there is some redundant code, but we
can support duble insted int, and we can support some additional units.

Regards

Pavel

Show quoted text

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

pg-size-bytes-poc.patchtext/x-patch; charset=US-ASCII; name=pg-size-bytes-poc.patchDownload+26-0
#18Jim Nasby
Jim.Nasby@BlueTreble.com
In reply to: Pavel Stehule (#17)
Re: custom function for converting human readable sizes to bytes

On 11/24/15 10:57 PM, Pavel Stehule wrote:

+ errmsg("parameter \"%s\" isn't valid size value",

Should read " isn't a valid size value"
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#19Kevin Grittner
Kevin.Grittner@wicourts.gov
In reply to: Jim Nasby (#18)
Re: custom function for converting human readable sizes to bytes

On Wed, Nov 25, 2015 at 12:57 AM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:

Should read " isn't a valid size value"

http://www.postgresql.org/docs/devel/static/error-style-guide.html#AEN110918

| Contractions. Avoid contractions, like "can't"; use "cannot" instead.

So " is not a valid size value" would be better.

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#20Pavel Stehule
pavel.stehule@gmail.com
In reply to: Pavel Stehule (#17)
Re: custom function for converting human readable sizes to bytes

Hi

2. using independent implementation - there is some redundant code, but we
can support duble insted int, and we can support some additional units.

new patch is based on own implementation - use numeric/bigint calculations

+ regress tests and doc

Regards

Pavel

Attachments:

pg-size-bytes-01.patchtext/x-patch; charset=US-ASCII; name=pg-size-bytes-01.patchDownload+201-0
#21Kyotaro Horiguchi
horikyota.ntt@gmail.com
In reply to: Pavel Stehule (#20)
#22Pavel Stehule
pavel.stehule@gmail.com
In reply to: Kyotaro Horiguchi (#21)
#23Pavel Stehule
pavel.stehule@gmail.com
In reply to: Pavel Stehule (#22)
#24Pavel Stehule
pavel.stehule@gmail.com
In reply to: Pavel Stehule (#23)
#25Robert Haas
robertmhaas@gmail.com
In reply to: Pavel Stehule (#24)
#26Michael Paquier
michael@paquier.xyz
In reply to: Robert Haas (#25)
#27Pavel Stehule
pavel.stehule@gmail.com
In reply to: Michael Paquier (#26)
#28Michael Paquier
michael@paquier.xyz
In reply to: Pavel Stehule (#27)
#29Pavel Stehule
pavel.stehule@gmail.com
In reply to: Michael Paquier (#28)
#30Pavel Stehule
pavel.stehule@gmail.com
In reply to: Robert Haas (#25)
#31Shulgin, Oleksandr
oleksandr.shulgin@zalando.de
In reply to: Pavel Stehule (#30)
#32Pavel Stehule
pavel.stehule@gmail.com
In reply to: Shulgin, Oleksandr (#31)
#33Shulgin, Oleksandr
oleksandr.shulgin@zalando.de
In reply to: Pavel Stehule (#32)
#34Robert Haas
robertmhaas@gmail.com
In reply to: Shulgin, Oleksandr (#31)
#35Pavel Stehule
pavel.stehule@gmail.com
In reply to: Robert Haas (#34)
#36Shulgin, Oleksandr
oleksandr.shulgin@zalando.de
In reply to: Pavel Stehule (#35)
#37Pavel Stehule
pavel.stehule@gmail.com
In reply to: Shulgin, Oleksandr (#36)
#38Robert Haas
robertmhaas@gmail.com
In reply to: Pavel Stehule (#37)
#39Pavel Stehule
pavel.stehule@gmail.com
In reply to: Robert Haas (#38)
#40Thom Brown
thom@linux.com
In reply to: Pavel Stehule (#37)
#41Shulgin, Oleksandr
oleksandr.shulgin@zalando.de
In reply to: Robert Haas (#38)
#42Pavel Stehule
pavel.stehule@gmail.com
In reply to: Shulgin, Oleksandr (#41)
#43Shulgin, Oleksandr
oleksandr.shulgin@zalando.de
In reply to: Pavel Stehule (#42)
#44Robert Haas
robertmhaas@gmail.com
In reply to: Shulgin, Oleksandr (#41)
#45Shulgin, Oleksandr
oleksandr.shulgin@zalando.de
In reply to: Robert Haas (#44)
#46Pavel Stehule
pavel.stehule@gmail.com
In reply to: Robert Haas (#44)
#47Pavel Stehule
pavel.stehule@gmail.com
In reply to: Shulgin, Oleksandr (#43)
#48Robert Haas
robertmhaas@gmail.com
In reply to: Shulgin, Oleksandr (#45)
#49Pavel Stehule
pavel.stehule@gmail.com
In reply to: Robert Haas (#48)
#50Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Robert Haas (#48)
#51Robert Haas
robertmhaas@gmail.com
In reply to: Alvaro Herrera (#50)
#52Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Robert Haas (#38)
#53Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Pavel Stehule (#47)
#54Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Vitaly Burovoy (#53)
#55Robert Haas
robertmhaas@gmail.com
In reply to: Vitaly Burovoy (#52)
#56Pavel Stehule
pavel.stehule@gmail.com
In reply to: Robert Haas (#55)
#57Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#52)
#58Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#53)
#59Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Pavel Stehule (#58)
#60Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#59)
#61Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Pavel Stehule (#58)
#62Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#61)
#63Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#61)
#64Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Pavel Stehule (#63)
#65Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#64)
#66Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Pavel Stehule (#65)
#67Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#66)
#68Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Pavel Stehule (#67)
#69Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#68)
#70Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Pavel Stehule (#69)
#71Tom Lane
tgl@sss.pgh.pa.us
In reply to: Vitaly Burovoy (#70)
#72Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#68)
#73Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Pavel Stehule (#72)
#74Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#73)
#75Pavel Stehule
pavel.stehule@gmail.com
In reply to: Pavel Stehule (#74)
#76Pavel Stehule
pavel.stehule@gmail.com
In reply to: Pavel Stehule (#74)
#77Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#73)
#78Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Pavel Stehule (#76)
#79Pavel Stehule
pavel.stehule@gmail.com
In reply to: Vitaly Burovoy (#78)
#80Dean Rasheed
dean.a.rasheed@gmail.com
In reply to: Pavel Stehule (#79)
#81Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Dean Rasheed (#80)
#82Dean Rasheed
dean.a.rasheed@gmail.com
In reply to: Heikki Linnakangas (#81)
#83Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Dean Rasheed (#82)
#84Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Vitaly Burovoy (#83)
#85Pavel Stehule
pavel.stehule@gmail.com
In reply to: Dean Rasheed (#82)
#86Dean Rasheed
dean.a.rasheed@gmail.com
In reply to: Pavel Stehule (#85)
#87Pavel Stehule
pavel.stehule@gmail.com
In reply to: Dean Rasheed (#86)
#88Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Dean Rasheed (#86)
#89Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Vitaly Burovoy (#88)
#90Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Vitaly Burovoy (#89)
#91Dean Rasheed
dean.a.rasheed@gmail.com
In reply to: Vitaly Burovoy (#90)
#92Vitaly Burovoy
vitaly.burovoy@gmail.com
In reply to: Dean Rasheed (#91)
#93Dean Rasheed
dean.a.rasheed@gmail.com
In reply to: Vitaly Burovoy (#92)
#94Dean Rasheed
dean.a.rasheed@gmail.com
In reply to: Dean Rasheed (#93)