Converting char to varchar automatically

On 10/6/2014 5:29 AM, Andrus wrote:

Database contains about 300 tables.
Most of them contain columns of char(n) type.
How to convert all those columns to varchar automatically ?
Is it possible to run some update commands in system tables for this ?
Or is it possible to create pgsql script which creates dynamically alter
table alter column commands and PERFORMs them ?
Any tables have primary keys with char(n) columns and foreign keys on
them. Foreign keys are deferrable and initially immediate.
Will foreign keys allow to perform such alter table alter column commands ?
Or is there better way.
Andrus.

I'd use a little perl.

Or if your editor has macros, you could use that.

change:
create table bob (
id char(50),
..
)

to

alter table bob alter id type varchar(50);

You might be able to query them out if you wanted:

select table_name, column_name, character_maximum_length
from information_schema.columns
where data_type = 'character'

Then use that to generate the alter table commands. Hum... this might
also work:

select 'alter table ' || table_name || ' alter ' || column_name .... etc

but that might try changing system tables which would be bad.

-Andy

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

This query might work for you, but double check all result statements first.

SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.' ||
quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE varchar;'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
WHERE t.typname = 'char'
AND n.nspname <> 'pg_catalog';

On Mon, Oct 6, 2014 at 6:29 AM, Andrus <kobruleht2@hot.ee> wrote:

Database contains about 300 tables.
Most of them contain columns of char(n) type.

How to convert all those columns to varchar automatically ?

Is it possible to run some update commands in system tables for this ?
Or is it possible to create pgsql script which creates dynamically alter
table alter column commands and PERFORMs them ?

Any tables have primary keys with char(n) columns and foreign keys on
them. Foreign keys are deferrable and initially immediate.
Will foreign keys allow to perform such alter table alter column commands ?
Or is there better way.

Andrus.

--
*Melvin Davidson*
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

Hi!

SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.' || quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE varchar;'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
WHERE t.typname = 'char'
AND n.nspname <> 'pg_catalog';

It does not return any data.
Andrus.

Melvin Davidson <melvin6925@gmail.com> writes:

This query might work for you, but double check all result statements first.

SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.' ||
quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE varchar;'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
WHERE t.typname = 'char'
AND n.nspname <> 'pg_catalog';

Make that "t.typname = 'bpchar'".

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 10/6/14, 12:41 PM, hari.fuchs@gmail.com wrote:

Melvin Davidson <melvin6925@gmail.com> writes:

This query might work for you, but double check all result statements first.

SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.' ||
quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE varchar;'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
WHERE t.typname = 'char'
AND n.nspname <> 'pg_catalog';

Make that "t.typname = 'bpchar'".

Just a heads-up: each of those ALTER's will rewrite the table, so unless your database is tiny this will be a slow process. There's ways to work around that, but they're significantly more complicated.

--
Jim Nasby, Data Architect, Blue Treble
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Also, don't forget to test for relkind = 'r'. My bad from before.
Revised query is below.

SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.'
|| quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE
varchar;'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
WHERE t.typname = 'bpchar'
AND c.relkind = 'r'
AND n.nspname <> 'pg_catalog';

On Mon, Oct 6, 2014 at 6:18 PM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:

On 10/6/14, 12:41 PM, hari.fuchs@gmail.com wrote:

Melvin Davidson <melvin6925@gmail.com> writes:

This query might work for you, but double check all result statements

first.

SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.' ||
quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE varchar;'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
WHERE t.typname = 'char'
AND n.nspname <> 'pg_catalog';

Make that "t.typname = 'bpchar'".

Just a heads-up: each of those ALTER's will rewrite the table, so unless

your database is tiny this will be a slow process. There's ways to work
around that, but they're significantly more complicated.

--
Jim Nasby, Data Architect, Blue Treble
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
*Melvin Davidson*
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

Jim Nasby <Jim.Nasby@BlueTreble.com> writes:

Just a heads-up: each of those ALTER's will rewrite the table, so unless your database is tiny this will be a slow process. There's ways to work around that, but they're significantly more complicated.

I think he's trying to get rid of all the blank-padding he's got right
now, so table rewrites are unavoidable.

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Melvin Davidson <melvin6925@gmail.com> writes:

Also, don't forget to test for relkind = 'r'. My bad from before.

In principle you need to ignore attisdropped columns as well.

Thinking about Jim's point about speed: it'd be wise to collapse any
updates for multiple columns in the same table into one ALTER command,
so that you only rewrite the table once, not once per column.

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

BTW, where can I find a list of type1->type2 pairs that doesn't
require full table lock for conversion?

ps. Sorry for top posting.

On Mon, Oct 6, 2014 at 4:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Melvin Davidson <melvin6925@gmail.com> writes:

Also, don't forget to test for relkind = 'r'. My bad from before.

In principle you need to ignore attisdropped columns as well.

Thinking about Jim's point about speed: it'd be wise to collapse any
updates for multiple columns in the same table into one ALTER command,
so that you only rewrite the table once, not once per column.

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
Kind regards,
Sergey Konoplev
PostgreSQL Consultant and DBA

http://www.linkedin.com/in/grayhemp
+1 (415) 867-9984, +7 (499) 346-7196, +7 (988) 888-1979
gray.ru@gmail.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Sergey Konoplev <gray.ru@gmail.com> writes:

BTW, where can I find a list of type1->type2 pairs that doesn't
require full table lock for conversion?

There aren't any. Sometimes you can skip a table rewrite, but that
doesn't mean that a lesser lock is possible.

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On Tue, Oct 7, 2014 at 10:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Sergey Konoplev <gray.ru@gmail.com> writes:

BTW, where can I find a list of type1->type2 pairs that doesn't
require full table lock for conversion?

There aren't any. Sometimes you can skip a table rewrite, but that
doesn't mean that a lesser lock is possible.

Oh, sorry, it was a typo, I meant "that doesn't require a full table rewrite".

--
Kind regards,
Sergey Konoplev
PostgreSQL Consultant and DBA

http://www.linkedin.com/in/grayhemp
+1 (415) 867-9984, +7 (499) 346-7196, +7 (988) 888-1979
gray.ru@gmail.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Sergey Konoplev <gray.ru@gmail.com> writes:

On Tue, Oct 7, 2014 at 10:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Sergey Konoplev <gray.ru@gmail.com> writes:

BTW, where can I find a list of type1->type2 pairs that doesn't
require full table lock for conversion?

There aren't any. Sometimes you can skip a table rewrite, but that
doesn't mean that a lesser lock is possible.

Oh, sorry, it was a typo, I meant "that doesn't require a full table rewrite".

Roughly speaking it's the pairs that have a binary (WITHOUT FUNCTION)
coercion according to pg_cast, although we have special logic for a few
cases such as varchar(M) -> varchar(N).

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On Tue, Oct 7, 2014 at 11:02 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Roughly speaking it's the pairs that have a binary (WITHOUT FUNCTION)
coercion according to pg_cast, although we have special logic for a few
cases such as varchar(M) -> varchar(N).

That ones?

select t1.typname, t2.typname
from pg_cast, pg_type as t1, pg_type as t2
where
t1.oid = castsource and t2.oid = casttarget and
castmethod = 'b' order by 1, 2;

--
Kind regards,
Sergey Konoplev
PostgreSQL Consultant and DBA

http://www.linkedin.com/in/grayhemp
+1 (415) 867-9984, +7 (499) 346-7196, +7 (988) 888-1979
gray.ru@gmail.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 10/6/14, 6:16 PM, Tom Lane wrote:

Jim Nasby <Jim.Nasby@BlueTreble.com> writes:

Just a heads-up: each of those ALTER's will rewrite the table, so unless your database is tiny this will be a slow process. There's ways to work around that, but they're significantly more complicated.

I think he's trying to get rid of all the blank-padding he's got right
now, so table rewrites are unavoidable.

Right, but there's other ways this could be done without requiring an outage. Like creating the new column with temporary name, put trigger on table, etc, etc.

Having dealt with an environment where downtime was thousands of dollars per minute I've gotten very creative at not taking outages. :)
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Hi!

also, it generates statement which tries to change all columns to one character length columns.

Andrus.

From: Andrus
Sent: Monday, October 06, 2014 8:11 PM
To: Melvin Davidson
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] Converting char to varchar automatically

Hi!

SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.' || quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE varchar;'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
WHERE t.typname = 'char'
AND n.nspname <> 'pg_catalog';

It does not return any data.
Andrus.

Hi!

Using Toms recommendation I added not attisdropped and now got the query

SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.'
|| quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE
varchar;'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
WHERE t.typname = 'bpchar'
AND c.relkind = 'r'
AND n.nspname <> 'pg_catalog' and not attisdropped;

Will this create commands which replace all user-defined char things in
database to varchar ?

"TYPE varchar" creates single character column so most alter table command
will fail.
How to change this so that original char column width is kept ?
I looked into tables used in this query but havent found column which holds
char column defined width.
How get it or is it better to re-write this query using informational_schema
?

How to change this query so that it creates single alter table command for
every table
(with multiple alter column clauses) to increase conversion speed ?

Andrus.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

This revised query should give you what you need:

SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.'
|| quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE
varchar(' || i.character_maximum_length || ');'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
JOIN information_schema.columns i ON (i.table_name = c.relname AND
i.column_name = a.attname)
WHERE t.typname = 'bpchar'
AND c.relkind = 'r'
AND n.nspname <> 'pg_catalog' and not attisdropped;

On Wed, Oct 8, 2014 at 3:34 PM, Andrus <kobruleht2@hot.ee> wrote:

Hi!

Using Toms recommendation I added not attisdropped and now got the query

SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.'
|| quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE
varchar;'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
WHERE t.typname = 'bpchar'
AND c.relkind = 'r'
AND n.nspname <> 'pg_catalog' and not attisdropped;

Will this create commands which replace all user-defined char things in
database to varchar ?

"TYPE varchar" creates single character column so most alter table command
will fail.
How to change this so that original char column width is kept ?
I looked into tables used in this query but havent found column which holds
char column defined width.
How get it or is it better to re-write this query using
informational_schema
?

How to change this query so that it creates single alter table command for
every table
(with multiple alter column clauses) to increase conversion speed ?

Andrus.

--
*Melvin Davidson*
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

Hi!

Thank you.

This revised query should give you what you need:
SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.'
|| quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE varchar(' || i.character_maximum_length || ');'
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
JOIN information_schema.columns i ON (i.table_name = c.relname AND i.column_name = a.attname)
WHERE t.typname = 'bpchar'
AND c.relkind = 'r'
AND n.nspname <> 'pg_catalog' and not attisdropped;

How to create single alter table command for every table ?
Can we use string concat aggregate function or window functions or plpgsql or something other ?

Andrus.

There really is no easy way to make a single ALTER for each table unless
you use a programming language. However, adding a
GROUP BY c.relname,
a.attname

would certainly simplify editing. Then you can combine all the
ALTER COLUMN's for each table.

On Wed, Oct 8, 2014 at 6:21 PM, Andrus <kobruleht2@hot.ee> wrote:

Hi!

Thank you.

This revised query should give you what you need:
SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.'
|| quote_ident(c.relname)
|| ' ALTER COLUMN ' || quote_ident(a.attname) || ' TYPE

varchar(' || i.character_maximum_length || ');'

FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid
JOIN pg_type t ON t.oid = a.atttypid
JOIN information_schema.columns i ON (i.table_name = c.relname AND

i.column_name = a.attname)

WHERE t.typname = 'bpchar'
AND c.relkind = 'r'
AND n.nspname <> 'pg_catalog' and not attisdropped;

How to create single alter table command for every table ?
Can we use string concat aggregate function or window functions or plpgsql
or something other ?

Andrus.

--
*Melvin Davidson*
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.