BUG #14197: ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no equivalent in encoding "UTF8"

On Thu, Jun 16, 2016 at 8:22 AM, <sheri.bhavani@cognizant.com> wrote:

The following bug has been logged on the website:

Bug reference: 14197
Logged by: bhavani sheri
Email address: sheri.bhavani@cognizant.com
PostgreSQL version: 9.5.3
Operating system: Windows7
Description:

I have migrated from postgreSQL 7 to postgreSQL 9.5.3 and thus getting this
error.Please help me to resolve it.

I'd start with:

Google: ​
fixing character with byte sequence postgresql

​You use the past tense here "I have migrated" - did the migration succeed
and you are getting errors during normal operations or has some aspect of
your migration failed?

David J.​

Hi,

The migration was successful .

• while creating backup for the database we used postgresql 7 version .

• While restoring the same backup using postgreSQL 9.5.3 we are facing the mentioned problem.

• ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no equivalent in encoding "UTF8" is thrown in postgreSQL 9.5.3 .
Please help me out to resolve this error.

Regards,
Bhavani

From: bhavani, sheri (Cognizant)
Sent: Friday, June 17, 2016 10:27 AM
To: Bhanu Priya (Cognizant) (Bhanu.Priya3@cognizant.com)
Subject: FW: [BUGS] BUG #14197: ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no equivalent in encoding "UTF8"

From: David G. Johnston [mailto:david.g.johnston@gmail.com]
Sent: Friday, June 17, 2016 10:21 AM
To: bhavani, sheri (Cognizant)
Subject: Re: [BUGS] BUG #14197: ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no equivalent in encoding "UTF8"

Please direct replies to the list.

On Friday, June 17, 2016, <sheri.bhavani@cognizant.com<mailto:sheri.bhavani@cognizant.com>> wrote:
Hi,

The migration was successful .

• while creating backup for the database we used postgresql 7 version .

• While restoring the same backup using postgreSQL 9.5.3 we are facing the mentioned problem.

• ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no equivalent in encoding "UTF8" is thrown in postgreSQL 9.5.3 .
Please help me out to resolve this error.

Regards,
Bhavani

From: David G. Johnston [mailto:david.g.johnston@gmail.com<javascript:_e(%7B%7D,'cvml','david.g.johnston@gmail.com');>]
Sent: Thursday, June 16, 2016 6:58 PM
To: bhavani, sheri (Cognizant)
Cc: pgsql-bugs@postgresql.org<javascript:_e(%7B%7D,'cvml','pgsql-bugs@postgresql.org');>
Subject: Re: [BUGS] BUG #14197: ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no equivalent in encoding "UTF8"

On Thu, Jun 16, 2016 at 8:22 AM, <sheri.bhavani@cognizant.com<javascript:_e(%7B%7D,'cvml','sheri.bhavani@cognizant.com');>> wrote:
The following bug has been logged on the website:

Bug reference: 14197
Logged by: bhavani sheri
Email address: sheri.bhavani@cognizant.com<javascript:_e(%7B%7D,'cvml','sheri.bhavani@cognizant.com');>
PostgreSQL version: 9.5.3
Operating system: Windows7
Description:

I have migrated from postgreSQL 7 to postgreSQL 9.5.3 and thus getting this
error.Please help me to resolve it.

I'd start with:

Google: ​
fixing character with byte sequence postgresql

​You use the past tense here "I have migrated" - did the migration succeed and you are getting errors during normal operations or has some aspect of your migration failed?

David J.​

This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.

On 6/16/2016 10:08 PM, sheri.bhavani@cognizant.com wrote:

·ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no
equivalent in encoding "UTF8" is thrown in postgreSQL 9.5.3 .

per https://en.wikipedia.org/wiki/Windows-1252 0x81 is not a valid
character in encoding win1252, so it can't be converted to UTF8

you need to determine what field of which row of what table has that
value in it and change it to something valid (perhaps 0x20 ?) before you
can load this data into a UTF8 database.

--
john r pierce, recycling bits in santa cruz

I have sent the sheri.bhavani a presentation I wrote on how to fix this issue after we ran into the same issues.

When I was upgrading our SQL_ASCII from Postgres 9.2 to 9.4, I found that some of our queries would no longer work. This is due to changes in the internal functions so that if they run across multiple encoding, they will throw an error. So this forced us to revisit the project for converting all our data to UTF8. The problem was finding all that data and getting it converted. We had ACSII, UTF8, LATIN1, and WIN1252 encodings within our database and sometime different encodings for different rows of data within the same table.

In this presentation I will show you the functions to write, code supplied, to find all the non-ASCII characters in your database so that you can then manually figure out how to backup each section of data. Then we will cover backing up and restoring each of those sections of data.

If anyone else is interested, drop me a note and I will send you a copy of the presentation.

--
Lloyd Albin
Database Manager / Statistical Center for HIV/AIDS Research and Prevention (SCHARP) / lalbin@fredhutch.org / Fred Hutch / Cures Start Here

From: pgsql-bugs-owner@postgresql.org [mailto:pgsql-bugs-owner@postgresql.org] On Behalf Of John R Pierce
Sent: Friday, June 17, 2016 1:04 AM
To: sheri.bhavani@cognizant.com; pgsql-bugs@postgresql.org
Subject: Re: [BUGS] BUG #14197: ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no equivalent in encoding "UTF8"

On 6/16/2016 10:08 PM, sheri.bhavani@cognizant.com<mailto:sheri.bhavani@cognizant.com> wrote:
• ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no equivalent in encoding "UTF8" is thrown in postgreSQL 9.5.3 .

per https://en.wikipedia.org/wiki/Windows-1252 0x81 is not a valid character in encoding win1252, so it can't be converted to UTF8

you need to determine what field of which row of what table has that value in it and change it to something valid (perhaps 0x20 ?) before you can load this data into a UTF8 database.

--

john r pierce, recycling bits in santa cruz