psql include text file with bom
*Summary:*
psql "include" or "\i" command chokes on UTF8 text files prefixed with BOM.
*Steps to reproduce:*
1. create a UTF8 file with three byte BOM 'EF BB BF'.
2. include the file from psql via the "include" or \i command.
Example output for file named "test.sql" below:
redacted-# \i test.sql
psql:test.sql:1: ERROR: syntax error at or near ""
LINE 1: 
*Background*
https://en.wikipedia.org/wiki/Byte_order_mark
Some text editors save text to a file prefixed by a BOM or byte marker.
This includes Visual Studio, VSCode and others.
I think it would be reasonable for the include command to skip over any
BOM found in the first two or three bytes of a file.
Rick Parrish <ai5jt@unitrunker.net> writes:
I think it would be reasonable for the include command to skip over any
BOM found in the first two or three bytes of a file.
This has been proposed before, and rejected before. psql has no
inherent knowledge of what encoding an input file is in, and therefore
no justification to assume that a bit-pattern it sees there is a BOM.
In non-UTF8 encodings it could very easily be valid data.
(For that matter, it's also valid data in UTF8: it's the same bit
pattern as U+FEFF ZERO WIDTH NO-BREAK SPACE. Programs that emit
one into UTF8 streams, and expect it not to be taken as data,
are frankly broken.)
regards, tom lane