Need help with quote escaping in exim for postgresql

On Fri, Jul 07, 2006 at 03:48:00PM +0200, Marc Haber wrote:

Hi,

I am the maintainer of Debian's packages for exim4, a powerful and
versatile Mail Transfer Agent developed in Cambridge and in wide use
throughout the Free Software Community (http://www.exim.org/).

One of our daemon flavours has PostgreSQL support. Our security guys
have found a flaw in exim regarding quote escaping for PostgreSQL. The
bug is filed in Debian's BTS as http://bugs.debian.org/369351 and was
transferred to exim's Bugzilla installation as
http://www.exim.org/bugzilla/show_bug.cgi?id=107.

Whether or not the quick fix works for you depends entirly on the
encoding used by the client to talk to the database. If the connection
is encoded using UTF-8 or any of the Latin series, then it will be
fine. The only time it does not work is if the encoding is an encoding
where the quote or backslash character can appear as the second
character of a multibyte char. This doesn't happen with UTF-8 or any
latin encoding.

http://www.postgresql.org/docs/techdocs.50

This bit may be useful also (especially the second point):

There are a number of mitigating factors that may keep particular
applications from being subject to these security risks:

* If application always sends untrusted strings as out-of-line
parameters, instead of embedding them into SQL commands, it is not
vulnerable.
* If client_encoding is a single-byte encoding (e.g., one of the
LATINx family), there is no vulnerability.
* If application cannot pass invalidly encoded data to the server,
there is no vulnerability (this probably includes all Java
applications, for example, because of Java's handling of Unicode
strings).

The easiest may be to simply always set the client encoding to
something like UTF-8 and work the escaping rules so they work with
that.

Hope this helps,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

On Fri, Jul 07, 2006 at 03:48:00PM +0200, Marc Haber wrote:

From what I understand, the correct way would be to use
PQescapeStringConn, but that function needs an established connection,
and exim performs string escape "early", way before the actual
connection is established.

I just downloaded the code and looked. The code never looks or checks
the encoding of the database. This is bad from a security point of
view because that means you have no idea how your queries are going to
be interpreted.

I'd suggest adding a PQsetClientEncoding(conn, "Latin1") right after
you establish a connection. I'm not sure if Exim has any kind of
declaration about what encoding strings have internally. You could use
UTF-8 but then postgres would complain if you pass any strings that
arn't valid UTF-8. They may or may not be desirable.

SQL_ASCII may also be an option (assign no special meaning to
characters at all), but I'm less sure of that. Can email address
contain multibyte characters? I didn't think so... What about the
configuration file?

Hope this helps,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

On Fri, Jul 07, 2006 at 04:53:14PM +0200, Martijn van Oosterhout wrote:

On Fri, Jul 07, 2006 at 03:48:00PM +0200, Marc Haber wrote:

I am the maintainer of Debian's packages for exim4, a powerful and
versatile Mail Transfer Agent developed in Cambridge and in wide use
throughout the Free Software Community (http://www.exim.org/).

One of our daemon flavours has PostgreSQL support. Our security guys
have found a flaw in exim regarding quote escaping for PostgreSQL. The
bug is filed in Debian's BTS as http://bugs.debian.org/369351 and was
transferred to exim's Bugzilla installation as
http://www.exim.org/bugzilla/show_bug.cgi?id=107.

Whether or not the quick fix works for you depends entirly on the
encoding used by the client to talk to the database.

I am afraid that we don't have any information about the encoding used
since exim allows arbitrary SQL commands to be given in the run-time
configuration file.

* If application always sends untrusted strings as out-of-line
parameters, instead of embedding them into SQL commands, it is not
vulnerable.

exim's configuration holds SQL string literals like
|server_condition = "${if and { \
| {!eq{$2}{}} \
| {!eq{$3}{}} \
| {crypteq{$3}{${lookup mysql{SELECT password FROM customers WHERE ( domain_name = \
| '${domain:$2}' \
| AND local_part = '${local_part:$2}') OR user_id='$2'}{$value}fail}} }} {yes}{no}}"
(which is code helping exim to do SMTP AUTH against a password
database stored in mysql. Using PostgreSQL is very similiar, so you'll
get the idea - taken from
http://www.exim.org/eximwiki/AuthenticatedSmtpUsingMysql).

* If client_encoding is a single-byte encoding (e.g., one of the
LATINx family), there is no vulnerability.

From what I can see, exim configuration does not allow an encoding to
be set fot the communication with pgsql, and the code does not match
on "client_encoding". Am I right to assume that without special
handling, anything passed to it from the configuration is passed on
verbatim to the pgsql server?

* If application cannot pass invalidly encoded data to the server,
there is no vulnerability (this probably includes all Java
applications, for example, because of Java's handling of Unicode
strings).

I am afraid that it is quite easy to (mis)configure exim so that it is
possible to pass invalidly encoded data.

The easiest may be to simply always set the client encoding to
something like UTF-8 and work the escaping rules so they work with
that.

If that's possible, it might. We'd need to know how our input data is
encoded and recode to UTF-8, right?

Greetings
Marc

--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany | lose things." Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835

Hi,

On Fri, Jul 07, 2006 at 05:15:11PM +0200, Martijn van Oosterhout wrote:

On Fri, Jul 07, 2006 at 03:48:00PM +0200, Marc Haber wrote:

From what I understand, the correct way would be to use
PQescapeStringConn, but that function needs an established connection,
and exim performs string escape "early", way before the actual
connection is established.

I just downloaded the code and looked.

Thank you very much.

The code never looks or checks the encoding of the database. This is
bad from a security point of view because that means you have no idea
how your queries are going to be interpreted.

That's the way exim has always done it.

I'd suggest adding a PQsetClientEncoding(conn, "Latin1") right after
you establish a connection. I'm not sure if Exim has any kind of
declaration about what encoding strings have internally.

No, it does not.

You could use UTF-8 but then postgres would complain if you pass any
strings that arn't valid UTF-8. They may or may not be desirable.

Possible not desireable.

SQL_ASCII may also be an option (assign no special meaning to
characters at all), but I'm less sure of that. Can email address
contain multibyte characters? I didn't think so...

E-Mail addreses themselves can't, but the "comment" field of an
address can.

What about the configuration file?

It probably can as well.

Please note that exim is so flexible that it is possible to implement
mail spool storage in an SQL database. In this case, we'd write data
which originated in an untrusted source to the database, not knowing
about encoding at all.

I'm going to point Philip to this thread.

Greetings
Marc

--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany | lose things." Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835

* Martijn van Oosterhout:

* If application always sends untrusted strings as out-of-line
parameters, instead of embedding them into SQL commands, it is not
vulnerable.

This paragraph should explictly mention PQexecParams (which everybody
should use anyway).

It seems that Exim's architecture prevents the use of PQexecParams,
though.

On Sun, Jul 09, 2006 at 06:16:48PM +0200, Marc Haber wrote:

I'd suggest adding a PQsetClientEncoding(conn, "Latin1") right after
you establish a connection. I'm not sure if Exim has any kind of
declaration about what encoding strings have internally.

No, it does not.

That's your fundamental problem. You cannot safely quote strings if you
do not know the encoding of the string you're escaping. That's
fundamentally what this bug is about.

SQL_ASCII may also be an option (assign no special meaning to
characters at all), but I'm less sure of that. Can email address
contain multibyte characters? I didn't think so...

E-Mail addreses themselves can't, but the "comment" field of an
address can.

But no validation is done on these strings, ofcourse.

What about the configuration file?

It probably can as well.

Ditto.

I'd suggest forcing the client_encoding to SQL_ASCII. That way
multibyte characters will simply get dumped into the strings in the
backend. But at least there won't be any issues with the server
misinterpreting your string.

It may be worthwhile allowing the user to set the encoding, but that
just opens the issue up again, because then you actually have to
validate the strings are the encoding you think they are.

Please note that exim is so flexible that it is possible to implement
mail spool storage in an SQL database. In this case, we'd write data
which originated in an untrusted source to the database, not knowing
about encoding at all.

That will work just fine, as long as the database encoding is
SQL_ASCII, in which case there is no problem... What we're talking
about here is invalidly coded UTF-8 and things like SJIS.

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

Marc Haber wrote:

Please note that exim is so flexible that it is possible to implement
mail spool storage in an SQL database. In this case, we'd write data
which originated in an untrusted source to the database, not knowing
about encoding at all.

If you are going to store things in multiple encodings and you don't
know (or don't want to waste time figuring out) which encoding each is
on, probably you should be using SQL_ASCII. This means Postgres itself
will never try to mess with bytes (it will never recode stuff). The
downside is that the sort order may be incorrect in some queries, but
I'm not sure if this is a great problem for a mail spool.

Otherwise you do have to declare the encoding you are going to enter
data in (using the client_encoding config option). Not doing it causes
all sort of weird problems.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

SQL_ASCII may also be an option (assign no special meaning to
characters at all), but I'm less sure of that. Can email address
contain multibyte characters? I didn't think so...

E-Mail addreses themselves can't, but the "comment" field of an
address can.

The comment field itself, in RFC2822 addresses is ascii. However there is
a hack, by which this ascii string may be interpretted as representing a
string in another encoding. I don't remember the number of the RFC describing
how this works, but it shouldn't be hard to find.

On Jul 11, 2006, at 6:29 PM, Bruno Wolff III wrote:

SQL_ASCII may also be an option (assign no special meaning to
characters at all), but I'm less sure of that. Can email address
contain multibyte characters? I didn't think so...

E-Mail addreses themselves can't, but the "comment" field of an
address can.

The comment field itself, in RFC2822 addresses is ascii. However
there is
a hack, by which this ascii string may be interpretted as
representing a
string in another encoding. I don't remember the number of the RFC
describing
how this works, but it shouldn't be hard to find.

RFC 1522. Whether you'd want to open that can of worms
by decoding headers in that format to some other character
set for email is a good question (especially as a lot of spam
has headers that end up containing illegal characters if
you do that).

If you were to do that you'd probably have to deal with
i18n domain name encoding too, which is even more hideous.

Fortunately all this stuff is MUA-side, not MTA-side, so exim
should ignore it. SQL_ASCII all the way.

Cheers,
Steve

[ Coming late to the thread... ]

Steve Atkins <steve@blighty.com> writes:

Fortunately all this stuff is MUA-side, not MTA-side, so exim
should ignore it. SQL_ASCII all the way.

I concur. The recent encoding fixes are for the situation where the
database server believes a multibyte encoding is in use, but the client
code is ignorant of that encoding and either (a) sends invalidly encoded
data or (b) does escaping that mangles multibyte characters.

If your client-side code is encoding agnostic, then using SQL_ASCII
(which is also effectively encoding agnostic) for both client_encoding
and server_encoding will work nicely.

A possibly safer choice is to use LATIN1 (or another single-byte
encoding) instead; this will avoid problems if someone connects to the
database with a client_encoding other than SQL_ASCII and expects data
to be delivered to him in that encoding.

I would *not* recommend using UTF8 if you want to store arbitrary data
without worrying about encoding issues.

regards, tom lane

On Fri, Jul 07, 2006 at 03:48:00PM +0200, Marc Haber wrote:

I am the maintainer of Debian's packages for exim4, a powerful and
versatile Mail Transfer Agent developed in Cambridge and in wide use
throughout the Free Software Community (http://www.exim.org/).

One of our daemon flavours has PostgreSQL support. Our security guys
have found a flaw in exim regarding quote escaping for PostgreSQL. The
bug is filed in Debian's BTS as http://bugs.debian.org/369351 and was
transferred to exim's Bugzilla installation as
http://www.exim.org/bugzilla/show_bug.cgi?id=107.

I would like to thank anybody who commented in this thread. Special
thanks go to Florian Weimer, who has done the task of communicating
with Philip Hazel, the exim author, and working out a fix which has
since been released as part of exim 4.63.

Greetings
Marc

--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany | lose things." Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835