RustgreSQL

On 08/01/17 22:09, Joel Jacobson wrote:

Hi all,

Is anyone working on porting PostgreSQL to Rust?

Corrode looks a bit limited for the task, but maybe it can be a start.
It doesn't support goto or switch, but maybe the gotos patterns are
not too complicated.

My motivation is primarily I don't want to learn all the
over-complicated details of C,
but at the same time I would like to be productive in a safe system
language,
a category in which Rust seems to be alone.

Porting PostgreSQL to Rust would be a multi-year project,
and it could only be done if the process could be fully automated,
by supporting all the coding patterns used by the project,
otherwise a Rust-port would quickly fall behind the master branch.
But if all git commits could be automatically converted to Rust,
then the RustgreSQL project could pull all commits from upstream
until all development has switched over to Rust among all developers.

Is this completely unrealistic or is it carved in stone PostgreSQL
will always be a C project forever and ever?

From my very limited understanding, PostgreSQL is more likely to be
converted to C++!

Cheers,
Gavin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Is this completely unrealistic or is it carved in stone PostgreSQL will
always be a C project forever and ever?

From my very limited understanding, PostgreSQL is more likely to be converted
to C++!

ISTM that currently pg is written C89. Personnaly I think that C99
(standard from 18 years ago...) would be a progress, but this has been
rejected in the past because of portability issues on some platforms (eg
MS Visual C++ started to support part of C99 in ... 2013).

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Joel Jacobson asked:

Is anyone working on porting PostgreSQL to Rust?

No; extremely unlikely.

My motivation is primarily I don't want to learn all the
over-complicated details of C

Well that's going to be a show-stopper right there. For a proper
port, a deep understanding of the current source code is necessary.
You'd need a team expert in both C and Rust to pull it off.

Porting PostgreSQL to Rust would be a multi-year project,
and it could only be done if the process could be fully automated,
by supporting all the coding patterns used by the project,
otherwise a Rust-port would quickly fall behind the master branch.
But if all git commits could be automatically converted to Rust,

Developing such a system is bordering on AI and likely more complex
than Postgres itself. :)

Is this completely unrealistic or is it carved in stone PostgreSQL will
always be a C project forever and ever?

It's unrealistic, but there is nothing to say Postgres will stay in C
forever. Right now, however, there is no compelling reason to move
away from it, and the porting effort to any language would be immense.
C++ would be the least painful option, probably.

- --
Greg Sabino Mullane greg@turnstep.com
End Point Corporation http://www.endpoint.com/
PGP Key: 0x14964AC8 201701080905
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAlhyR44ACgkQvJuQZxSWSsimzgCg97QZZ47BfNtema5aoN2QIpY9
wTUAn3B042YDH82GPLDwXmDSgJMzsoGD
=PH10
-----END PGP SIGNATURE-----

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 8 Jan. 2017 17:10, "Joel Jacobson" <joel@trustly.com> wrote:

Is this completely unrealistic or is it carved in stone PostgreSQL will
always be a C project forever and ever?

Incredibly unrealistic.

PostgreSQL makes heavy use of variable length arrays. longjmp() is critical
to its error handling. Lots of other "unsafe" things.

More to the point though, we struggle to get any acceptance of the most
trivial added dependences. Even optionally. We still support a
more-than-10-year-old Perl. We still use C89. The project is VERY
conservative and expects new releases to run on ancient UNIXes long
forgotten by the rest of the world.

This has some advantages. Those weird compilers and platforms help catch
bugs. Most of the time they cost us little but minor inconvenience and
they're part of why some contributors stay around.

But it also makes it way harder to make significant change.

IMO the chances of the project switching to Rust are about as high as
Oracle Database going open source, or MongoDB declaring that it's changing
to SQL as the primary and preferred query language.

The ONLY way I could imagine it happening would be if you could show that
it could be done incrementally, in a way that retained support for 10+ year
old platforms, with a significant increase in performance and decrease in
code size/complexity for converted modules. With minimal or no "rust
droppings" (macros everywhere etc) to help such an incremental adaptation
along. Even then you'd have to convince a lot of people who know C well (or
well enough, or think they do) that it was worth such massive change. If a
low pain seamless conversation/adaptation like that were possible I'd have
to wonder what Rust could actually offer us over C since it clearly has the
same scope for issues if such an intermixture is possible. Kind of a
catch-22.

A restricted subset of C++ is a lot more likely. Even then longjmp will
cause us pain... I suspect we'd land up having to move to C++ exceptions.

Take a look at elog.c, the memory contexts code, etc. If you think Rust can
play well with that, cool. I can't imagine how though.

You'd have a lot more chance writing extensions in Rust though. If you can
make it play OK with the exception handling in postgres and our memory
management, at least. If you really wanted to push this forward... start
there. Show how great it is.

Then come up with a plan for how you'd handle existing extensions
(PostGIS?), external PLs, ecpg, pgxs, etc. Make sure libpq stays totally
compatible. All that fun.

I just don't see it happening. Not do I see you suggesting any possible
reason why we'd care or want to. You don't want to to learn C so
dozens/hundreds of people need to learn Rust. What the?

On 8 Jan. 2017 18:14, "Fabien COELHO" <coelho@cri.ensmp.fr> wrote:

Is this completely unrealistic or is it carved in stone PostgreSQL will

always be a C project forever and ever?

From my very limited understanding, PostgreSQL is more likely to be
converted to C++!

ISTM that currently pg is written C89. Personnaly I think that C99
(standard from 18 years ago...) would be a progress, but this has been
rejected in the past because of portability issues on some platforms (eg MS
Visual C++ started to support part of C99 in ... 2013).

MSVC was really the main issue. MS really insisted that C++ was the future
and C99 was a pointless diversion.

I kinda agree with them TBH, albeit with a preference for a small-ish and
carefully used subset of C++. But they've recognised that it matters to
enough people to add support now anyways. I suspect their increased
interest in open source and Linux is related.

Is this completely unrealistic or is it carved in stone PostgreSQL will
always be a C project forever and ever?

Another thing to look at if you want to approach this as a serious,
practical effort is the atomics, memory barrier, spinlock and lwlock code.

I just don't see it happening.

Thank you Craig for explaining the current C state of the project,
very interesting to learn about.

On Sun, Jan 8, 2017 at 4:19 AM, Craig Ringer
<craig.ringer@2ndquadrant.com> wrote:

If a low pain seamless conversation/adaptation like that were possible I'd have to wonder what Rust could actually offer us over C since it clearly has the same scope for issues if such an intermixture is possible. Kind of a catch-22.

Not necessarily. If you write non-idiomatic Rust and preserve as much
of the code style as possible from the original C code, and just focus
on getting ride of all usage of unsafe, then it will be more easily
understandable by existing C developers than if strictly writing
idiomatic Rust code.

You'd have a lot more chance writing extensions in Rust though. If you can make it play OK with the exception handling in postgres and our memory management, at least. If you really wanted to push this forward... start there. Show how great it is.

Funny you mention, that's actually exactly how these thoughts started
in my head. I realized I would need to write a C function to do some
computations that needed to be fast, but since C is dangerous I don't
know the language well enough I thought "I wish there was a safe and
fast system language you could
write database functions in!", and that's how I first found Rust and
then later this project:
https://github.com/thehydroimpulse/postgres-extension.rs
Which allows doing exactly that, writing extensions in Rust.

You don't want to to learn C so dozens/hundreds of people need to learn Rust. What the?

Oh, I don't think you seriously think I meant to suggest others should
learn Rust just because I don't want to learn C.
I don't want to learn the complicated details of C, that's true.
But that has nothing to do why others would need to learn Rust. They
don't, unless the majority of the project would also want to move to
Rust, and that has of course nothing to do with me.
I'm just asking possibly stupid questions and having possibly stupid
theories, trying to understand why such a project would be possible or
not.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 1/8/17 10:28 AM, Joel Jacobson wrote:

You'd have a lot more chance writing extensions in Rust though. If you can make it play OK with the exception handling in postgres and our memory management, at least. If you really wanted to push this forward... start there. Show how great it is.

Funny you mention, that's actually exactly how these thoughts started
in my head. I realized I would need to write a C function to do some
computations that needed to be fast, but since C is dangerous I don't
know the language well enough I thought "I wish there was a safe and
fast system language you could
write database functions in!", and that's how I first found Rust and
then later this project:
https://github.com/thehydroimpulse/postgres-extension.rs
Which allows doing exactly that, writing extensions in Rust.

Somewhat related to that... it would be useful if Postgres had "fenced"
functions; functions that ran in a separate process and only talked to a
backend via a well defined API (such as libpq). There's two major
advantages that would give us:

- Untrusted languages could be made trusted. Currently there's no way to
limit what a user could do with a function like plpythonu or pl/r, and
there's no good way to make either of those languages trusted. But if
the function code was running in a separate process, that process could
be owned by a user and group (ie: nobody:nogroup) that has no
permissions to do anything at the OS level. This would make it possible
for hosting platforms like RDS to offer these languages without undue
risk. I believe Joe Conway has done some investigation in this area.

- If you had a buggy C function that crashed it wouldn't force us to
panic the entire database. The fenced process would die, we'd detect
that and just through a normal error. Currently, C functions can access
*everything* in backend memory, including shared memory. That's why if
one crashes we have to panic. Even worse, a bug that corrupts data but
doesn't always crash could do enormous amounts of damage. The worst a
bug in a fenced function could do is screw up whatever it's returning to
the backend.

As for Postgres in general, I think it would be nice if it was easier to
do some things in a language other than C, for code simplicity reasons.
The issue with that though is not significantly expanding what's
necessary to build Postgres. I doubt that rust would meet that criteria.

That said, the community will at least consider things that offer
*significant* advantages. For example, there's been some discussion
about making use of LLVM for the executor. Since LLVM is meant to make
it easier to build new languages, it's possible that it could be used to
simplify other pieces of code as well.

BTW, I just came across http://safecode.cs.illinois.edu; that might get
you a lot of the benefits you're looking for in Rust. The description on
http://llvm.org claims it can be used like Valgrind, which the project
currently supports.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Sunday, January 8, 2017 6:28:17 AM EST Joel Jacobson wrote:

I don't want to learn the complicated details of C, that's true.

And this is where I think you're wrong, and why conversion would be hard. C
has very few complicated details. I don't think it has any, frankly. It
basically says "If you want your datastructure to outlive a function call,
I'll give you a chunk of memory and you're now responsible for it. Have fun".
That's not complicated: it's two functions, malloc() and free(), basically.

What's hard and complicated is keeping track of all those little chunks of
memory you have laying around. That management is deeply intertwined with the
algorithmics of the system, and separating memory management from the actual
work done will be very hard. In many cases the algorithm will have been
implemented with cheap memory management in mind, and porting it to a
different paradigm (garbage collection or rust's reference ownership) can
result in either a lot of work and probably bugs, or bad performing code.

I personally find keeping track of allocated memory easier than rust's
convoluted (to me) rules regarding reference ownership.

Your fear of C in unfounded. The definitive c89 reference is a little book of
about 250 pages, more than half of which is about the standard library of
which you'll never use more than half. If you have some notepaper laying about
on which to scribble pointer diagrams you can be a C programmer to :-)

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Mon, Jan 9, 2017 at 6:51 AM, Jan de Visser <jan@de-visser.net> wrote:

Your fear of C in unfounded. The definitive c89 reference is a little book of
about 250 pages, more than half of which is about the standard library of
which you'll never use more than half. If you have some notepaper laying about
on which to scribble pointer diagrams you can be a C programmer to :-)

The reference guide of Brian Kernighan and Dennis Ritchie? Definitely
a must-have!
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Sun, Jan 8, 2017 at 1:51 PM, Jan de Visser <jan@de-visser.net> wrote:

And this is where I think you're wrong, and why conversion would be hard. C
has very few complicated details. I don't think it has any, frankly. It
basically says "If you want your datastructure to outlive a function call,
I'll give you a chunk of memory and you're now responsible for it. Have fun".
That's not complicated: it's two functions, malloc() and free(), basically.

I don't think that that is true in practice. This paper summarizes why
this is the case: https://www.cl.cam.ac.uk/~pes20/cerberus/pldi16.pdf

At the same time, I don't think it would be a good idea to adopt Rust
for Postgres development, and not purely because of our legacy (it
might be interesting as a language for extensions, however). The
contradictory goals of C are what results in the kind of ambiguity
that that paper goes into. C may have contradictory goals, but that
doesn't mean they're the wrong goals, even when considered as a whole.
The culture that C is steeped in still makes a lot of sense for a
system like Postgres.

--
Peter Geoghegan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 09/01/17 11:31, Michael Paquier wrote:

On Mon, Jan 9, 2017 at 6:51 AM, Jan de Visser <jan@de-visser.net> wrote:

Your fear of C in unfounded. The definitive c89 reference is a little book of
about 250 pages, more than half of which is about the standard library of
which you'll never use more than half. If you have some notepaper laying about
on which to scribble pointer diagrams you can be a C programmer to :-)

The reference guide of Brian Kernighan and Dennis Ritchie? Definitely
a must-have!

I learnt C from the original version of K&R, and bought the ANSI version
when that came out - at the time I was a COBOL programmer on a mighty
MainFrame (with a 'massive' 1 MB of Core Memory and a 'fast' 2 MHz
processor). Now I use Java.

Cheers,
Gavin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 9 Jan. 2017 05:51, "Jan de Visser" <jan@de-visser.net> wrote:

On Sunday, January 8, 2017 6:28:17 AM EST Joel Jacobson wrote:

I don't want to learn the complicated details of C, that's true.

And this is where I think you're wrong, and why conversion would be hard. C
has very few complicated details. I don't think it has any, frankly.

Oh, that's really rather optimistic.

C is a small-ish language. But achieving a good understanding of its memory
model and its implications isn't easy at all. There is lots of undefined
behaviour in the language too, which makes it easy to write code that
works... mostly. Usually. Until you hit some edge case, run on a different
architecture/platform, etc.

Then the system libraries and their implementations add complexity. ptheads
may not be part of C but for many projects a solid understanding of them is
crucial... and not that easy.

Do you comprehensively understand the rules for memory ordering when
processes interact in shared memory? Can you explain the correct uses of
volatile and when declaring something volatile is / isn't sufficient for
ensuring safe concurrent access? What happens if you dereference a pointer
to a struct allocated on the stack of a just-returned function?

I had a quick look at Rust and it sounds like it tries to make this sort of
stuff simpler. I didn't see any formal definition of a memory model though
- it seems like it figures you'll just use its concurrency primitives. And
C looks simple enough at first too... emergent complexity from seemingly
simple rules is hard.

It is easy to write C programs of moderate complexity that work reliably
within tested conditions and are somewhat portable to a set of tested-for
architectures. It is very hard to write C that is generally portable,
robust in the face of various edge-case inputs and environmental
conditions, are free from race conditions and memory ordering problems, and
rely on no undefined behaviour.

This is only partly a deficiency of C. Lots of it is down to low level
systems being complex, hard and varied. Weak vs strong memory ordering,
LP64 vs ILP64, etc etc etc.

I know only just enough C to be dangerous. Admittedly I haven't adequately
studied the language, but I have some idea how much I don't know. I doubt
there are a lot of people who can write truly error-free C. But that's also
true of pretty much any language, even ones that purport to be safe.

On 1/8/17 5:06 PM, Craig Ringer wrote:

It is very hard to write C that is generally portable, robust in the
face of various edge-case inputs and environmental conditions, are free
from race conditions and memory ordering problems, and rely on no
undefined behaviour.

BTW, if you s/memory/set/ then that exactly describes building
non-trivial systems on top of relational databases. The devil is always
in the details.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 01/08/2017 09:19 AM, Craig Ringer wrote:

On 8 Jan. 2017 17:10, "Joel Jacobson" <joel@trustly.com
<mailto:joel@trustly.com>> wrote:

Is this completely unrealistic or is it carved in stone PostgreSQL
will always be a C project forever and ever?

Incredibly unrealistic.

[lots of other stuff I agree with]

You'd have a lot more chance writing extensions in Rust though. If you
can make it play OK with the exception handling in postgres and our
memory management, at least. If you really wanted to push this
forward... start there. Show how great it is.

Yeah. A few years ago Tom Dunstan and I started creating a sample
extension in Rust while he was here on a short visit. We ran out of time
so we didn't quite get it finished. Bottom line is it's possible but far
from straightforward. Rust's inbuilt build system makes life, er,
interesting. The fun of getting PG_MODULE_MAGIC in was one of the things
we had to deal with. We ended up using a small amount of C glue.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 8 January 2017 at 21:50, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:

Somewhat related to that... it would be useful if Postgres had "fenced"
functions; functions that ran in a separate process and only talked to a
backend via a well defined API (such as libpq). There's two major advantages
that would give us:

The problem with this is that any of the "interesting" extensions need
to use the server API. That is, they need to be able to do things like
throw errors, expand toast data, etc.

IMHO just about anything you could do in an external process would be
something you could much more easily and conveniently do in the
client. And it would be more flexible and scalable as well as it's a
lot easier to add more clients than it is to scale up the database.

That said, there were several pl language implementations that worked
this way. IIRC one of the Java pl languages ran in a separate Java
process.

I think the solution to the problem you're describing is the project
formerly known as NaCl
https://en.wikipedia.org/wiki/Google_Native_Client

--
greg

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Monday, January 9, 2017 7:06:21 AM EST Craig Ringer wrote:

On 9 Jan. 2017 05:51, "Jan de Visser" <jan@de-visser.net> wrote:

On Sunday, January 8, 2017 6:28:17 AM EST Joel Jacobson wrote:

I don't want to learn the complicated details of C, that's true.

And this is where I think you're wrong, and why conversion would be hard. C
has very few complicated details. I don't think it has any, frankly.

Oh, that's really rather optimistic.

[snip]

Allow me to be snarky and summarize your (very true) points as: "writing
complicated software systems is hard".

That's not the fault of the language (in most cases). The complexity can
somewhat be abstracted by the language, which Rust and Java try to do, or
completely left to the design, as you're forced to do in C. The former gives
you either a more complicated language or severly limit what you can do, and
in the end you will find walls to bang your head against and edges of cliffs to
fall off of.

The latter, of course, makes your head explode if your system is large enough
and you don't have Tom Lane and Robert Haas around.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 1/8/17 5:56 PM, Greg Stark wrote:

On 8 January 2017 at 21:50, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:

Somewhat related to that... it would be useful if Postgres had "fenced"
functions; functions that ran in a separate process and only talked to a
backend via a well defined API (such as libpq). There's two major advantages
that would give us:

The problem with this is that any of the "interesting" extensions need
to use the server API. That is, they need to be able to do things like
throw errors, expand toast data, etc.

There's plenty of interesting things you can do in python or R, even
without that ability.

IMHO just about anything you could do in an external process would be
something you could much more easily and conveniently do in the
client. And it would be more flexible and scalable as well as it's a
lot easier to add more clients than it is to scale up the database.

Well, then you're suffering from serious network latency, and you're
forced into worrying about endian issues and what-not. Those problems
don't exist when you're running on the same server. There's also things
that might make sense on a local-only protocol but would make no sense
with an external one. My guess is that you'd ultimately want a protocol
that's something "in between" SPI and libpq.

That said, there were several pl language implementations that worked
this way. IIRC one of the Java pl languages ran in a separate Java
process.

I think the solution to the problem you're describing is the project
formerly known as NaCl
https://en.wikipedia.org/wiki/Google_Native_Client

Possibly; depends on if it would allow running things like R or python.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

2017-01-09 1:21 GMT+01:00 Jim Nasby <Jim.Nasby@bluetreble.com>:

On 1/8/17 5:56 PM, Greg Stark wrote:

On 8 January 2017 at 21:50, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:

Somewhat related to that... it would be useful if Postgres had "fenced"
functions; functions that ran in a separate process and only talked to a
backend via a well defined API (such as libpq). There's two major
advantages
that would give us:

The problem with this is that any of the "interesting" extensions need
to use the server API. That is, they need to be able to do things like
throw errors, expand toast data, etc.

There's plenty of interesting things you can do in python or R, even
without that ability.

IMHO just about anything you could do in an external process would be

something you could much more easily and conveniently do in the
client. And it would be more flexible and scalable as well as it's a
lot easier to add more clients than it is to scale up the database.

Well, then you're suffering from serious network latency, and you're
forced into worrying about endian issues and what-not. Those problems don't
exist when you're running on the same server. There's also things that
might make sense on a local-only protocol but would make no sense with an
external one. My guess is that you'd ultimately want a protocol that's
something "in between" SPI and libpq.

The running unsafe PL in own managed processes is good idea - Years, I have
a one diploma theme "better management of unsafe PL in Postgres" - but
still without any interest from students :(. I had two possibilities to
see catastrophic errors related to wrong usage of PLPerlu. If we can locks
interpret/environment in some safe sandbox, then it should be great.

Regards

Pavel