PL/pgSQL: EXCEPTION NOSAVEPOINT

Started by Matt Millerover 20 years ago17 messages
#1Matt Miller
mattm@epx.com
1 attachment(s)

This was motivated by the SELECT INTO EXACT discussion at
http://archives.postgresql.org/pgsql-patches/2005-07/msg00559.php.

The idea is to allow a PL/pgSQL exception to not automatically rollback
the work done by the current block. The benefit is that exception
handling can be used as a program flow control technique, without
invoking transaction management mechanisms. This also adds additional
means to enhanced Oracle PL/SQL compatibility.

The patch implements an optional NOSAVEPOINT keyword after the EXCEPTION
keyword that begins the exception handler definition. Here is an
excerpt from the patched documentation:

--------beginning of excerpt-----------------------
If NOSAVEPOINT is not specified then a transaction savepoint is
established immediately prior to the execution of statements. If an
exception is raised then the effects of statements on the database are
rolled back to this savepoint. If NOSAVEPOINT is specified then no
savepoint is established. In this case a handled exception does not roll
back the effects of statements. An unhandled exception, however, will
still propagate out as usual, and any database effects may or may not be
rolled back, depending on the characteristics of the enclosing
block(s).

Tip: Establishing a savepoint can be expensive. If you do not
need the ability rollback the block's effect on the database,
then either use the NOSAVEPOINT option, or avoid the EXCEPTION
clause altogether.
--------end of excerpt-----------------------

Implementation question:

In pl_exec.c the new option causes the "BeginInternalSubTransaction,"
"ReleaseCurrentSubTransaction," and
"RollbackAndReleaseCurrentSubTransaction" function calls to be skipped.
However, the corresponding "MemoryContextSwitchTo" and related calls are
still performed. Should these calls also be dependent on the new
option? Would that be more correct, and/or a performance improvement?

Attachments:

exception_nosavepoint.patchtext/x-patch; charset=UTF-8; name=exception_nosavepoint.patchDownload
Index: doc/src/sgml/plpgsql.sgml
===================================================================
RCS file: /var/local/pgcvs/pgsql/doc/src/sgml/plpgsql.sgml,v
retrieving revision 1.75
diff -c -r1.75 plpgsql.sgml
*** doc/src/sgml/plpgsql.sgml	2 Jul 2005 08:59:47 -0000	1.75
--- doc/src/sgml/plpgsql.sgml	3 Aug 2005 19:42:48 -0000
***************
*** 2086,2092 ****
      <replaceable>declarations</replaceable> </optional>
  BEGIN
      <replaceable>statements</replaceable>
! EXCEPTION
      WHEN <replaceable>condition</replaceable> <optional> OR <replaceable>condition</replaceable> ... </optional> THEN
          <replaceable>handler_statements</replaceable>
      <optional> WHEN <replaceable>condition</replaceable> <optional> OR <replaceable>condition</replaceable> ... </optional> THEN
--- 2086,2092 ----
      <replaceable>declarations</replaceable> </optional>
  BEGIN
      <replaceable>statements</replaceable>
! EXCEPTION <optional>NOSAVEPOINT</optional>
      WHEN <replaceable>condition</replaceable> <optional> OR <replaceable>condition</replaceable> ... </optional> THEN
          <replaceable>handler_statements</replaceable>
      <optional> WHEN <replaceable>condition</replaceable> <optional> OR <replaceable>condition</replaceable> ... </optional> THEN
***************
*** 2104,2117 ****
       processing of the <replaceable>statements</replaceable> is
       abandoned, and control passes to the <literal>EXCEPTION</> list.
       The list is searched for the first <replaceable>condition</replaceable>
!      matching the error that occurred.  If a match is found, the
!      corresponding <replaceable>handler_statements</replaceable> are
!      executed, and then control passes to the next statement after
!      <literal>END</>.  If no match is found, the error propagates out
!      as though the <literal>EXCEPTION</> clause were not there at all:
!      the error can be caught by an enclosing block with
!      <literal>EXCEPTION</>, or if there is none it aborts processing
!      of the function.
      </para>
  
      <para>
--- 2104,2140 ----
       processing of the <replaceable>statements</replaceable> is
       abandoned, and control passes to the <literal>EXCEPTION</> list.
       The list is searched for the first <replaceable>condition</replaceable>
!      matching the error that occurred.  If a match is found, then the
!      exception is considered handled, and the corresponding
!      <replaceable>handler_statements</replaceable> are executed.  Control
!      then passes to the next statement after <literal>END</>.  If no match
!      is found, the unhandled error propagates out as though the
!      <literal>EXCEPTION</> clause were not there at all.  The error can then
!      be caught by an enclosing block with <literal>EXCEPTION</>, or if there
!      is none it aborts processing of the function.
!     </para>
! 
!     <para>
!      If <literal>NOSAVEPOINT</literal> is not specified then a transaction
!      savepoint is established immediately prior to the execution of
!      <replaceable>statements</replaceable>.  If an exception is raised then
!      the effects of <replaceable>statements</replaceable> on the database
!      are rolled back to this savepoint.  If <literal>NOSAVEPOINT</literal>
!      is specified then no savepoint is established.  In this case a handled
!      exception does not roll back the effects of
!      <replaceable>statements</replaceable>.  An unhandled exception,
!      however, will still propagate out as usual, and any database effects may
!      or may not be rolled back, depending on the characteristics of the enclosing
!      block(s).
! 
!     <tip>
!      <para>
!       Establishing a savepoint can be expensive.   If you do not need the
!       ability rollback the block's effect on the database, then either use
!       the <literal>NOSAVEPOINT</> option, or avoid the
!       <literal>EXCEPTION</literal> clause altogether.
!      </para>
!     </tip>
      </para>
  
      <para>
***************
*** 2128,2141 ****
      <para>
       If a new error occurs within the selected
       <replaceable>handler_statements</replaceable>, it cannot be caught
!      by this <literal>EXCEPTION</> clause, but is propagated out.
       A surrounding <literal>EXCEPTION</> clause could catch it.
      </para>
  
      <para>
       When an error is caught by an <literal>EXCEPTION</> clause,
       the local variables of the <application>PL/pgSQL</> function
!      remain as they were when the error occurred, but all changes
       to persistent database state within the block are rolled back.
       As an example, consider this fragment:
  
--- 2151,2165 ----
      <para>
       If a new error occurs within the selected
       <replaceable>handler_statements</replaceable>, it cannot be caught
!      By this <literal>EXCEPTION</> clause, but is propagated out.
       A surrounding <literal>EXCEPTION</> clause could catch it.
      </para>
  
      <para>
       When an error is caught by an <literal>EXCEPTION</> clause,
       the local variables of the <application>PL/pgSQL</> function
!      remain as they were when the error occurred.  However, unless
!      <literal>NOSAVEPOINT</literal> is specified, then all changes
       to persistent database state within the block are rolled back.
       As an example, consider this fragment:
  
***************
*** 2162,2175 ****
       contains <literal>Tom Jones</> not <literal>Joe Jones</>.
      </para>
  
-     <tip>
-      <para>
-       A block containing an <literal>EXCEPTION</> clause is significantly
-       more expensive to enter and exit than a block without one.  Therefore,
-       don't use <literal>EXCEPTION</> without need.
-      </para>
-     </tip>
- 
      <para>
       Within an exception handler, the <varname>SQLSTATE</varname>
       variable contains the error code that corresponds to the
--- 2186,2191 ----
Index: src/pl/plpgsql/src/gram.y
===================================================================
RCS file: /var/local/pgcvs/pgsql/src/pl/plpgsql/src/gram.y,v
retrieving revision 1.80
diff -c -r1.80 gram.y
*** src/pl/plpgsql/src/gram.y	2 Jul 2005 17:01:59 -0000	1.80
--- src/pl/plpgsql/src/gram.y	3 Aug 2005 19:18:13 -0000
***************
*** 186,191 ****
--- 186,192 ----
  %token	K_LOG
  %token	K_LOOP
  %token	K_NEXT
+ %token	K_NOSAVEPOINT
  %token	K_NOT
  %token	K_NOTICE
  %token	K_NULL
***************
*** 1534,1539 ****
--- 1535,1541 ----
  						 */
  						PLpgSQL_exception_block *new = palloc(sizeof(PLpgSQL_exception_block));
  						PLpgSQL_variable *var;
+ 						int tok;
  
  						var = plpgsql_build_variable("sqlstate", $2,
  													 plpgsql_build_datatype(TEXTOID, -1),
***************
*** 1547,1552 ****
--- 1549,1561 ----
  						((PLpgSQL_var *) var)->isconst = true;
  						new->sqlerrm_varno = var->dno;
  
+ 						new->nosavepoint = false;
+ 						tok = yylex();
+ 						if (tok == K_NOSAVEPOINT)
+ 						    new->nosavepoint = true;
+ 						else
+ 						    plpgsql_push_back_token(tok);						
+ 
  						$<exception_block>$ = new;
  					}
  					proc_exceptions
Index: src/pl/plpgsql/src/pl_exec.c
===================================================================
RCS file: /var/local/pgcvs/pgsql/src/pl/plpgsql/src/pl_exec.c,v
retrieving revision 1.151
diff -c -r1.151 pl_exec.c
*** src/pl/plpgsql/src/pl_exec.c	28 Jul 2005 07:51:13 -0000	1.151
--- src/pl/plpgsql/src/pl_exec.c	3 Aug 2005 19:18:13 -0000
***************
*** 842,848 ****
  		MemoryContext oldcontext = CurrentMemoryContext;
  		ResourceOwner oldowner = CurrentResourceOwner;
  
! 		BeginInternalSubTransaction(NULL);
  		/* Want to run statements inside function's memory context */
  		MemoryContextSwitchTo(oldcontext);
  
--- 842,850 ----
  		MemoryContext oldcontext = CurrentMemoryContext;
  		ResourceOwner oldowner = CurrentResourceOwner;
  
! 		if (!block->exceptions->nosavepoint)
! 			BeginInternalSubTransaction(NULL);
! 
  		/* Want to run statements inside function's memory context */
  		MemoryContextSwitchTo(oldcontext);
  
***************
*** 851,857 ****
  			rc = exec_stmts(estate, block->body);
  
  			/* Commit the inner transaction, return to outer xact context */
! 			ReleaseCurrentSubTransaction();
  			MemoryContextSwitchTo(oldcontext);
  			CurrentResourceOwner = oldowner;
  
--- 853,861 ----
  			rc = exec_stmts(estate, block->body);
  
  			/* Commit the inner transaction, return to outer xact context */
! 			if (!block->exceptions->nosavepoint)
! 				ReleaseCurrentSubTransaction();
! 
  			MemoryContextSwitchTo(oldcontext);
  			CurrentResourceOwner = oldowner;
  
***************
*** 872,878 ****
  			FlushErrorState();
  
  			/* Abort the inner transaction */
! 			RollbackAndReleaseCurrentSubTransaction();
  			MemoryContextSwitchTo(oldcontext);
  			CurrentResourceOwner = oldowner;
  
--- 876,884 ----
  			FlushErrorState();
  
  			/* Abort the inner transaction */
! 			if (!block->exceptions->nosavepoint)
! 				RollbackAndReleaseCurrentSubTransaction();
! 
  			MemoryContextSwitchTo(oldcontext);
  			CurrentResourceOwner = oldowner;
  
Index: src/pl/plpgsql/src/plpgsql.h
===================================================================
RCS file: /var/local/pgcvs/pgsql/src/pl/plpgsql/src/plpgsql.h,v
retrieving revision 1.64
diff -c -r1.64 plpgsql.h
*** src/pl/plpgsql/src/plpgsql.h	22 Jun 2005 01:35:02 -0000	1.64
--- src/pl/plpgsql/src/plpgsql.h	3 Aug 2005 19:18:13 -0000
***************
*** 327,332 ****
--- 327,333 ----
  {
  	int			sqlstate_varno;
  	int			sqlerrm_varno;
+ 	bool		nosavepoint;
  	List	   *exc_list;		/* List of WHEN clauses */
  } PLpgSQL_exception_block;
  
Index: src/pl/plpgsql/src/scan.l
===================================================================
RCS file: /var/local/pgcvs/pgsql/src/pl/plpgsql/src/scan.l,v
retrieving revision 1.42
diff -c -r1.42 scan.l
*** src/pl/plpgsql/src/scan.l	26 Jun 2005 19:16:07 -0000	1.42
--- src/pl/plpgsql/src/scan.l	3 Aug 2005 19:18:13 -0000
***************
*** 164,169 ****
--- 164,170 ----
  log				{ return K_LOG;				}
  loop			{ return K_LOOP;			}
  next			{ return K_NEXT;			}
+ nosavepoint		{ return K_NOSAVEPOINT;		}
  not				{ return K_NOT;				}
  notice			{ return K_NOTICE;			}
  null			{ return K_NULL;			}
#2Tom Lane
tgl@sss.pgh.pa.us
In reply to: Matt Miller (#1)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

Matt Miller <mattm@epx.com> writes:

The idea is to allow a PL/pgSQL exception to not automatically rollback
the work done by the current block.

This fundamentally breaks the entire backend. You do not have the
option to continue processing after elog(ERROR); the (sub)transaction
rollback is necessary to clean up inconsistent state.

regards, tom lane

#3Matt Miller
mattm@epx.com
In reply to: Tom Lane (#2)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

On Wed, 2005-08-03 at 16:25 -0400, Tom Lane wrote:

The idea is to allow a PL/pgSQL exception to not automatically
rollback the work done by the current block.

This fundamentally breaks the entire backend.

Yeah, but besides that, can you quick commit this to HEAD so I don't
have to keep track of it locally?

Just kidding.

You do not have the
option to continue processing after elog(ERROR); the (sub)transaction
rollback is necessary to clean up inconsistent state.

Okay, I'll look at this more closely. Can you give me an example of
what can go wrong?

#4Tom Lane
tgl@sss.pgh.pa.us
In reply to: Matt Miller (#3)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

Matt Miller <mattm@epx.com> writes:

On Wed, 2005-08-03 at 16:25 -0400, Tom Lane wrote:

You do not have the
option to continue processing after elog(ERROR); the (sub)transaction
rollback is necessary to clean up inconsistent state.

Okay, I'll look at this more closely. Can you give me an example of
what can go wrong?

Well, for example, failure to release locks and buffer pins held by an
abandoned query. Memory leaks. Row versions inserted into the database
that will be seen as good because they're marked as being generated by
the outer transaction, rather than coming from a subtransaction that can
be separately marked as aborted. Pretty much everything done by
AbortSubTransaction can be seen as cleanup...

The only way you could get the effect you are after would be to run a
new subtransaction for each executed query; which is not impossible
but the overhead would be appalling :-(

regards, tom lane

#5Matt Miller
mattm@epx.com
In reply to: Tom Lane (#2)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

[redirected from -patches]

On Wed, 2005-08-03 at 16:25 -0400, Tom Lane wrote:

Matt Miller <mattm@epx.com> writes:

allow a PL/pgSQL exception to not automatically rollback
the work done by the current block.

This fundamentally breaks the entire backend. You do not have the
option to continue processing after elog(ERROR);

Okay, I think I'm beginning to see the naivete of that patch's
simplistic attempt to decouple backend error handling from transaction
management. But I still haven't found a way to meet my original need:

On Wed, 2005-08-03 at 19:58 +0000, Matt Miller wrote:

The benefit is that [PL/pgSQL] exception
handling can be used as a program flow control technique, without
invoking transaction management mechanisms. This also adds additional
means to enhanced Oracle PL/SQL compatibility.

Basically I'd like my Pl/pgSQL code to be able to utilize the try/catch
paradigm of error handling without the overhead of subtransactions and
without the effect of a rollback. If I catch the exception then
everything should be fine as far as the transaction is concerned. If
don't catch the exception, or if I re-raise it, then the enclosing block
can decide to rollback. This is more consistent with Oracle, and I have
hundreds of Oracle procs to convert across multiple databases.

I'm still thinking that some kind of hack to
src/pl/plpgsql/src/pl_exec.c is probably where I'm headed, but I'm open
to consider other approaches/advice.

#6Tom Lane
tgl@sss.pgh.pa.us
In reply to: Matt Miller (#5)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

[ redirected to -hackers, where it's actually on topic ]

Matt Miller <mattm@epx.com> writes:

[redirected from -patches]
On Wed, 2005-08-03 at 16:25 -0400, Tom Lane wrote:

This fundamentally breaks the entire backend. You do not have the
option to continue processing after elog(ERROR);

Okay, I think I'm beginning to see the naivete of that patch's
simplistic attempt to decouple backend error handling from transaction
management. But I still haven't found a way to meet my original need:

On Wed, 2005-08-03 at 19:58 +0000, Matt Miller wrote:

The benefit is that [PL/pgSQL] exception
handling can be used as a program flow control technique, without
invoking transaction management mechanisms. This also adds additional
means to enhanced Oracle PL/SQL compatibility.

Basically I'd like my Pl/pgSQL code to be able to utilize the try/catch
paradigm of error handling without the overhead of subtransactions and
without the effect of a rollback. If I catch the exception then
everything should be fine as far as the transaction is concerned.

The reason you aren't going to be able to manage this in the current
state of plpgsql is that plpgsql doesn't really have any interesting
computational ability "of its own". It can't even do 2+2 without
calling the main executor --- and recovering from elog(ERROR) without a
transaction rollback is not part of the executor's contract. So while
you could theoretically make a try/catch construct within plpgsql that
doesn't have subtransaction semantics, there'd basically be no way to
do anything useful within it.

You might take a look at the other PLs such as plperl; those have
behavior much closer to what you are looking for, since their
computational engine is separate from the SQL engine.

regards, tom lane

#7Matt Miller
mattm@epx.com
In reply to: Tom Lane (#6)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

On Thu, 2005-09-01 at 18:28 -0400, Tom Lane wrote:

Matt Miller <mattm@epx.com> writes:

Basically I'd like my Pl/pgSQL code to be able to utilize the try/catch
paradigm of error handling without the overhead of subtransactions

[Pl/pgSQL] can't even do 2+2 without
calling the main executor --- and recovering from elog(ERROR) without a
transaction rollback is not part of the executor's contract.

Okay, so that's the crux regarding PL/pgSQL.

You might take a look at the other PLs such as plperl

That would defeat my goal of not rewriting all my Oracle code.

If I were fool enough to plan an attack on the main executor's exception
handling to try and disarm it of its subtransaction semantics, where
would I start? Where would I end? What would I do in between? Can New
Orleans be rebuilt above sea level?

Seriously, though, I'm willing to devote considerable time to this.
Rewriting all my Oracle code function-by-function could be painful, and
I would end up dragging other people around this company into it. I'm
still trying to hold on to my fantasy that I can hack Postgres (and
contrib/ora2pg) into submission. In the end I'm hoping that the move
from Oracle will be made easier for others.

#8Tom Lane
tgl@sss.pgh.pa.us
In reply to: Matt Miller (#7)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

Matt Miller <mattm@epx.com> writes:

If I were fool enough to plan an attack on the main executor's exception
handling to try and disarm it of its subtransaction semantics, where
would I start? Where would I end? What would I do in between? Can New
Orleans be rebuilt above sea level?

In general I don't think it even makes sense to think of making executor
rollback non-transactional. If the executor was running, say, an INSERT
or UPDATE that had already made some database changes before failing,
you certainly don't want those partial results to be seen as good.

ISTM what you are after is to provide some computational capability in
plpgsql that is restricted from doing any database access, and therefore
isn't subject to the above complaint. I'm not sure about a good way to
do this. Seems like your choices are to re-invent the wheel by
implementing a separate expression evaluator inside plpgsql, or to try
to create a locked-down, limited-capability option in execQual.c.
(The main problem with the latter route is to do it without adding any
overhead for normal execution, as otherwise you'll probably get shot
down on that basis.) In either case it's not immediately obvious how
you tell what is safe to allow --- Postgres' model that everything is
embedded within black-box functions doesn't help you here. The
IMMUTABLE/STABLE/VOLATILE marking of functions is close to what you
want, but not close enough.

Also, it might be possible to make the restriction "read-only database
access" instead of "no database access"; this would certainly fit a lot
better with the existing function volatility categories, but then you
still have the problem that aborting the executor is just not a
low-overhead control path. And you would have to go through just about
all of the existing subxact cleanup, such as releasing locks and buffer
pins acquired within the failing query.

regards, tom lane

#9Christopher Kings-Lynne
chriskl@familyhealth.com.au
In reply to: Matt Miller (#7)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

Why don't you just use EnterpriseDB?

Chris

Show quoted text

That would defeat my goal of not rewriting all my Oracle code.

If I were fool enough to plan an attack on the main executor's exception
handling to try and disarm it of its subtransaction semantics, where
would I start? Where would I end? What would I do in between? Can New
Orleans be rebuilt above sea level?

Seriously, though, I'm willing to devote considerable time to this.
Rewriting all my Oracle code function-by-function could be painful, and
I would end up dragging other people around this company into it. I'm
still trying to hold on to my fantasy that I can hack Postgres (and
contrib/ora2pg) into submission. In the end I'm hoping that the move
from Oracle will be made easier for others.

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

#10Matthew Miller
mattm@epx.com
In reply to: Christopher Kings-Lynne (#9)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

In general I don't think it even makes sense to think of making
executor rollback non-transactional.

Agreed.

I would not want to rollback some statements and not others within a
transaction. I would like a complete rollback to happen, but only when
the exception propogates out of the exception block unhandled, not when
the exception is first thrown. Maybe if I could get into the TRY
section of the PG_CATCH()/PG_TRY() construct without an intervening
elog(ERROR) then I'd have a chance ...

Seems like your choices are
...
implementing a separate expression evaluator
...
make the restriction "read-only database access"
...

Thanks for all these ideas. I'm just getting to know the PG code tree,
and I appreciate the guidance.

#11Matthew Miller
mattm@epx.com
In reply to: Matthew Miller (#10)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

if I could get into the TRY section of the PG_CATCH()/PG_TRY()
construct without an intervening elog(ERROR) then I'd have a
chance ...

Sorry, I meant "the CATCH section of the PG_TRY()/PG_CATCH()
construct."

#12Matt Miller
mattm@epx.com
In reply to: Christopher Kings-Lynne (#9)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

Rewriting all my Oracle code function-by-function could be painful
...
I'm still trying to hold on to my fantasy that I can hack Postgres (and
contrib/ora2pg) into submission.

Why don't you just use EnterpriseDB?

I looked at EnterpriseDB a few months ago. The installation errored.
It left stuff in /var/opt, which I consider non-standard for a Red Hat
machine. The whole product just didn't feel clean to me. I admit
that's a pretty limited and subjective evaluation, especially for a beta
product, but I was in the mode of broadly evaluating alternatives, so I
moved on. Maybe I need to look at it again.

Basically I feel more secure tracking the core project, even if I need
to maintain some of my own patches.

#13Alvaro Herrera
alvherre@alvh.no-ip.org
In reply to: Matt Miller (#12)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

On Fri, Sep 02, 2005 at 02:34:50PM +0000, Matt Miller wrote:

Rewriting all my Oracle code function-by-function could be painful
...
I'm still trying to hold on to my fantasy that I can hack Postgres (and
contrib/ora2pg) into submission.

Why don't you just use EnterpriseDB?

I looked at EnterpriseDB a few months ago. The installation errored.
It left stuff in /var/opt, which I consider non-standard for a Red Hat
machine. The whole product just didn't feel clean to me. I admit
that's a pretty limited and subjective evaluation, especially for a beta
product, but I was in the mode of broadly evaluating alternatives, so I
moved on. Maybe I need to look at it again.

Well, the EnterpriseDB has much more support for Oracle syntax in
general, and PL/SQL in particular. If you didn't get past the
installation step, certainly you didn't have a feel of how the real
features work. May I suggest you at least get it working and try to
port your functions to it?

(Disclaimer: while I work for EDB, I haven't had any relationship to the
Oracle-PL/SQL layer yet, nor have I been involved at all in their
commercial offering.)

--
Alvaro Herrera -- Valdivia, Chile Architect, www.EnterpriseDB.com
"Having your biases confirmed independently is how scientific progress is
made, and hence made our great society what it is today" (Mary Gardiner)

#14Christopher Kings-Lynne
chriskl@familyhealth.com.au
In reply to: Matt Miller (#12)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

I looked at EnterpriseDB a few months ago. The installation errored.
It left stuff in /var/opt, which I consider non-standard for a Red Hat
machine. The whole product just didn't feel clean to me. I admit
that's a pretty limited and subjective evaluation, especially for a beta
product, but I was in the mode of broadly evaluating alternatives, so I
moved on. Maybe I need to look at it again.

Basically I feel more secure tracking the core project, even if I need
to maintain some of my own patches.

The EnterpriseDB guys have a final product now, and it's designed to
emulate Oracle as much as possible. I'd prefer that in production than
my own patches :)

Chris

#15Tom Lane
tgl@sss.pgh.pa.us
In reply to: Christopher Kings-Lynne (#14)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes:

The EnterpriseDB guys have a final product now, and it's designed to
emulate Oracle as much as possible.

The question at hand is whether "as much as possible" includes having
reinvented plpgsql's execution engine ... I have not seen their product,
but if they've gotten that far then they've accomplished a heck of a
lot in a very short time ...

regards, tom lane

#16Josh Berkus
josh@agliodbs.com
In reply to: Matt Miller (#7)
Re: PL/pgSQL: EXCEPTION NOSAVEPOINT

Matt,

Seriously, though, I'm willing to devote considerable time to this.
Rewriting all my Oracle code function-by-function could be painful, and
I would end up dragging other people around this company into it.  I'm
still trying to hold on to my fantasy that I can hack Postgres (and
contrib/ora2pg) into submission.  In the end I'm hoping that the move
from Oracle will be made easier for others.

I'm happy to work with you on ora2pg, as long as we can use Perl. Joe
Conway has some useful oracle-table-bulkloading stuff I can probably talk
him out of.

--
--Josh

Josh Berkus
Aglio Database Solutions
San Francisco

#17Matt Miller
mattm@epx.com
In reply to: Josh Berkus (#16)
1 attachment(s)
Re: Ora2Pg (was PL/pgSQL: EXCEPTION NOSAVEPOINT)

On Fri, 2005-09-02 at 12:29 -0700, Josh Berkus wrote:

still trying to hold on to my fantasy that I can hack Postgres (and
contrib/ora2pg) into submission.

I'm happy to work with you on ora2pg

Cool.

It looks like I should have referred to contrib/oracle, not
contrib/ora2pg, but you got my point.

The latest version I found of ora2pg is at
http://www.samse.fr/GPL/ora2pg/ora2pg-3.3.tar.gz This seems to be more
recent than the version at contrib/oracle. For example, this newer
version has tablespace support. Given this as a starting point, I've
made the attached changes. Mostly I've added a few new config options,
but I also made a correction to the existing EXCLUDE option, and I
corrected a couple spelling/English errors along the way.

A big thing that's lacking is conversion for stored procedures and
functions. My initial approach to this was to use Perl to post-process
the PL/SQL code dumped by the export, making it look more like proper
Pl/pgSQL (e.g. VARCHAR2->VARCHAR). I'm no Perl hacker, and when I came
across significant PL/SQL <--> PL/pgSQL differences (e.g. PL/pgSQL
exception == rollback), I added to my approach the idea of hacking
PL/pgSQL to make it look more like PL/SQL. Attacking the problem from
both ends like this, I imagined that Nirvana would be reached somewhere
in the middle.

The beginning of my Perl-based attempt to convert PL/SQL into PL/pgSQL
is a pretty simple stand-alone script. I can send it if you like, but
I'm a Perl newbie, so you can probably do much better. My attempts to
make PL/pgSQL look like PL/SQL have been posted to -hackers and -patches
over the last couple months.

Attachments:

ora2pg.patchtext/x-patch; charset=UTF-8; name=ora2pg.patchDownload
diff -c ora2pg_3.3/ora2pg.conf ora2pg/ora2pg.conf
*** ora2pg_3.3/ora2pg.conf	2004-12-24 16:05:40.000000000 +0000
--- ora2pg/ora2pg.conf	2005-09-02 20:38:48.900376220 +0000
***************
*** 56,61 ****
--- 56,68 ----
  # Value must be a list of table name separated by space.
  #EXCLUDE		OTHER_TABLES
  
+ # Set whether to include invalid functions, procedures, and packages.
+ # Under Oracle's on-the-fly invalidation/recompilation model there
+ # may be any number of objects that have status of 'INVALID' but that
+ # are actually viable.
+ INCLUDE_INVALID 1
+ 
+ 
  # Display table indice and exit program (do not perform any export)
  SHOWTABLEID	0
  
***************
*** 139,148 ****
  # Constraints will be checked at the end of each transaction.
  DEFER_FKEY	0
  
! # If set to 1 replace portable numeric type into PostgreSQL internal type.
  # Oracle data type NUMBER(p,s) is approximatively converted to smallint,
  # integer, bigint, real and float PostgreSQL data type. If you have monetary
  # fields you should preserve the numeric(p,s) PostgreSQL data type if you need
! # very good precision. NUMBER without precision are set to float.
! PG_NUMERIC_TYPE	1
  
--- 146,171 ----
  # Constraints will be checked at the end of each transaction.
  DEFER_FKEY	0
  
! # If set to 1 replace portable numeric type with PostgreSQL internal type.
  # Oracle data type NUMBER(p,s) is approximatively converted to smallint,
  # integer, bigint, real and float PostgreSQL data type. If you have monetary
  # fields you should preserve the numeric(p,s) PostgreSQL data type if you need
! # very good precision (see PG_INTEGER_TYPE). NUMBER without precision are set to
! # float.
! PG_NUMERIC_TYPE	0
! 
! # If set to 1 replace portable numeric type with PostgreSQL internal type,
! # for integers only.  This behaves as PG_NUMERIC_TYPE with respect to
! # Oracle data type NUMBER(p), but preserves exact arithmetic on NUMBER(p,s)
! # columns by converting to PostgreSQL numeric(p,s).  NUMBER without precision
! # maps to "numeric" without precision.
! PG_INTEGER_TYPE	1
! 
! # If set to 1 map Oracle's DATE type to PostgreSQL DATE type.  Oracle DATE type
! # can contain time information, so PostgreSQL "timestamp" should, in general, be
! # used to hold Oracle DATEs.  However, Oracle also supports TIMESTAMP.  Setting
! # PG_DATE_TYPE indicates that Oracle TIMESTAMPs are the only incoming date columns
! # with a time portion that needs to be preserved, and that incoming Oracle DATEs
! # effectively contain only a date portion.
! PG_DATE_TYPE	1
  
diff -c ora2pg_3.3/ora2pg.pl ora2pg/ora2pg.pl
*** ora2pg_3.3/ora2pg.pl	2004-12-24 16:05:40.000000000 +0000
--- ora2pg/ora2pg.pl	2005-07-07 18:01:53.000000000 +0000
***************
*** 40,45 ****
--- 40,46 ----
  	#tables => \@{$Config{'TABLES'}},
  	tables => $Config{'TABLES'},
  	exclude => $Config{'EXCLUDE'},
+ 	include_invalid => $Config{'INCLUDE_INVALID'} || 0,
  	showtableid => $Config{'SHOWTABLEID'} || 0,
  	min => $Config{'MIN'} || 0,
  	max => $Config{'MAX'} || 0,
***************
*** 56,66 ****
  	fkey_deferrable => $Config{'FKEY_DEFERRABLE'} || 0,
  	defer_fkey => $Config{'DEFER_FKEY'} || 0,
  	pg_numeric_type => $Config{'PG_NUMERIC_TYPE'} || 0,
  );
  
  exit 0 if ($Config{'SHOWTABLEID'});
  
! # Mofify export structure if required
  if ($Config{'TYPE'} =~ /^(DATA|COPY)$/) {
  	for my $t (keys %{$Config{'MODIFY_STRUCT'}}) {
  		$schema->modify_struct($t, @{$Config{'MODIFY_STRUCT'}{$t}});
--- 57,69 ----
  	fkey_deferrable => $Config{'FKEY_DEFERRABLE'} || 0,
  	defer_fkey => $Config{'DEFER_FKEY'} || 0,
  	pg_numeric_type => $Config{'PG_NUMERIC_TYPE'} || 0,
+ 	pg_integer_type => $Config{'PG_INTEGER_TYPE'} || 0,
+ 	pg_date_type => $Config{'PG_DATE_TYPE'} || 0,
  );
  
  exit 0 if ($Config{'SHOWTABLEID'});
  
! # Modify export structure if required
  if ($Config{'TYPE'} =~ /^(DATA|COPY)$/) {
  	for my $t (keys %{$Config{'MODIFY_STRUCT'}}) {
  		$schema->modify_struct($t, @{$Config{'MODIFY_STRUCT'}{$t}});
***************
*** 123,129 ****
  					$Config{"skip_\L$_\E"} = 1;
  				}
  			}
! 		} elsif (!grep(/^$var$/i, 'TABLES', 'MODIFY_STRUCT', 'REPLACE_TABLES', 'REPLACE_COLS', 'WHERE')) {
  			$Config{"\U$var\E"} = $val;
  		} elsif ( (uc($var) eq 'TABLES') || (uc($var) eq 'EXCLUDE') ) {
  			push(@{$Config{"\U$var\E"}}, split(/\s+/, $val) );
--- 126,132 ----
  					$Config{"skip_\L$_\E"} = 1;
  				}
  			}
! 		} elsif (!grep(/^$var$/i, 'TABLES', 'MODIFY_STRUCT', 'REPLACE_TABLES', 'REPLACE_COLS', 'WHERE', 'EXCLUDE')) {
  			$Config{"\U$var\E"} = $val;
  		} elsif ( (uc($var) eq 'TABLES') || (uc($var) eq 'EXCLUDE') ) {
  			push(@{$Config{"\U$var\E"}}, split(/\s+/, $val) );
diff -c ora2pg_3.3/Ora2Pg.pm ora2pg/Ora2Pg.pm
*** ora2pg_3.3/Ora2Pg.pm	2005-02-22 17:21:41.000000000 +0000
--- ora2pg/Ora2Pg.pm	2005-07-07 20:46:54.000000000 +0000
***************
*** 271,302 ****
  
  Supported options are:
  
! 	- datasource	: DBD datasource (required)
! 	- user		: DBD user (optional with public access)
! 	- password	: DBD password (optional with public access)
! 	- schema	: Oracle internal schema to extract
! 	- type		: Type of data to extract, can be TABLE,VIEW,GRANT,SEQUENCE,
! 			  TRIGGER,FUNCTION,PROCEDURE,DATA,COPY,PACKAGE,TABLESPACE
! 	- debug		: Print the current state of the parsing
! 	- export_schema	: Export Oracle schema to PostgreSQL 7.3 schema
! 	- tables	: Extract only the given tables (arrayref) and set the extracting order
! 	- exclude	: Exclude the given tables from extract (arrayref)
! 	- showtableid	: Display only the table indice during extraction
! 	- min		: Indice to begin extraction. Default to 0
! 	- max		: Indice to end extraction. Default to 0 mean no limits
! 	- data_limit	: Number max of tuples to return during data extraction (default 0 no limit)
! 	- case_sensitive: Allow to preserve Oracle object name as they are written. Default is not.
! 	- skip_fkeys	: Skip foreign key constraints extraction. Default to 0 (extraction)
! 	- skip_pkeys	: Skip primary keys extraction. Default to 0 (extraction)
! 	- skip_ukeys	: Skip unique column constraints extraction. Default to 0 (extraction)
! 	- skip_indices	: Skip all other index types extraction. Default to 0 (extraction)
! 	- skip_checks	: Skip checks constraints extraction. Default to 0 (extraction)
! 	- bzip2		: Path to the Bzip2 program to compress data export. Default /usr/bin/bzip2
! 	- gen_user_pwd	: When set to 1 replace default password 'change_my_secret' with a random string.
  	- fkey_deferrable: Force foreign key constraints to be exported as deferrable. Default 0: asis.
! 	- defer_fkey	: Force all foreign key constraints to be deferred during data import. Default 0: asis.
! 	- pg_numeric_type: Convert Oracle NUMBER data type to internal PostgreSQL data type instead of use
  			   of the slow numeric(p,s) data type 
  
  
  Attempt that this list should grow a little more because all initialization is
--- 271,307 ----
  
  Supported options are:
  
! 	- datasource	 : DBD datasource (required)
! 	- user		 : DBD user (optional with public access)
! 	- password	 : DBD password (optional with public access)
! 	- schema	 : Oracle internal schema to extract
! 	- type		 : Type of data to extract, can be TABLE,VIEW,GRANT,SEQUENCE,
! 			   TRIGGER,FUNCTION,PROCEDURE,DATA,COPY,PACKAGE,TABLESPACE
! 	- debug		 : Print the current state of the parsing
! 	- export_schema	 : Export Oracle schema to PostgreSQL schema
! 	- tables	 : Extract only the given tables (arrayref) and set the extracting order
! 	- exclude	 : Exclude the given tables from extract (arrayref)
! 	- include_invalid: Include invalid functions/procedures/packages
! 	- showtableid	 : Display only the table indice during extraction
! 	- min		 : Indice to begin extraction. Default to 0
! 	- max		 : Indice to end extraction. Default to 0 mean no limits
! 	- data_limit	 : Number max of tuples to return during data extraction (default 0 no limit)
! 	- case_sensitive : Allow to preserve Oracle object name as they are written. Default is not.
! 	- skip_fkeys	 : Skip foreign key constraints extraction. Default to 0 (extraction)
! 	- skip_pkeys	 : Skip primary keys extraction. Default to 0 (extraction)
! 	- skip_ukeys	 : Skip unique column constraints extraction. Default to 0 (extraction)
! 	- skip_indices	 : Skip all other index types extraction. Default to 0 (extraction)
! 	- skip_checks	 : Skip checks constraints extraction. Default to 0 (extraction)
! 	- bzip2		 : Path to the Bzip2 program to compress data export. Default /usr/bin/bzip2
! 	- gen_user_pwd	 : When set to 1 replace default password 'change_my_secret' with a random string.
  	- fkey_deferrable: Force foreign key constraints to be exported as deferrable. Default 0: asis.
! 	- defer_fkey	 : Force all foreign key constraints to be deferred during data import. Default 0: asis.
! 	- pg_numeric_type: Convert Oracle NUMBER data type to internal PostgreSQL data type instead of using
  			   of the slow numeric(p,s) data type 
+ 	- pg_integer_type: Convert Oracle NUMBER data type to internal PostgreSQL data type instead of using
+ 			   of the slow numeric(p) data type -- integers only
+ 	- pg_date_type   : Convert Oracle DATE data type to PostgreSQL DATE data type instead of using
+ 			   the TIMESTAMP data type for all dates
  
  
  Attempt that this list should grow a little more because all initialization is
***************
*** 529,534 ****
--- 534,541 ----
  
  	$self->{max} = $options{max} || 0;
  
+ 	$self->{include_invalid} = $options{include_invalid} || 0;
+ 
  	$self->{showtableid} = $options{showtableid} || 0;
  
  	$self->{dbh}->{LongReadLen} = 0;
***************
*** 560,565 ****
--- 567,574 ----
  	$self->{fkey_deferrable} = $options{fkey_deferrable} || 0;
  	$self->{defer_fkey} = $options{defer_fkey} || 0;
  	$self->{pg_numeric_type} = $options{pg_numeric_type} || 0;
+ 	$self->{pg_integer_type} = $options{pg_integer_type} || 0;
+ 	$self->{pg_date_type} = $options{pg_date_type} || 0;
  	
  	$self->{type} = $options{type} || 'TABLE';
  
***************
*** 909,915 ****
  	$sql_header .= "--\n";
  	$sql_header .= "-- This program is free software; you can redistribute it and/or modify it under\n";
  	$sql_header .= "-- the same terms as Perl itself.\n\n";
! 	if ($self->{type} ne 'COPY' || $self->{defer_fkey}) {
  		$sql_header .= "BEGIN TRANSACTION;\n\n";
  	}
  
--- 918,924 ----
  	$sql_header .= "--\n";
  	$sql_header .= "-- This program is free software; you can redistribute it and/or modify it under\n";
  	$sql_header .= "-- the same terms as Perl itself.\n\n";
! 	if ($self->{type} ne 'COPY' && $self->{type} ne 'FUNCTION' && $self->{type} ne 'PROCEDURE' || $self->{defer_fkey}) {
  		$sql_header .= "BEGIN TRANSACTION;\n\n";
  	}
  
***************
*** 1145,1156 ****
  			map { s/^.* out //is } @argu;
  			map { $_ = $self->_sql_type(uc($_)) } @argu;
  			$self->{functions}{$fct} =~ /return ([^\s]*) is/is;
! 			$self->{functions}{$fct} = "-- Oracle function declaration, please edit to match PostgreSQL syntax.\n$self->{functions}{$fct}";
! 			if (!$self->{case_sensitive}) {
! 				$sql_output .= "-- PostgreSQL possible function declaration, please edit to match your needs.\nCREATE FUNCTION \L$fct\E(" . join(',', @argu) . ") RETURNS " . $self->_sql_type(uc($1)) . " AS '\n$self->{functions}{$fct}\n' LANGUAGE 'sql'\n\n";
! 			} else {
! 				$sql_output .= "-- PostgreSQL possible function declaration, please edit to match your needs.\nCREATE FUNCTION $fct(" . join(',', @argu) . ") RETURNS " . $self->_sql_type(uc($1)) . " AS '\n$self->{functions}{$fct}\n' LANGUAGE 'sql'\n\n";
! 			}
  		}
  
  		if (!$sql_output) {
--- 1154,1160 ----
  			map { s/^.* out //is } @argu;
  			map { $_ = $self->_sql_type(uc($_)) } @argu;
  			$self->{functions}{$fct} =~ /return ([^\s]*) is/is;
! 			$sql_output .= "\nCREATE\n $self->{functions}{$fct}";
  		}
  
  		if (!$sql_output) {
***************
*** 1681,1687 ****
                  'CHAR' => 'char',
                  'NCHAR' => 'char',
  		# VARCHAR types the limit is 2000 bytes in Oracle 7 and 4000 in Oracle 8.
! 		# PG varchar type has max length iset to 8104 so it should match all needs
                  'VARCHAR' => 'varchar',
                  'NVARCHAR' => 'varchar',
                  'VARCHAR2' => 'varchar',
--- 1685,1691 ----
                  'CHAR' => 'char',
                  'NCHAR' => 'char',
  		# VARCHAR types the limit is 2000 bytes in Oracle 7 and 4000 in Oracle 8.
! 		# PG varchar type has max length set to 8104 so it should match all needs
                  'VARCHAR' => 'varchar',
                  'NVARCHAR' => 'varchar',
                  'VARCHAR2' => 'varchar',
***************
*** 1689,1694 ****
--- 1693,1699 ----
  		# The DATE data type is used to store the date and time information.
  		# Pg type timestamp should match all needs
                  'DATE' => 'timestamp',
+                 'TIMESTAMP(6)' => 'timestamp',
  		# Type LONG is like VARCHAR2 but with up to 2Gb.
  		# PG type text should match all needs or if you want you could use blob
                  'LONG' => 'text', # Character data of variable length
***************
*** 1723,1733 ****
  				# Type VARCHAR(2) must have a given length
  				$len = 1 if (!$len && ($type eq "CHAR"));
                  		return "$TYPE{$type}($len)";
  			} elsif ($type eq "NUMBER") {
  				# This is an integer
  				if (!$scale) {
  					if ($precision) {
! 						if ($self->{pg_numeric_type}) {
  							if ($precision < 5) {
  								return 'smallint';
  							} elsif ($precision < 10) {
--- 1728,1743 ----
  				# Type VARCHAR(2) must have a given length
  				$len = 1 if (!$len && ($type eq "CHAR"));
                  		return "$TYPE{$type}($len)";
+ 			} elsif ($type eq "DATE") {
+ 				 if ($self->{pg_date_type}) {
+ 					return 'date';
+ 				}
+                 		return "$TYPE{$type}";
  			} elsif ($type eq "NUMBER") {
  				# This is an integer
  				if (!$scale) {
  					if ($precision) {
! 						if ($self->{pg_numeric_type} || $self->{pg_integer_type}) {
  							if ($precision < 5) {
  								return 'smallint';
  							} elsif ($precision < 10) {
***************
*** 1738,1744 ****
  						}
  						return "numeric($precision)";
  					} elsif ($self->{pg_numeric_type}) {
! 						# Most of the time interger should be enought?
  						return 'float';
  					}
  				} else {
--- 1748,1754 ----
  						}
  						return "numeric($precision)";
  					} elsif ($self->{pg_numeric_type}) {
! 						# Most of the time integer should be enough?
  						return 'float';
  					}
  				} else {
***************
*** 2268,2274 ****
  	my($self, $type) = @_;
  
  	# Retrieve all indexes 
! 	my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE='$type' AND STATUS='VALID'";
  	if (!$self->{schema}) {
  		$str .= " AND OWNER NOT IN ('SYS','SYSTEM','DBSNMP','OUTLN','PERFSTAT')";
  	} else {
--- 2278,2287 ----
  	my($self, $type) = @_;
  
  	# Retrieve all indexes 
! 	my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE='$type'";
!         if (!$self->{include_invalid}) {
! 		$str .=  " AND STATUS='VALID'";
! 	}
  	if (!$self->{schema}) {
  		$str .= " AND OWNER NOT IN ('SYS','SYSTEM','DBSNMP','OUTLN','PERFSTAT')";
  	} else {
***************
*** 2307,2313 ****
  	my ($self) = @_;
  
  	# Retrieve all indexes 
! 	my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE='PACKAGE' AND STATUS='VALID'";
  	if (!$self->{schema}) {
  		$str .= " AND OWNER NOT IN ('SYS','SYSTEM','DBSNMP','OUTLN','PERFSTAT')";
  	} else {
--- 2320,2329 ----
  	my ($self) = @_;
  
  	# Retrieve all indexes 
! 	my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE='PACKAGE'";
!         if (!$self->{include_invalid}) {
! 		$str .=  " AND STATUS='VALID'";
! 	}
  	if (!$self->{schema}) {
  		$str .= " AND OWNER NOT IN ('SYS','SYSTEM','DBSNMP','OUTLN','PERFSTAT')";
  	} else {