Jsonb transform for pl/python

Started by Anthony Bykovabout 8 years ago30 messages
#1Anthony Bykov
a.bykov@postgrespro.ru
1 attachment(s)

Hi.
I've implemented jsonb transform
(https://www.postgresql.org/docs/9.5/static/sql-createtransform.html)
for pl/python.

1. '{"1":1}'::jsonb is transformed into dict {"1"=>1}, while
'["1",2]'::jsonb is transformed into list(not tuple!) ["1", 2]

2. If there is a numeric value appear in jsonb, it will be transformed
to decimal through string (Numeric->String->Decimal). Not the best
solution, but as far as I understand this is usual practise in
postgresql to serialize Numerics and de-serialize them.

3. Decimal is transformed into jsonb through string
(Decimal->String->Numeric).

An example may also be helpful to understand extension. So, as an
example, function "test" transforms incoming jsonb into python,
transforms it back into jsonb and returns it.

create extension jsonb_plpython2u cascade;

create or replace function test(val jsonb)
returns jsonb
transform for type jsonb
language plpython2u
as $$
return (val);
$$;

select test('{"1":1,"example": null}'::jsonb);

--
Anthony Bykov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attachments:

0001-jsonb_plpython-extension.patchtext/x-patchDownload
diff --git a/contrib/Makefile b/contrib/Makefile
index e84eb67..d6b7170 100644
--- a/contrib/Makefile
+++ b/contrib/Makefile
@@ -82,9 +82,9 @@ ALWAYS_SUBDIRS += hstore_plperl
 endif
 
 ifeq ($(with_python),yes)
-SUBDIRS += hstore_plpython ltree_plpython
+SUBDIRS += hstore_plpython ltree_plpython jsonb_plpython
 else
-ALWAYS_SUBDIRS += hstore_plpython ltree_plpython
+ALWAYS_SUBDIRS += hstore_plpython ltree_plpython jsonb_plpython
 endif
 
 # Missing:
diff --git a/contrib/jsonb_plpython/Makefile b/contrib/jsonb_plpython/Makefile
new file mode 100644
index 0000000..1e34d86
--- /dev/null
+++ b/contrib/jsonb_plpython/Makefile
@@ -0,0 +1,39 @@
+# contrib/jsonb_plpython/Makefile
+
+MODULE_big = jsonb_plpython$(python_majorversion)u
+OBJS = jsonb_plpython.o $(WIN32RES)
+PGFILEDESC = "jsonb_plpython - transform between jsonb and plpythonu"
+
+PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plpython $(python_includespec) -DPLPYTHON_LIBNAME='"plpython$(python_majorversion)"'
+
+EXTENSION = jsonb_plpython$(python_majorversion)u
+DATA = jsonb_plpython$(python_majorversion)u--1.0.sql
+
+REGRESS = jsonb_plpython$(python_majorversion)
+REGRESS_PLPYTHON3_MANGLE := $(REGRESS)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/jsonb_plpython
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+# We must link libpython explicitly
+ifeq ($(PORTNAME), win32)
+# ... see silliness in plpython Makefile ...
+SHLIB_LINK += $(sort $(wildcard ../../src/pl/plpython/libpython*.a))
+else
+rpathdir = $(python_libdir)
+SHLIB_LINK += $(python_libspec) $(python_additional_libs)
+endif
+
+ifeq ($(python_majorversion),2)
+REGRESS_OPTS += --load-extension=plpython2u
+else
+REGRESS_OPTS += --load-extension=plpython3u
+endif
diff --git a/contrib/jsonb_plpython/expected/jsonb_plpython.out b/contrib/jsonb_plpython/expected/jsonb_plpython.out
new file mode 100644
index 0000000..be104af
--- /dev/null
+++ b/contrib/jsonb_plpython/expected/jsonb_plpython.out
@@ -0,0 +1,118 @@
+CREATE EXTENSION jsonb_plpythonu CASCADE;
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+INFO:  [('a', Decimal('1')), ('c', 'NULL')]
+ test1 
+-------
+     2
+(1 row)
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+ test1complex 
+--------------
+            1
+(1 row)
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+ test1arr 
+----------
+        1
+(1 row)
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+SELECT test2arr('[12,1]'::jsonb);
+ test2arr 
+----------
+        2
+(1 row)
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+ test3arr 
+----------
+        2
+(1 row)
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+SELECT test1int('1'::jsonb);
+ test1int 
+----------
+        1
+(1 row)
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+SELECT test1string('"a"'::jsonb);
+ test1string 
+-------------
+           1
+(1 row)
+
+DROP EXTENSION jsonb_plpythonu CASCADE;
+NOTICE:  drop cascades to 7 other objects
+DETAIL:  drop cascades to function test1(jsonb)
+drop cascades to function test1complex(jsonb)
+drop cascades to function test1arr(jsonb)
+drop cascades to function test2arr(jsonb)
+drop cascades to function test3arr(jsonb)
+drop cascades to function test1int(jsonb)
+drop cascades to function test1string(jsonb)
diff --git a/contrib/jsonb_plpython/expected/jsonb_plpython2.out b/contrib/jsonb_plpython/expected/jsonb_plpython2.out
new file mode 100644
index 0000000..80afb2d
--- /dev/null
+++ b/contrib/jsonb_plpython/expected/jsonb_plpython2.out
@@ -0,0 +1,119 @@
+CREATE EXTENSION jsonb_plpython2u CASCADE;
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+INFO:  [('a', Decimal('1')), ('c', 'NULL')]
+ test1 
+-------
+     2
+(1 row)
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+ test1complex 
+--------------
+            1
+(1 row)
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+ test1arr 
+----------
+        1
+(1 row)
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+SELECT test2arr('[12,1]'::jsonb);
+ test2arr 
+----------
+        2
+(1 row)
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+ test3arr 
+----------
+        2
+(1 row)
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+SELECT test1int('1'::jsonb);
+ test1int 
+----------
+        1
+(1 row)
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+SELECT test1string('"a"'::jsonb);
+ test1string 
+-------------
+           1
+(1 row)
+
+DROP EXTENSION plpython2u CASCADE;
+NOTICE:  drop cascades to 8 other objects
+DETAIL:  drop cascades to extension jsonb_plpython2u
+drop cascades to function test1(jsonb)
+drop cascades to function test1complex(jsonb)
+drop cascades to function test1arr(jsonb)
+drop cascades to function test2arr(jsonb)
+drop cascades to function test3arr(jsonb)
+drop cascades to function test1int(jsonb)
+drop cascades to function test1string(jsonb)
diff --git a/contrib/jsonb_plpython/expected/jsonb_plpython3.out b/contrib/jsonb_plpython/expected/jsonb_plpython3.out
new file mode 100644
index 0000000..80202b0
--- /dev/null
+++ b/contrib/jsonb_plpython/expected/jsonb_plpython3.out
@@ -0,0 +1,119 @@
+CREATE EXTENSION jsonb_plpython3u CASCADE;
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+INFO:  [('a', Decimal('1')), ('c', 'NULL')]
+ test1 
+-------
+     2
+(1 row)
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+ test1complex 
+--------------
+            1
+(1 row)
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+ test1arr 
+----------
+        1
+(1 row)
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+SELECT test2arr('[12,1]'::jsonb);
+ test2arr 
+----------
+        2
+(1 row)
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+ test3arr 
+----------
+        2
+(1 row)
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+SELECT test1int('1'::jsonb);
+ test1int 
+----------
+        1
+(1 row)
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+SELECT test1string('"a"'::jsonb);
+ test1string 
+-------------
+           1
+(1 row)
+
+DROP EXTENSION plpython3u CASCADE;
+NOTICE:  drop cascades to 8 other objects
+DETAIL:  drop cascades to extension jsonb_plpython3u
+drop cascades to function test1(jsonb)
+drop cascades to function test1complex(jsonb)
+drop cascades to function test1arr(jsonb)
+drop cascades to function test2arr(jsonb)
+drop cascades to function test3arr(jsonb)
+drop cascades to function test1int(jsonb)
+drop cascades to function test1string(jsonb)
diff --git a/contrib/jsonb_plpython/jsonb_plpython.c b/contrib/jsonb_plpython/jsonb_plpython.c
new file mode 100644
index 0000000..97960ac
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpython.c
@@ -0,0 +1,417 @@
+/* This document contains an implementation of transformations from python
+ * object to jsonb and vise versa.
+ * In this file you can find implementation of transformations:
+ * - JsonbValue transformation in  PyObject_FromJsonbValue
+ * - JsonbContainer(jsonb) transformation in PyObject_FromJsonb
+ * - PyMapping object(dict) transformation in PyMapping_ToJsonbValue
+ * - PyString object transformation in PyString_ToJsonbValue
+ * - PySequence object(list) transformation in PySequence_ToJsonbValue
+ * - PyNumeric object transformation in PyNumeric_ToJsonbValue
+ * - PyMapping object transformation in PyObject_ToJsonbValue
+ * */
+#include "postgres.h"
+
+#include "plpython.h"
+#include "plpy_typeio.h"
+
+#include "utils/jsonb.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+extern void _PG_init(void);
+
+/* Linkage to functions in plpython module */
+typedef char *(*PLyObject_AsString_t) (PyObject *plrv);
+static PLyObject_AsString_t PLyObject_AsString_p;
+#if PY_MAJOR_VERSION >= 3
+typedef PyObject *(*PLyUnicode_FromStringAndSize_t) (const char *s, Py_ssize_t size);
+static PLyUnicode_FromStringAndSize_t PLyUnicode_FromStringAndSize_p;
+#endif
+
+/*
+ * Module initialize function: fetch function pointers for cross-module calls.
+ */
+void
+_PG_init(void)
+{
+	/* Asserts verify that typedefs above match original declarations */
+	AssertVariableIsOfType(&PLyObject_AsString, PLyObject_AsString_t);
+	PLyObject_AsString_p = (PLyObject_AsString_t)
+		load_external_function("$libdir/" PLPYTHON_LIBNAME, "PLyObject_AsString",
+							   true, NULL);
+#if PY_MAJOR_VERSION >= 3
+	AssertVariableIsOfType(&PLyUnicode_FromStringAndSize, PLyUnicode_FromStringAndSize_t);
+	PLyUnicode_FromStringAndSize_p = (PLyUnicode_FromStringAndSize_t)
+		load_external_function("$libdir/" PLPYTHON_LIBNAME, "PLyUnicode_FromStringAndSize",
+							   true, NULL);
+#endif
+}
+
+
+/* These defines must be after the module init function */
+#define PLyObject_AsString PLyObject_AsString_p
+#define PLyUnicode_FromStringAndSize PLyUnicode_FromStringAndSize_p
+
+/*
+ * decimal_constructor is a link to Python library
+ * for transforming strings into python decimal type
+ * */
+static PyObject *decimal_constructor;
+
+static PyObject *PyObject_FromJsonb(JsonbContainer *jsonb);
+static JsonbValue *PyObject_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state);
+
+/*
+ * PyObject_FromJsonbValue(JsonsValue *jsonbValue)
+ * Function for transforming JsonbValue type into Python Object
+ * The first argument defines the JsonbValue which will be transformed into PyObject
+ * Return value is the pointer to Jsonb structure containing the transformed object.
+ * */
+static PyObject *
+PyObject_FromJsonbValue(JsonbValue *jsonbValue)
+{
+	PyObject   *result;
+	char	   *str;
+
+	switch (jsonbValue->type)
+	{
+		case jbvNull:
+			result = Py_None;
+			break;
+		case jbvBinary:
+			result = PyObject_FromJsonb(jsonbValue->val.binary.data);
+			break;
+		case jbvNumeric:
+
+			/*
+			 * XXX There should be a better way. Right now Numeric is
+			 * transformed into string and then this string is parsed into py
+			 * numeric
+			 */
+			str = DatumGetCString(
+								  DirectFunctionCall1(numeric_out, NumericGetDatum(jsonbValue->val.numeric))
+				);
+			result = PyObject_CallFunction(decimal_constructor, "s", str);
+			break;
+		case jbvString:
+			result = PyString_FromStringAndSize(
+												jsonbValue->val.string.val,
+												jsonbValue->val.string.len
+				);
+			break;
+		case jbvBool:
+			result = jsonbValue->val.boolean ? Py_True : Py_False;
+			break;
+		case jbvArray:
+		case jbvObject:
+			result = PyObject_FromJsonb(jsonbValue->val.binary.data);
+			break;
+	}
+	return (result);
+}
+
+/*
+ * PyObject_FromJsonb(JsonbContainer *jsonb)
+ * Function for transforming JsonbContainer(jsonb) into PyObject
+ * The first argument should represent the data for transformation.
+ * Return value is the pointer to Python object.
+ * */
+
+static PyObject *
+PyObject_FromJsonb(JsonbContainer *jsonb)
+{
+	PyObject   *object = Py_None;
+	JsonbIterator *it;
+	JsonbIteratorToken r;
+	JsonbValue	v;
+
+	object = PyDict_New();
+	it = JsonbIteratorInit(jsonb);
+
+	/*
+	 * Iterate trhrough Jsonb object.
+	 */
+	while ((r = JsonbIteratorNext(&it, &v, true)) != WJB_DONE)
+	{
+		PyObject   *key = Py_None;
+		PyObject   *value = Py_None;
+
+		switch (r)
+		{
+			case (WJB_KEY):
+				/* dict key in v */
+				key = PyString_FromStringAndSize(
+												 v.val.string.val,
+												 v.val.string.len
+					);
+
+				r = JsonbIteratorNext(&it, &v, true);
+				value = PyObject_FromJsonbValue(&v);
+				PyDict_SetItem(object, key, value);
+				break;
+			case (WJB_BEGIN_ARRAY):
+				/* array in v */
+				object = PyList_New(0);
+				while (
+					   ((r = JsonbIteratorNext(&it, &v, true)) == WJB_ELEM)
+					   && (r != WJB_DONE)
+					)
+					PyList_Append(object, PyObject_FromJsonbValue(&v));
+				return (object);
+				break;
+			case (WJB_END_OBJECT):
+			case (WJB_BEGIN_OBJECT):
+				/* no object are in v */
+				break;
+			default:
+				/* simple objects */
+				object = PyObject_FromJsonbValue(&v);
+				break;
+		}
+		Py_XDECREF(value);
+		Py_XDECREF(key);
+	}
+	return (object);
+}
+
+
+/*
+ * jsonb_to_plpython(Jsonb *in)
+ * Function to transform jsonb object to corresponding python object.
+ * The first argument is the Jsonb object to be transformed.
+ * Return value is the pointer to Python object.
+ * */
+PG_FUNCTION_INFO_V1(jsonb_to_plpython);
+Datum
+jsonb_to_plpython(PG_FUNCTION_ARGS)
+{
+	Jsonb	   *in;
+	PyObject   *dict;
+	PyObject   *decimal_module;
+
+	in = PG_GETARG_JSONB_P(0);
+
+	/* Import python cdecimal library and if there is no cdecimal library, */
+	/* import decimal library */
+	if (!decimal_constructor)
+	{
+		decimal_module = PyImport_ImportModule("cdecimal");
+		if (!decimal_module)
+		{
+			PyErr_Clear();
+			decimal_module = PyImport_ImportModule("decimal");
+		}
+		decimal_constructor = PyObject_GetAttrString(decimal_module, "Decimal");
+	}
+
+	dict = PyObject_FromJsonb(&in->root);
+	return PointerGetDatum(dict);
+}
+
+
+/*
+ * PyMapping_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+ * Function to transform Python lists to jsonbValue
+ * The first argument is the python object to be transformed.
+ * Return value is the pointer to JsonbValue structure containing the list.
+ * */
+static JsonbValue *
+PyMapping_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+{
+	volatile PyObject *items_v = NULL;
+	int32		pcount;
+	JsonbValue *out = NULL;
+
+	pcount = PyMapping_Size(obj);
+	items_v = PyMapping_Items(obj);
+
+	PG_TRY();
+	{
+		int32		i;
+		PyObject   *items;
+		JsonbValue *jbvValue;
+		JsonbValue	jbvKey;
+
+		items = (PyObject *) items_v;
+		pushJsonbValue(&jsonb_state, WJB_BEGIN_OBJECT, NULL);
+
+		for (i = 0; i < pcount; i++)
+		{
+			PyObject   *tuple;
+			PyObject   *key;
+			PyObject   *value;
+
+			tuple = PyList_GetItem(items, i);
+			key = PyTuple_GetItem(tuple, 0);
+			value = PyTuple_GetItem(tuple, 1);
+
+			if (key == Py_None)
+			{
+				jbvKey.type = jbvString;
+				jbvKey.val.string.len = 0;
+				jbvKey.val.string.val = "";
+			}
+			else
+			{
+				jbvKey.type = jbvString;
+				jbvKey.val.string.val = PLyObject_AsString(key);
+				jbvKey.val.string.len = strlen(jbvKey.val.string.val);
+			}
+			pushJsonbValue(&jsonb_state, WJB_KEY, &jbvKey);
+			jbvValue = PyObject_ToJsonbValue(value, jsonb_state);
+			if (IsAJsonbScalar(jbvValue))
+				pushJsonbValue(&jsonb_state, WJB_VALUE, jbvValue);
+		}
+		out = pushJsonbValue(&jsonb_state, WJB_END_OBJECT, NULL);
+	}
+	PG_CATCH();
+	{
+		Py_DECREF(items_v);
+		PG_RE_THROW();
+	}
+	PG_END_TRY();
+	return (out);
+}
+
+/*
+ * PyString_ToJsonbValue(PyObject *obj)
+ * Function to transform python string object to jsonbValue object.
+ * The first argument is the Python String object to be transformed.
+ * Return value is the pointer to JsonbValue structure containing the String.
+ * */
+static JsonbValue *
+PyString_ToJsonbValue(PyObject *obj)
+{
+	JsonbValue *out = NULL;
+	JsonbValue *jbvElem;
+
+	jbvElem = palloc(sizeof(JsonbValue));
+	jbvElem->type = jbvString;
+	jbvElem->val.string.val = PLyObject_AsString(obj);
+	jbvElem->val.string.len = strlen(jbvElem->val.string.val);
+	out = jbvElem;
+
+	return (out);
+}
+
+/*
+ * PySequence_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+ * Function to transform python lists to jsonbValue object.
+ * The first argument is the Python list to be transformed.
+ * The second one is conversion state.
+ * Return value is the pointer to JsonbValue structure containing array.
+ * */
+static JsonbValue *
+PySequence_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+{
+	JsonbValue *jbvElem;
+	JsonbValue *out = NULL;
+	int32		pcount;
+	int32		i;
+
+	pcount = PySequence_Size(obj);
+
+
+	pushJsonbValue(&jsonb_state, WJB_BEGIN_ARRAY, NULL);
+
+	for (i = 0; i < pcount; i++)
+	{
+		PyObject   *value;
+
+		value = PySequence_GetItem(obj, i);
+		jbvElem = PyObject_ToJsonbValue(value, jsonb_state);
+		if (IsAJsonbScalar(jbvElem))
+			pushJsonbValue(&jsonb_state, WJB_ELEM, jbvElem);
+	}
+	out = pushJsonbValue(&jsonb_state, WJB_END_ARRAY, NULL);
+	return (out);
+}
+
+/*
+ * PyNumeric_ToJsonbValue(PyObject *obj)
+ * Function to transform python numerics to jsonbValue object.
+ * The first argument is the Python numeric object to be transformed.
+ * Return value is the pointer to JsonbValue structure containing the String.
+ * */
+static JsonbValue *
+PyNumeric_ToJsonbValue(PyObject *obj)
+{
+	JsonbValue *out = NULL;
+	JsonbValue *jbvInt;
+
+	jbvInt = palloc(sizeof(JsonbValue));
+	jbvInt->type = jbvNumeric;
+	jbvInt->val.numeric = DatumGetNumeric(DirectFunctionCall1(
+															  numeric_in,
+															  CStringGetDatum(PLyObject_AsString(obj))
+															  ));
+	out = jbvInt;
+	return (out);
+}
+
+/*
+ * PyObject_ToJsonbValue(PyObject *obj)
+ * Function to transform python objects to jsonbValue object.
+ * The first argument is the Python object to be transformed.
+ * Return value is the pointer to JsonbValue structure containing the transformed object.
+ * */
+static JsonbValue *
+PyObject_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+{
+	JsonbValue *out = NULL;
+
+	if (PyMapping_Check(obj))
+	{
+		/* DICT */
+		out = PyMapping_ToJsonbValue(obj, jsonb_state);
+	}
+	else if (PyString_Check(obj))
+	{
+		/* STRING */
+		out = PyString_ToJsonbValue(obj);
+	}
+	else if (PySequence_Check(obj))
+	{
+		/* LIST or STRING */
+		/* but we have checked on STRING */
+		out = PySequence_ToJsonbValue(obj, jsonb_state);
+	}
+	else if (PyNumber_Check(obj))
+	{
+		/* NUMERIC */
+		out = PyNumeric_ToJsonbValue(obj);
+	}
+	else
+	{
+		/* EVERYTHING ELSE */
+		/* Handle it as it's repr */
+		JsonbValue *jbvElem;
+
+		jbvElem = palloc(sizeof(JsonbValue));
+		jbvElem->type = jbvString;
+		jbvElem->val.string.val = PLyObject_AsString(obj);
+		jbvElem->val.string.len = strlen(jbvElem->val.string.val);
+		out = jbvElem;
+	}
+	return (out);
+}
+
+/*
+ * plpython_to_jsonb(PyObject *obj)
+ * Function to transform python objects to jsonb object.
+ * The first argument is the Python object to be transformed.
+ * Return value is the pointer to Jsonb structure containing the transformed object.
+ * */
+PG_FUNCTION_INFO_V1(plpython_to_jsonb);
+Datum
+plpython_to_jsonb(PG_FUNCTION_ARGS)
+{
+	PyObject   *obj;
+	JsonbValue *out;
+	JsonbParseState *jsonb_state = NULL;
+
+	obj = (PyObject *) PG_GETARG_POINTER(0);
+	out = PyObject_ToJsonbValue(obj, jsonb_state);
+	PG_RETURN_POINTER(JsonbValueToJsonb(out));
+}
diff --git a/contrib/jsonb_plpython/jsonb_plpython2u--1.0.sql b/contrib/jsonb_plpython/jsonb_plpython2u--1.0.sql
new file mode 100644
index 0000000..1e38847
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpython2u--1.0.sql
@@ -0,0 +1,19 @@
+/* contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION jsonb_plpython" to load this file. \quit
+
+CREATE FUNCTION jsonb_to_plpython(val internal) RETURNS internal
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE FUNCTION plpython_to_jsonb(val internal) RETURNS jsonb
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE TRANSFORM FOR jsonb LANGUAGE plpython2u (
+	FROM SQL WITH FUNCTION jsonb_to_plpython(internal),
+	TO SQL WITH FUNCTION plpython_to_jsonb(internal)
+);
+
+COMMENT ON TRANSFORM FOR jsonb LANGUAGE plpython2u IS 'transform between jsonb and Python dict';
diff --git a/contrib/jsonb_plpython/jsonb_plpython2u.control b/contrib/jsonb_plpython/jsonb_plpython2u.control
new file mode 100644
index 0000000..3514cbc
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpython2u.control
@@ -0,0 +1,6 @@
+# jsonb_plpython2u extension
+comment = 'transform between jsonb and plpythonu'
+default_version = '1.0'
+module_pathname = '$libdir/jsonb_plpython2u'
+relocatable = true
+requires = 'plpython2u'
diff --git a/contrib/jsonb_plpython/jsonb_plpython3u--1.0.sql b/contrib/jsonb_plpython/jsonb_plpython3u--1.0.sql
new file mode 100644
index 0000000..0958db7
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpython3u--1.0.sql
@@ -0,0 +1,19 @@
+/* contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION jsonb_plpython3" to load this file. \quit
+
+CREATE FUNCTION jsonb_to_plpython(val internal) RETURNS internal
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE FUNCTION plpython_to_jsonb(val internal) RETURNS jsonb
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE TRANSFORM FOR jsonb LANGUAGE plpython3u (
+	FROM SQL WITH FUNCTION jsonb_to_plpython(internal),
+	TO SQL WITH FUNCTION plpython_to_jsonb(internal)
+);
+
+COMMENT ON TRANSFORM FOR jsonb LANGUAGE plpython3u IS 'transform between jsonb and Python dict';
diff --git a/contrib/jsonb_plpython/jsonb_plpython3u.control b/contrib/jsonb_plpython/jsonb_plpython3u.control
new file mode 100644
index 0000000..bbf6ed7
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpython3u.control
@@ -0,0 +1,6 @@
+# jsonb_plpython3u extension
+comment = 'transform between jsonb and plpythonu'
+default_version = '1.0'
+module_pathname = '$libdir/jsonb_plpython3'
+relocatable = false
+requires = 'plpython3u'
diff --git a/contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql b/contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql
new file mode 100644
index 0000000..7c9460d
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql
@@ -0,0 +1,19 @@
+/* contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION jsonb_plpython" to load this file. \quit
+
+CREATE FUNCTION jsonb_to_plpython(val internal) RETURNS internal
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE FUNCTION plpython_to_jsonb(val internal) RETURNS jsonb
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE TRANSFORM FOR jsonb LANGUAGE plpythonu (
+	FROM SQL WITH FUNCTION jsonb_to_plpython(internal),
+	TO SQL WITH FUNCTION plpython_to_jsonb(internal)
+);
+
+COMMENT ON TRANSFORM FOR jsonb LANGUAGE plpythonu IS 'transform between jsonb and Python dict';
diff --git a/contrib/jsonb_plpython/jsonb_plpythonu.control b/contrib/jsonb_plpython/jsonb_plpythonu.control
new file mode 100644
index 0000000..d696dfb
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpythonu.control
@@ -0,0 +1,6 @@
+# jsonb_plpython2u extension
+comment = 'transform between jsonb and plpythonu'
+default_version = '1.0'
+module_pathname = '$libdir/jsonb_plpython'
+relocatable = true
+requires = 'plpython2u'
diff --git a/contrib/jsonb_plpython/sql/jsonb_plpython.sql b/contrib/jsonb_plpython/sql/jsonb_plpython.sql
new file mode 100644
index 0000000..dda714e
--- /dev/null
+++ b/contrib/jsonb_plpython/sql/jsonb_plpython.sql
@@ -0,0 +1,91 @@
+CREATE EXTENSION jsonb_plpythonu CASCADE;
+
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+
+SELECT test2arr('[12,1]'::jsonb);
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+
+SELECT test1int('1'::jsonb);
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+
+SELECT test1string('"a"'::jsonb);
+
+
+DROP EXTENSION plpython cascade;
diff --git a/contrib/jsonb_plpython/sql/jsonb_plpython2.sql b/contrib/jsonb_plpython/sql/jsonb_plpython2.sql
new file mode 100644
index 0000000..96478a2
--- /dev/null
+++ b/contrib/jsonb_plpython/sql/jsonb_plpython2.sql
@@ -0,0 +1,91 @@
+CREATE EXTENSION jsonb_plpython2u CASCADE;
+
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+
+SELECT test2arr('[12,1]'::jsonb);
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+
+SELECT test1int('1'::jsonb);
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+
+SELECT test1string('"a"'::jsonb);
+
+
+DROP EXTENSION plpython2u CASCADE;
diff --git a/contrib/jsonb_plpython/sql/jsonb_plpython3.sql b/contrib/jsonb_plpython/sql/jsonb_plpython3.sql
new file mode 100644
index 0000000..80e22a1
--- /dev/null
+++ b/contrib/jsonb_plpython/sql/jsonb_plpython3.sql
@@ -0,0 +1,91 @@
+CREATE EXTENSION jsonb_plpython3u CASCADE;
+
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+
+SELECT test2arr('[12,1]'::jsonb);
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+
+SELECT test1int('1'::jsonb);
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+
+SELECT test1string('"a"'::jsonb);
+
+
+DROP EXTENSION plpython3u CASCADE;
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index 3cf78d6..66d307c 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -569,4 +569,20 @@ SELECT jdoc-&gt;'guid', jdoc-&gt;'name' FROM api WHERE jdoc @&gt; '{"tags": ["qu
       compared using the default database collation.
   </para>
  </sect2>
+  <sect2>
+  <title>Transforms</title>
+
+  <para>
+   Additional extensions are available that implement transforms for
+   the <type>jsonb</type> type for the language PL/Python.  The
+   extensions for PL/Perl are called
+   <literal>jsonb_plpythonu</literal>, <literal>jsonb_plpython2u</literal>,
+   and <literal>jsonb_plpython3u</literal>
+   (see <xref linkend="plpython-python23"> for the PL/Python naming
+   convention).  If you use them, <type>jsonb</type> values are mapped to
+   Python dictionaries.
+  </para>
+ </sect2>
+
+
 </sect1>
#2David Fetter
david@fetter.org
In reply to: Anthony Bykov (#1)
Re: Jsonb transform for pl/python

On Wed, Oct 25, 2017 at 02:51:00PM +0300, Anthony Bykov wrote:

Hi.
I've implemented jsonb transform
(https://www.postgresql.org/docs/9.5/static/sql-createtransform.html)
for pl/python.

1. '{"1":1}'::jsonb is transformed into dict {"1"=>1}, while
'["1",2]'::jsonb is transformed into list(not tuple!) ["1", 2]

2. If there is a numeric value appear in jsonb, it will be transformed
to decimal through string (Numeric->String->Decimal). Not the best
solution, but as far as I understand this is usual practise in
postgresql to serialize Numerics and de-serialize them.

3. Decimal is transformed into jsonb through string
(Decimal->String->Numeric).

An example may also be helpful to understand extension. So, as an
example, function "test" transforms incoming jsonb into python,
transforms it back into jsonb and returns it.

create extension jsonb_plpython2u cascade;

Thanks for your hard work!

Should there also be one for PL/Python3U?

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#3Anthony Bykov
a.bykov@postgrespro.ru
In reply to: David Fetter (#2)
Re: Jsonb transform for pl/python

On Sun, 29 Oct 2017 19:11:02 +0100
David Fetter <david@fetter.org> wrote:

Thanks for your hard work!

Should there also be one for PL/Python3U?

Best,
David.

Hi.
Actually, there is one for PL/Python3U. This patch contains following
extensions:
jsonb_plpythonu
jsonb_plpython2u
jsonb_plpython3u
"make install" checks which python major version was your postgresql
configured with and installs corresponding extension.

--
Anthony Bykov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#4David Fetter
david@fetter.org
In reply to: Anthony Bykov (#3)
Re: Jsonb transform for pl/python

On Mon, Oct 30, 2017 at 11:15:00AM +0300, Anthony Bykov wrote:

On Sun, 29 Oct 2017 19:11:02 +0100
David Fetter <david@fetter.org> wrote:

Thanks for your hard work!

Should there also be one for PL/Python3U?

Best,
David.

Hi.
Actually, there is one for PL/Python3U. This patch contains following
extensions:
jsonb_plpythonu
jsonb_plpython2u
jsonb_plpython3u
"make install" checks which python major version was your postgresql
configured with and installs corresponding extension.

My mistake. Sorry about the noise.

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#5Aleksander Alekseev
a.alekseev@postgrespro.ru
In reply to: Anthony Bykov (#1)
Re: Jsonb transform for pl/python

The following review has been posted through the commitfest application:
make installcheck-world: tested, failed
Implements feature: tested, passed
Spec compliant: tested, passed
Documentation: tested, passed

Hello Anthony,

Great job!

I decided to take a closer look on your patch. Here are some defects I
discovered.

+   Additional extensions are available that implement transforms for
+   the <type>jsonb</type> type for the language PL/Python.  The
+   extensions for PL/Perl are called

1. The part regarding PL/Perl is obviously from another patch.

2. jsonb_plpython2u and jsonb_plpythonu are marked as relocatable, while
jsonb_plpython3u is not. Is it a mistake? Anyway if an extension is relocatable
there should be a test that checks this.

3. Not all json types are test-covered. Tests for 'true' :: jsonb, '3.14' ::
jsonb and 'null' :: jsonb are missing.

4. jsonb_plpython.c:133 - "Iterate trhrough Jsonb object." Typo, it should be
"through" or probably even "over".

5. It looks like you've implemented transform in two directions Python <->
JSONB, however I see tests only for Python <- JSONB case.

6. Tests passed on Python 2.7.14 but failed on 3.6.2:

CREATE EXTENSION jsonb_plpython3u CASCADE;
+ ERROR: could not access file "$libdir/jsonb_plpython3": No such file or
directory

module_pathname in jsonb_plpython3u.control should be $libdir/jsonb_plpython3u,
not $libdir/jsonb_plpython3.

Tested on Arch Linux x64, GCC 7.2.0.

The new status of this patch is: Waiting on Author

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#6Anthony Bykov
a.bykov@postgrespro.ru
In reply to: Aleksander Alekseev (#5)
1 attachment(s)
Re: Jsonb transform for pl/python

On Thu, 09 Nov 2017 12:26:46 +0000
Aleksander Alekseev <a.alekseev@postgrespro.ru> wrote:

The following review has been posted through the commitfest
application: make installcheck-world: tested, failed
Implements feature: tested, passed
Spec compliant: tested, passed
Documentation: tested, passed

Hello Anthony,

Great job!

I decided to take a closer look on your patch. Here are some defects
I discovered.

+   Additional extensions are available that implement transforms
for
+   the <type>jsonb</type> type for the language PL/Python.  The
+   extensions for PL/Perl are called  

1. The part regarding PL/Perl is obviously from another patch.

2. jsonb_plpython2u and jsonb_plpythonu are marked as relocatable,
while jsonb_plpython3u is not. Is it a mistake? Anyway if an
extension is relocatable there should be a test that checks this.

3. Not all json types are test-covered. Tests for 'true' :: jsonb,
'3.14' :: jsonb and 'null' :: jsonb are missing.

4. jsonb_plpython.c:133 - "Iterate trhrough Jsonb object." Typo, it
should be "through" or probably even "over".

5. It looks like you've implemented transform in two directions
Python <-> JSONB, however I see tests only for Python <- JSONB case.

6. Tests passed on Python 2.7.14 but failed on 3.6.2:

CREATE EXTENSION jsonb_plpython3u CASCADE;
+ ERROR: could not access file "$libdir/jsonb_plpython3": No such
file or directory

module_pathname in jsonb_plpython3u.control should be
$libdir/jsonb_plpython3u, not $libdir/jsonb_plpython3.

Tested on Arch Linux x64, GCC 7.2.0.

The new status of this patch is: Waiting on Author

Hello, Aleksander.
Thank you for your time. The defects you have noticed were fixed.
Please, find in attachments new version of the patch (it is called
0001-jsonb_plpython-extension-v2.patch).

Most of changes were made to fix defects(list of the defects may be
found in citation in the beginning of this message), but the algorithm
of iterating through incoming jsonb was changed so that it looks tidier.

--
Anthony Bykov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attachments:

0001-jsonb_plpython-extension-v2.patchtext/x-patchDownload
diff --git a/contrib/Makefile b/contrib/Makefile
index 8046ca4..d9d9817 100644
--- a/contrib/Makefile
+++ b/contrib/Makefile
@@ -81,9 +81,9 @@ ALWAYS_SUBDIRS += hstore_plperl
 endif
 
 ifeq ($(with_python),yes)
-SUBDIRS += hstore_plpython ltree_plpython
+SUBDIRS += hstore_plpython ltree_plpython jsonb_plpython
 else
-ALWAYS_SUBDIRS += hstore_plpython ltree_plpython
+ALWAYS_SUBDIRS += hstore_plpython ltree_plpython jsonb_plpython
 endif
 
 # Missing:
diff --git a/contrib/jsonb_plpython/Makefile b/contrib/jsonb_plpython/Makefile
new file mode 100644
index 0000000..6371d11
--- /dev/null
+++ b/contrib/jsonb_plpython/Makefile
@@ -0,0 +1,39 @@
+# contrib/jsonb_plpython/Makefile
+
+MODULE_big = jsonb_plpython$(python_majorversion)u
+OBJS = jsonb_plpython.o $(WIN32RES)
+PGFILEDESC = "jsonb_plpython - transform between jsonb and plpythonu"
+
+PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plpython $(python_includespec) -DPLPYTHON_LIBNAME='"plpython$(python_majorversion)"'
+
+EXTENSION = jsonb_plpythonu jsonb_plpython2u jsonb_plpython3u
+DATA = jsonb_plpythonu--1.0.sql jsonb_plpython2u--1.0.sql jsonb_plpython3u--1.0.sql
+
+REGRESS = jsonb_plpython$(python_majorversion)
+REGRESS_PLPYTHON3_MANGLE := $(REGRESS)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/jsonb_plpython
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+# We must link libpython explicitly
+ifeq ($(PORTNAME), win32)
+# ... see silliness in plpython Makefile ...
+SHLIB_LINK += $(sort $(wildcard ../../src/pl/plpython/libpython*.a))
+else
+rpathdir = $(python_libdir)
+SHLIB_LINK += $(python_libspec) $(python_additional_libs)
+endif
+
+ifeq ($(python_majorversion),2)
+REGRESS_OPTS += --load-extension=plpython2u
+else
+REGRESS_OPTS += --load-extension=plpython3u
+endif
diff --git a/contrib/jsonb_plpython/expected/jsonb_plpython2.out b/contrib/jsonb_plpython/expected/jsonb_plpython2.out
new file mode 100644
index 0000000..8ad5338
--- /dev/null
+++ b/contrib/jsonb_plpython/expected/jsonb_plpython2.out
@@ -0,0 +1,478 @@
+CREATE EXTENSION jsonb_plpython2u CASCADE;
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+INFO:  [('a', Decimal('1')), ('c', 'NULL')]
+ test1 
+-------
+     2
+(1 row)
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+ test1complex 
+--------------
+            1
+(1 row)
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+ test1arr 
+----------
+        1
+(1 row)
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+SELECT test2arr('[12,1]'::jsonb);
+ test2arr 
+----------
+        2
+(1 row)
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+ test3arr 
+----------
+        2
+(1 row)
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+SELECT test1int('1'::jsonb);
+ test1int 
+----------
+        1
+(1 row)
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+SELECT test1string('"a"'::jsonb);
+ test1string 
+-------------
+           1
+(1 row)
+
+-- test jsonb null -> python None
+CREATE FUNCTION test1null(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == None)
+return val==None
+$$;
+SELECT test1null('"a"'::jsonb);
+ERROR:  AssertionError: 
+CONTEXT:  Traceback (most recent call last):
+  PL/Python function "test1null", line 2, in <module>
+    assert(val == None)
+PL/Python function "test1null"
+-- test python -> jsonb
+CREATE FUNCTION back(val jsonb) RETURNS jsonb
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+as $$
+return val
+$$;
+SELECT back('null'::jsonb);
+  back  
+--------
+ [null]
+(1 row)
+
+SELECT back('1'::jsonb);
+ back 
+------
+ [1]
+(1 row)
+
+SELECT back('true'::jsonb);
+  back  
+--------
+ [true]
+(1 row)
+
+SELECT back('"string"'::jsonb);
+    back    
+------------
+ ["string"]
+(1 row)
+
+SELECT back('{"1":null}'::jsonb);
+    back     
+-------------
+ {"1": null}
+(1 row)
+
+SELECT back('{"1":1}'::jsonb);
+   back   
+----------
+ {"1": 1}
+(1 row)
+
+SELECT back('{"1":true}'::jsonb);
+    back     
+-------------
+ {"1": true}
+(1 row)
+
+SELECT back('{"1":"string"}'::jsonb);
+      back       
+-----------------
+ {"1": "string"}
+(1 row)
+
+SELECT back('[null]'::jsonb);
+  back  
+--------
+ [null]
+(1 row)
+
+SELECT back('[1]'::jsonb);
+ back 
+------
+ [1]
+(1 row)
+
+SELECT back('[true]'::jsonb);
+  back  
+--------
+ [true]
+(1 row)
+
+SELECT back('["string"]'::jsonb);
+    back    
+------------
+ ["string"]
+(1 row)
+
+SELECT back('[null,1]'::jsonb);
+   back    
+-----------
+ [null, 1]
+(1 row)
+
+SELECT back('[1,true]'::jsonb);
+   back    
+-----------
+ [1, true]
+(1 row)
+
+SELECT back('[true,"string"]'::jsonb);
+       back       
+------------------
+ [true, "string"]
+(1 row)
+
+SELECT back('["string","string2"]'::jsonb);
+         back          
+-----------------------
+ ["string", "string2"]
+(1 row)
+
+DROP EXTENSION plpython2u CASCADE;
+NOTICE:  drop cascades to 10 other objects
+DETAIL:  drop cascades to extension jsonb_plpython2u
+drop cascades to function test1(jsonb)
+drop cascades to function test1complex(jsonb)
+drop cascades to function test1arr(jsonb)
+drop cascades to function test2arr(jsonb)
+drop cascades to function test3arr(jsonb)
+drop cascades to function test1int(jsonb)
+drop cascades to function test1string(jsonb)
+drop cascades to function test1null(jsonb)
+drop cascades to function back(jsonb)
+-- Testing plpythonu extension.
+CREATE EXTENSION jsonb_plpythonu CASCADE;
+NOTICE:  installing required extension "plpythonu"
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+INFO:  [('a', Decimal('1')), ('c', 'NULL')]
+ test1 
+-------
+     2
+(1 row)
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+ test1complex 
+--------------
+            1
+(1 row)
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+ test1arr 
+----------
+        1
+(1 row)
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+SELECT test2arr('[12,1]'::jsonb);
+ test2arr 
+----------
+        2
+(1 row)
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+ test3arr 
+----------
+        2
+(1 row)
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+SELECT test1int('1'::jsonb);
+ test1int 
+----------
+        1
+(1 row)
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+SELECT test1string('"a"'::jsonb);
+ test1string 
+-------------
+           1
+(1 row)
+
+-- test jsonb null -> python None
+CREATE FUNCTION test1null(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == None)
+return val==None
+$$;
+SELECT test1null('"a"'::jsonb);
+ERROR:  AssertionError: 
+CONTEXT:  Traceback (most recent call last):
+  PL/Python function "test1null", line 2, in <module>
+    assert(val == None)
+PL/Python function "test1null"
+-- test python -> jsonb
+CREATE FUNCTION back(val jsonb) RETURNS jsonb
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+as $$
+return val
+$$;
+SELECT back('null'::jsonb);
+  back  
+--------
+ [null]
+(1 row)
+
+SELECT back('1'::jsonb);
+ back 
+------
+ [1]
+(1 row)
+
+SELECT back('true'::jsonb);
+  back  
+--------
+ [true]
+(1 row)
+
+SELECT back('"string"'::jsonb);
+    back    
+------------
+ ["string"]
+(1 row)
+
+SELECT back('{"1":null}'::jsonb);
+    back     
+-------------
+ {"1": null}
+(1 row)
+
+SELECT back('{"1":1}'::jsonb);
+   back   
+----------
+ {"1": 1}
+(1 row)
+
+SELECT back('{"1":true}'::jsonb);
+    back     
+-------------
+ {"1": true}
+(1 row)
+
+SELECT back('{"1":"string"}'::jsonb);
+      back       
+-----------------
+ {"1": "string"}
+(1 row)
+
+SELECT back('[null]'::jsonb);
+  back  
+--------
+ [null]
+(1 row)
+
+SELECT back('[1]'::jsonb);
+ back 
+------
+ [1]
+(1 row)
+
+SELECT back('[true]'::jsonb);
+  back  
+--------
+ [true]
+(1 row)
+
+SELECT back('["string"]'::jsonb);
+    back    
+------------
+ ["string"]
+(1 row)
+
+SELECT back('[null,1]'::jsonb);
+   back    
+-----------
+ [null, 1]
+(1 row)
+
+SELECT back('[1,true]'::jsonb);
+   back    
+-----------
+ [1, true]
+(1 row)
+
+SELECT back('[true,"string"]'::jsonb);
+       back       
+------------------
+ [true, "string"]
+(1 row)
+
+SELECT back('["string","string2"]'::jsonb);
+         back          
+-----------------------
+ ["string", "string2"]
+(1 row)
+
+DROP EXTENSION plpythonu CASCADE;
+NOTICE:  drop cascades to 10 other objects
+DETAIL:  drop cascades to extension jsonb_plpythonu
+drop cascades to function test1(jsonb)
+drop cascades to function test1complex(jsonb)
+drop cascades to function test1arr(jsonb)
+drop cascades to function test2arr(jsonb)
+drop cascades to function test3arr(jsonb)
+drop cascades to function test1int(jsonb)
+drop cascades to function test1string(jsonb)
+drop cascades to function test1null(jsonb)
+drop cascades to function back(jsonb)
diff --git a/contrib/jsonb_plpython/expected/jsonb_plpython3.out b/contrib/jsonb_plpython/expected/jsonb_plpython3.out
new file mode 100644
index 0000000..bcac5d3
--- /dev/null
+++ b/contrib/jsonb_plpython/expected/jsonb_plpython3.out
@@ -0,0 +1,238 @@
+CREATE EXTENSION jsonb_plpython3u CASCADE;
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+INFO:  [('a', Decimal('1')), ('c', 'NULL')]
+ test1 
+-------
+     2
+(1 row)
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+ test1complex 
+--------------
+            1
+(1 row)
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+ test1arr 
+----------
+        1
+(1 row)
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+SELECT test2arr('[12,1]'::jsonb);
+ test2arr 
+----------
+        2
+(1 row)
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+ test3arr 
+----------
+        2
+(1 row)
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+SELECT test1int('1'::jsonb);
+ test1int 
+----------
+        1
+(1 row)
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+SELECT test1string('"a"'::jsonb);
+ test1string 
+-------------
+           1
+(1 row)
+
+-- test jsonb null -> python None
+CREATE FUNCTION test1null(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == None)
+return val==None
+$$;
+SELECT test1null('"a"'::jsonb);
+ERROR:  AssertionError: 
+CONTEXT:  Traceback (most recent call last):
+  PL/Python function "test1null", line 2, in <module>
+    assert(val == None)
+PL/Python function "test1null"
+-- test python -> jsonb
+CREATE FUNCTION back(val jsonb) RETURNS jsonb
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+as $$
+return val
+$$;
+SELECT back('null'::jsonb);
+  back  
+--------
+ [null]
+(1 row)
+
+SELECT back('1'::jsonb);
+ back 
+------
+ [1]
+(1 row)
+
+SELECT back('true'::jsonb);
+  back  
+--------
+ [true]
+(1 row)
+
+SELECT back('"string"'::jsonb);
+    back    
+------------
+ ["string"]
+(1 row)
+
+SELECT back('{"1":null}'::jsonb);
+    back     
+-------------
+ {"1": null}
+(1 row)
+
+SELECT back('{"1":1}'::jsonb);
+   back   
+----------
+ {"1": 1}
+(1 row)
+
+SELECT back('{"1":true}'::jsonb);
+    back     
+-------------
+ {"1": true}
+(1 row)
+
+SELECT back('{"1":"string"}'::jsonb);
+      back       
+-----------------
+ {"1": "string"}
+(1 row)
+
+SELECT back('[null]'::jsonb);
+  back  
+--------
+ [null]
+(1 row)
+
+SELECT back('[1]'::jsonb);
+ back 
+------
+ [1]
+(1 row)
+
+SELECT back('[true]'::jsonb);
+  back  
+--------
+ [true]
+(1 row)
+
+SELECT back('["string"]'::jsonb);
+    back    
+------------
+ ["string"]
+(1 row)
+
+SELECT back('[null,1]'::jsonb);
+   back    
+-----------
+ [null, 1]
+(1 row)
+
+SELECT back('[1,true]'::jsonb);
+   back    
+-----------
+ [1, true]
+(1 row)
+
+SELECT back('[true,"string"]'::jsonb);
+       back       
+------------------
+ [true, "string"]
+(1 row)
+
+SELECT back('["string","string2"]'::jsonb);
+         back          
+-----------------------
+ ["string", "string2"]
+(1 row)
+
+DROP EXTENSION plpython3u CASCADE;
+NOTICE:  drop cascades to 10 other objects
+DETAIL:  drop cascades to extension jsonb_plpython3u
+drop cascades to function test1(jsonb)
+drop cascades to function test1complex(jsonb)
+drop cascades to function test1arr(jsonb)
+drop cascades to function test2arr(jsonb)
+drop cascades to function test3arr(jsonb)
+drop cascades to function test1int(jsonb)
+drop cascades to function test1string(jsonb)
+drop cascades to function test1null(jsonb)
+drop cascades to function back(jsonb)
diff --git a/contrib/jsonb_plpython/jsonb_plpython.c b/contrib/jsonb_plpython/jsonb_plpython.c
new file mode 100644
index 0000000..6722337
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpython.c
@@ -0,0 +1,418 @@
+/* This document contains an implementation of transformations from python
+ * object to jsonb and vise versa.
+ * In this file you can find implementation of transformations:
+ * - JsonbValue transformation in  PyObject_FromJsonbValue
+ * - JsonbContainer(jsonb) transformation in PyObject_FromJsonb
+ * - PyMapping object(dict) transformation in PyMapping_ToJsonbValue
+ * - PyString object transformation in PyString_ToJsonbValue
+ * - PySequence object(list) transformation in PySequence_ToJsonbValue
+ * - PyNumeric object transformation in PyNumeric_ToJsonbValue
+ * - PyMapping object transformation in PyObject_ToJsonbValue
+ * */
+#include "postgres.h"
+
+#include "plpython.h"
+#include "plpy_typeio.h"
+
+#include "utils/jsonb.h"
+#include "utils/fmgrprotos.h"
+
+PG_MODULE_MAGIC;
+
+extern void _PG_init(void);
+
+/* Linkage to functions in plpython module */
+typedef char *(*PLyObject_AsString_t) (PyObject *plrv);
+static PLyObject_AsString_t PLyObject_AsString_p;
+#if PY_MAJOR_VERSION >= 3
+typedef PyObject *(*PLyUnicode_FromStringAndSize_t) (const char *s, Py_ssize_t size);
+static PLyUnicode_FromStringAndSize_t PLyUnicode_FromStringAndSize_p;
+#endif
+
+/*
+ * Module initialize function: fetch function pointers for cross-module calls.
+ */
+void
+_PG_init(void)
+{
+	/* Asserts verify that typedefs above match original declarations */
+	AssertVariableIsOfType(&PLyObject_AsString, PLyObject_AsString_t);
+	PLyObject_AsString_p = (PLyObject_AsString_t)
+		load_external_function("$libdir/" PLPYTHON_LIBNAME, "PLyObject_AsString",
+							   true, NULL);
+#if PY_MAJOR_VERSION >= 3
+	AssertVariableIsOfType(&PLyUnicode_FromStringAndSize, PLyUnicode_FromStringAndSize_t);
+	PLyUnicode_FromStringAndSize_p = (PLyUnicode_FromStringAndSize_t)
+		load_external_function("$libdir/" PLPYTHON_LIBNAME, "PLyUnicode_FromStringAndSize",
+							   true, NULL);
+#endif
+}
+
+
+/* These defines must be after the module init function */
+#define PLyObject_AsString PLyObject_AsString_p
+#define PLyUnicode_FromStringAndSize PLyUnicode_FromStringAndSize_p
+
+/*
+ * decimal_constructor is a link to Python library
+ * for transforming strings into python decimal type
+ * */
+static PyObject *decimal_constructor;
+
+static PyObject *PyObject_FromJsonb(JsonbContainer *jsonb);
+static JsonbValue *PyObject_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state);
+
+/*
+ * PyObject_FromJsonbValue(JsonsValue *jsonbValue)
+ * Function for transforming JsonbValue type into Python Object
+ * The first argument defines the JsonbValue which will be transformed into PyObject
+ * Return value is the pointer to Jsonb structure containing the transformed object.
+ * */
+static PyObject *
+PyObject_FromJsonbValue(JsonbValue *jsonbValue)
+{
+	PyObject   *result;
+	char	   *str;
+
+	switch (jsonbValue->type)
+	{
+		case jbvNull:
+			result = Py_None;
+			break;
+		case jbvBinary:
+			result = PyObject_FromJsonb(jsonbValue->val.binary.data);
+			break;
+		case jbvNumeric:
+
+			/*
+			 * XXX There should be a better way. Right now Numeric is
+			 * transformed into string and then this string is parsed into py
+			 * numeric
+			 */
+			str = DatumGetCString(
+								  DirectFunctionCall1(numeric_out, NumericGetDatum(jsonbValue->val.numeric))
+				);
+			result = PyObject_CallFunction(decimal_constructor, "s", str);
+			break;
+		case jbvString:
+			result = PyString_FromStringAndSize(
+												jsonbValue->val.string.val,
+												jsonbValue->val.string.len
+				);
+			break;
+		case jbvBool:
+			result = jsonbValue->val.boolean ? Py_True : Py_False;
+			break;
+		case jbvArray:
+		case jbvObject:
+			result = PyObject_FromJsonb(jsonbValue->val.binary.data);
+			break;
+	}
+	return (result);
+}
+
+/*
+ * PyObject_FromJsonb(JsonbContainer *jsonb)
+ * Function for transforming JsonbContainer(jsonb) into PyObject
+ * The first argument should represent the data for transformation.
+ * Return value is the pointer to Python object.
+ * */
+
+static PyObject *
+PyObject_FromJsonb(JsonbContainer *jsonb)
+{
+	PyObject   *object;
+	PyObject   *key;
+	PyObject   *value;
+
+	JsonbIterator *it;
+	JsonbIteratorToken r;
+	JsonbValue	v;
+
+	it = JsonbIteratorInit(jsonb);
+
+	r = JsonbIteratorNext(&it, &v, true);
+
+	switch (r)
+	{
+		case (WJB_BEGIN_ARRAY):
+			/* array in v */
+			object = PyList_New(0);
+			while ((r = JsonbIteratorNext(&it, &v, true)) == WJB_ELEM)
+				PyList_Append(object, PyObject_FromJsonbValue(&v));
+			break;
+		case (WJB_BEGIN_OBJECT):
+			object = PyDict_New();
+			while ((r = JsonbIteratorNext(&it, &v, true)) == WJB_KEY)
+			{
+				key = PyString_FromStringAndSize(
+												 v.val.string.val,
+												 v.val.string.len
+					);
+				r = JsonbIteratorNext(&it, &v, true);
+				value = PyObject_FromJsonbValue(&v);
+				PyDict_SetItem(object, key, value);
+			}
+			break;
+		case (WJB_END_OBJECT):
+			break;
+		default:
+			object = PyObject_FromJsonbValue(&v);
+			break;
+	}
+	return (object);
+}
+
+
+/*
+ * jsonb_to_plpython(Jsonb *in)
+ * Function to transform jsonb object to corresponding python object.
+ * The first argument is the Jsonb object to be transformed.
+ * Return value is the pointer to Python object.
+ * */
+PG_FUNCTION_INFO_V1(jsonb_to_plpython);
+Datum
+jsonb_to_plpython(PG_FUNCTION_ARGS)
+{
+	Jsonb	   *in;
+	PyObject   *dict;
+	PyObject   *decimal_module;
+
+	in = PG_GETARG_JSONB_P(0);
+
+	/* Import python cdecimal library and if there is no cdecimal library, */
+	/* import decimal library */
+	if (!decimal_constructor)
+	{
+		decimal_module = PyImport_ImportModule("cdecimal");
+		if (!decimal_module)
+		{
+			PyErr_Clear();
+			decimal_module = PyImport_ImportModule("decimal");
+		}
+		decimal_constructor = PyObject_GetAttrString(decimal_module, "Decimal");
+	}
+
+	dict = PyObject_FromJsonb(&in->root);
+	return PointerGetDatum(dict);
+}
+
+
+/*
+ * PyMapping_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+ * Function to transform Python lists to jsonbValue
+ * The first argument is the python object to be transformed.
+ * Return value is the pointer to JsonbValue structure containing the list.
+ * */
+static JsonbValue *
+PyMapping_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+{
+	volatile PyObject *items_v = NULL;
+	int32		pcount;
+	JsonbValue *out = NULL;
+
+	pcount = PyMapping_Size(obj);
+	items_v = PyMapping_Items(obj);
+
+	PG_TRY();
+	{
+		int32		i;
+		PyObject   *items;
+		JsonbValue *jbvValue;
+		JsonbValue	jbvKey;
+
+		items = (PyObject *) items_v;
+		pushJsonbValue(&jsonb_state, WJB_BEGIN_OBJECT, NULL);
+
+		for (i = 0; i < pcount; i++)
+		{
+			PyObject   *tuple;
+			PyObject   *key;
+			PyObject   *value;
+
+			tuple = PyList_GetItem(items, i);
+			key = PyTuple_GetItem(tuple, 0);
+			value = PyTuple_GetItem(tuple, 1);
+
+			if (key == Py_None)
+			{
+				jbvKey.type = jbvString;
+				jbvKey.val.string.len = 0;
+				jbvKey.val.string.val = "";
+			}
+			else
+			{
+				jbvKey.type = jbvString;
+				jbvKey.val.string.val = PLyObject_AsString(key);
+				jbvKey.val.string.len = strlen(jbvKey.val.string.val);
+			}
+			pushJsonbValue(&jsonb_state, WJB_KEY, &jbvKey);
+			jbvValue = PyObject_ToJsonbValue(value, jsonb_state);
+			if (IsAJsonbScalar(jbvValue))
+				pushJsonbValue(&jsonb_state, WJB_VALUE, jbvValue);
+		}
+		out = pushJsonbValue(&jsonb_state, WJB_END_OBJECT, NULL);
+	}
+	PG_CATCH();
+	{
+		Py_DECREF(items_v);
+		PG_RE_THROW();
+	}
+	PG_END_TRY();
+	return (out);
+}
+
+/*
+ * PyString_ToJsonbValue(PyObject *obj)
+ * Function to transform python string object to jsonbValue object.
+ * The first argument is the Python String object to be transformed.
+ * Return value is the pointer to JsonbValue structure containing the String.
+ * */
+static JsonbValue *
+PyString_ToJsonbValue(PyObject *obj)
+{
+	JsonbValue *out = NULL;
+	JsonbValue *jbvElem;
+
+	jbvElem = palloc(sizeof(JsonbValue));
+	jbvElem->type = jbvString;
+	jbvElem->val.string.val = PLyObject_AsString(obj);
+	jbvElem->val.string.len = strlen(jbvElem->val.string.val);
+	out = jbvElem;
+
+	return (out);
+}
+
+/*
+ * PySequence_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+ * Function to transform python lists to jsonbValue object.
+ * The first argument is the Python list to be transformed.
+ * The second one is conversion state.
+ * Return value is the pointer to JsonbValue structure containing array.
+ * */
+static JsonbValue *
+PySequence_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+{
+	JsonbValue *jbvElem;
+	JsonbValue *out = NULL;
+	int32		pcount;
+	int32		i;
+
+	pcount = PySequence_Size(obj);
+
+
+	pushJsonbValue(&jsonb_state, WJB_BEGIN_ARRAY, NULL);
+
+	for (i = 0; i < pcount; i++)
+	{
+		PyObject   *value;
+
+		value = PySequence_GetItem(obj, i);
+		jbvElem = PyObject_ToJsonbValue(value, jsonb_state);
+		if (IsAJsonbScalar(jbvElem))
+			pushJsonbValue(&jsonb_state, WJB_ELEM, jbvElem);
+	}
+	out = pushJsonbValue(&jsonb_state, WJB_END_ARRAY, NULL);
+	return (out);
+}
+
+/*
+ * PyNumeric_ToJsonbValue(PyObject *obj)
+ * Function to transform python numerics to jsonbValue object.
+ * The first argument is the Python numeric object to be transformed.
+ * Return value is the pointer to JsonbValue structure containing the String.
+ * */
+static JsonbValue *
+PyNumeric_ToJsonbValue(PyObject *obj)
+{
+	JsonbValue *out = NULL;
+	JsonbValue *jbvInt;
+
+	jbvInt = palloc(sizeof(JsonbValue));
+	jbvInt->type = jbvNumeric;
+	jbvInt->val.numeric = DatumGetNumeric(DirectFunctionCall3(
+															  numeric_in,
+															  CStringGetDatum(PLyObject_AsString(obj)),
+															  0,
+															  -1
+															  ));
+	out = jbvInt;
+	return (out);
+}
+
+/*
+ * PyObject_ToJsonbValue(PyObject *obj)
+ * Function to transform python objects to jsonbValue object.
+ * The first argument is the Python object to be transformed.
+ * Return value is the pointer to JsonbValue structure containing the transformed object.
+ * */
+static JsonbValue *
+PyObject_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+{
+	JsonbValue *out = NULL;
+	PyObject   *type = PyObject_Type(obj);
+
+	if (type == PyObject_Type(PyDict_New()))
+	{
+		/* DICT */
+		out = PyMapping_ToJsonbValue(obj, jsonb_state);
+	}
+	else if (type == PyObject_Type(PyList_New(0)))
+	{
+		/* LIST or STRING */
+		/* but we have checked on STRING */
+		out = PySequence_ToJsonbValue(obj, jsonb_state);
+	}
+	else if (PyBool_Check(obj))
+	{
+		/* Boolean */
+		JsonbValue *jbvElem;
+
+		jbvElem = palloc(sizeof(JsonbValue));
+		jbvElem->type = jbvBool;
+		jbvElem->val.boolean = obj == Py_True;
+		out = jbvElem;
+
+	}
+	else if (obj == Py_None)
+	{
+		/* None */
+		JsonbValue *jbvElem;
+
+		jbvElem = palloc(sizeof(JsonbValue));
+		jbvElem->type = jbvNull;
+		out = jbvElem;
+
+	}
+	else if (PyNumber_Check(obj))
+	{
+		/* NUMERIC */
+		out = PyNumeric_ToJsonbValue(obj);
+	}
+	else
+	{
+		/* EVERYTHING ELSE */
+		/* Handle it as it's repr */
+		out = PyString_ToJsonbValue(obj);
+	}
+	return (out);
+}
+
+/*
+ * plpython_to_jsonb(PyObject *obj)
+ * Function to transform python objects to jsonb object.
+ * The first argument is the Python object to be transformed.
+ * Return value is the pointer to Jsonb structure containing the transformed object.
+ * */
+PG_FUNCTION_INFO_V1(plpython_to_jsonb);
+Datum
+plpython_to_jsonb(PG_FUNCTION_ARGS)
+{
+	PyObject   *obj;
+	JsonbValue *out;
+	JsonbParseState *jsonb_state = NULL;
+
+	obj = (PyObject *) PG_GETARG_POINTER(0);
+	out = PyObject_ToJsonbValue(obj, jsonb_state);
+	PG_RETURN_POINTER(JsonbValueToJsonb(out));
+}
diff --git a/contrib/jsonb_plpython/jsonb_plpython2u--1.0.sql b/contrib/jsonb_plpython/jsonb_plpython2u--1.0.sql
new file mode 100644
index 0000000..1e38847
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpython2u--1.0.sql
@@ -0,0 +1,19 @@
+/* contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION jsonb_plpython" to load this file. \quit
+
+CREATE FUNCTION jsonb_to_plpython(val internal) RETURNS internal
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE FUNCTION plpython_to_jsonb(val internal) RETURNS jsonb
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE TRANSFORM FOR jsonb LANGUAGE plpython2u (
+	FROM SQL WITH FUNCTION jsonb_to_plpython(internal),
+	TO SQL WITH FUNCTION plpython_to_jsonb(internal)
+);
+
+COMMENT ON TRANSFORM FOR jsonb LANGUAGE plpython2u IS 'transform between jsonb and Python dict';
diff --git a/contrib/jsonb_plpython/jsonb_plpython2u.control b/contrib/jsonb_plpython/jsonb_plpython2u.control
new file mode 100644
index 0000000..e7d7da3
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpython2u.control
@@ -0,0 +1,6 @@
+# jsonb_plpython2u extension
+comment = 'transform between jsonb and plpythonu'
+default_version = '1.0'
+module_pathname = '$libdir/jsonb_plpython2u'
+relocatable = false
+requires = 'plpython2u'
diff --git a/contrib/jsonb_plpython/jsonb_plpython3u--1.0.sql b/contrib/jsonb_plpython/jsonb_plpython3u--1.0.sql
new file mode 100644
index 0000000..0958db7
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpython3u--1.0.sql
@@ -0,0 +1,19 @@
+/* contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION jsonb_plpython3" to load this file. \quit
+
+CREATE FUNCTION jsonb_to_plpython(val internal) RETURNS internal
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE FUNCTION plpython_to_jsonb(val internal) RETURNS jsonb
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE TRANSFORM FOR jsonb LANGUAGE plpython3u (
+	FROM SQL WITH FUNCTION jsonb_to_plpython(internal),
+	TO SQL WITH FUNCTION plpython_to_jsonb(internal)
+);
+
+COMMENT ON TRANSFORM FOR jsonb LANGUAGE plpython3u IS 'transform between jsonb and Python dict';
diff --git a/contrib/jsonb_plpython/jsonb_plpython3u.control b/contrib/jsonb_plpython/jsonb_plpython3u.control
new file mode 100644
index 0000000..5e08544
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpython3u.control
@@ -0,0 +1,6 @@
+# jsonb_plpython3u extension
+comment = 'transform between jsonb and plpython3u'
+default_version = '1.0'
+module_pathname = '$libdir/jsonb_plpython3u'
+relocatable = false
+requires = 'plpython3u'
diff --git a/contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql b/contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql
new file mode 100644
index 0000000..7c9460d
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql
@@ -0,0 +1,19 @@
+/* contrib/jsonb_plpython/jsonb_plpythonu--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION jsonb_plpython" to load this file. \quit
+
+CREATE FUNCTION jsonb_to_plpython(val internal) RETURNS internal
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE FUNCTION plpython_to_jsonb(val internal) RETURNS jsonb
+LANGUAGE C STRICT IMMUTABLE
+AS 'MODULE_PATHNAME';
+
+CREATE TRANSFORM FOR jsonb LANGUAGE plpythonu (
+	FROM SQL WITH FUNCTION jsonb_to_plpython(internal),
+	TO SQL WITH FUNCTION plpython_to_jsonb(internal)
+);
+
+COMMENT ON TRANSFORM FOR jsonb LANGUAGE plpythonu IS 'transform between jsonb and Python dict';
diff --git a/contrib/jsonb_plpython/jsonb_plpythonu.control b/contrib/jsonb_plpython/jsonb_plpythonu.control
new file mode 100644
index 0000000..9bbeb38
--- /dev/null
+++ b/contrib/jsonb_plpython/jsonb_plpythonu.control
@@ -0,0 +1,6 @@
+# jsonb_plpython2u extension
+comment = 'transform between jsonb and plpythonu'
+default_version = '1.0'
+module_pathname = '$libdir/jsonb_plpython2u'
+relocatable = false
+requires = 'plpythonu'
diff --git a/contrib/jsonb_plpython/sql/jsonb_plpython2.sql b/contrib/jsonb_plpython/sql/jsonb_plpython2.sql
new file mode 100644
index 0000000..7a5445f
--- /dev/null
+++ b/contrib/jsonb_plpython/sql/jsonb_plpython2.sql
@@ -0,0 +1,261 @@
+CREATE EXTENSION jsonb_plpython2u CASCADE;
+
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+
+SELECT test2arr('[12,1]'::jsonb);
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+
+SELECT test1int('1'::jsonb);
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+
+SELECT test1string('"a"'::jsonb);
+
+-- test jsonb null -> python None
+CREATE FUNCTION test1null(val jsonb) RETURNS int
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == None)
+return val==None
+$$;
+
+SELECT test1null('"a"'::jsonb);
+
+-- test python -> jsonb
+CREATE FUNCTION back(val jsonb) RETURNS jsonb
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+as $$
+return val
+$$;
+
+SELECT back('null'::jsonb);
+SELECT back('1'::jsonb);
+SELECT back('true'::jsonb);
+SELECT back('"string"'::jsonb);
+
+SELECT back('{"1":null}'::jsonb);
+SELECT back('{"1":1}'::jsonb);
+SELECT back('{"1":true}'::jsonb);
+SELECT back('{"1":"string"}'::jsonb);
+
+SELECT back('[null]'::jsonb);
+SELECT back('[1]'::jsonb);
+SELECT back('[true]'::jsonb);
+SELECT back('["string"]'::jsonb);
+SELECT back('[null,1]'::jsonb);
+SELECT back('[1,true]'::jsonb);
+SELECT back('[true,"string"]'::jsonb);
+SELECT back('["string","string2"]'::jsonb);
+
+
+DROP EXTENSION plpython2u CASCADE;
+
+
+-- Testing plpythonu extension.
+CREATE EXTENSION jsonb_plpythonu CASCADE;
+
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+
+SELECT test2arr('[12,1]'::jsonb);
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+
+SELECT test1int('1'::jsonb);
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+
+SELECT test1string('"a"'::jsonb);
+
+-- test jsonb null -> python None
+CREATE FUNCTION test1null(val jsonb) RETURNS int
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == None)
+return val==None
+$$;
+
+SELECT test1null('"a"'::jsonb);
+
+-- test python -> jsonb
+CREATE FUNCTION back(val jsonb) RETURNS jsonb
+LANGUAGE plpythonu
+TRANSFORM FOR TYPE jsonb
+as $$
+return val
+$$;
+
+SELECT back('null'::jsonb);
+SELECT back('1'::jsonb);
+SELECT back('true'::jsonb);
+SELECT back('"string"'::jsonb);
+
+SELECT back('{"1":null}'::jsonb);
+SELECT back('{"1":1}'::jsonb);
+SELECT back('{"1":true}'::jsonb);
+SELECT back('{"1":"string"}'::jsonb);
+
+SELECT back('[null]'::jsonb);
+SELECT back('[1]'::jsonb);
+SELECT back('[true]'::jsonb);
+SELECT back('["string"]'::jsonb);
+SELECT back('[null,1]'::jsonb);
+SELECT back('[1,true]'::jsonb);
+SELECT back('[true,"string"]'::jsonb);
+SELECT back('["string","string2"]'::jsonb);
+
+
+DROP EXTENSION plpythonu CASCADE;
diff --git a/contrib/jsonb_plpython/sql/jsonb_plpython3.sql b/contrib/jsonb_plpython/sql/jsonb_plpython3.sql
new file mode 100644
index 0000000..eaddf9a
--- /dev/null
+++ b/contrib/jsonb_plpython/sql/jsonb_plpython3.sql
@@ -0,0 +1,129 @@
+CREATE EXTENSION jsonb_plpython3u CASCADE;
+
+-- test jsonb -> python dict
+CREATE FUNCTION test1(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+plpy.info(sorted(val.items()))
+return len(val)
+$$;
+
+SELECT test1('{"a":1, "c":"NULL"}'::jsonb);
+
+-- test jsonb -> python dict
+-- complex dict with dicts as value
+CREATE FUNCTION test1complex(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d":{"d": 1}})
+return len(val)
+$$;
+
+SELECT test1complex('{"d":{"d": 1}}'::jsonb);
+
+
+-- test jsonb[] -> python dict
+-- dict with array as value
+CREATE FUNCTION test1arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, dict)
+assert(val == {"d": [12,1]})
+return len(val)
+$$;
+
+SELECT test1arr('{"d":[12,1]}'::jsonb);
+
+-- test jsonb[] -> python list
+-- simple list
+CREATE FUNCTION test2arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [12,1])
+return len(val)
+$$;
+
+SELECT test2arr('[12,1]'::jsonb);
+
+-- test jsonb[] -> python list
+-- array of dicts
+CREATE FUNCTION test3arr(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert isinstance(val, list)
+assert(val == [{"a":1,"b":2},{"c":3,"d":4}])
+return len(val)
+$$;
+
+SELECT test3arr('[{"a":1,"b":2},{"c":3,"d":4}]'::jsonb);
+
+-- test jsonb int -> python int
+CREATE FUNCTION test1int(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == [1])
+return len(val)
+$$;
+
+SELECT test1int('1'::jsonb);
+
+-- test jsonb string -> python string
+CREATE FUNCTION test1string(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == ["a"])
+return len(val)
+$$;
+
+SELECT test1string('"a"'::jsonb);
+
+-- test jsonb null -> python None
+CREATE FUNCTION test1null(val jsonb) RETURNS int
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+AS $$
+assert(val == None)
+return val==None
+$$;
+
+SELECT test1null('"a"'::jsonb);
+
+-- test python -> jsonb
+CREATE FUNCTION back(val jsonb) RETURNS jsonb
+LANGUAGE plpython3u
+TRANSFORM FOR TYPE jsonb
+as $$
+return val
+$$;
+
+SELECT back('null'::jsonb);
+SELECT back('1'::jsonb);
+SELECT back('true'::jsonb);
+SELECT back('"string"'::jsonb);
+
+SELECT back('{"1":null}'::jsonb);
+SELECT back('{"1":1}'::jsonb);
+SELECT back('{"1":true}'::jsonb);
+SELECT back('{"1":"string"}'::jsonb);
+
+SELECT back('[null]'::jsonb);
+SELECT back('[1]'::jsonb);
+SELECT back('[true]'::jsonb);
+SELECT back('["string"]'::jsonb);
+SELECT back('[null,1]'::jsonb);
+SELECT back('[1,true]'::jsonb);
+SELECT back('[true,"string"]'::jsonb);
+SELECT back('["string","string2"]'::jsonb);
+
+
+DROP EXTENSION plpython3u CASCADE;
diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index 05ecef2..4e8a7a0 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -569,4 +569,20 @@ SELECT jdoc-&gt;'guid', jdoc-&gt;'name' FROM api WHERE jdoc @&gt; '{"tags": ["qu
       compared using the default database collation.
   </para>
  </sect2>
+  <sect2>
+  <title>Transforms</title>
+
+  <para>
+   Additional extensions are available that implement transforms for
+   the <type>jsonb</type> type for the language PL/Python.  The
+   extensions for PL/Python are called
+   <literal>jsonb_plpythonu</literal>, <literal>jsonb_plpython2u</literal>,
+   and <literal>jsonb_plpython3u</literal>
+   (see <xref linkend="plpython-python23"> for the PL/Python naming
+   convention).  If you use them, <type>jsonb</type> values are mapped to
+   Python dictionaries.
+  </para>
+ </sect2>
+
+
 </sect1>
#7Aleksander Alekseev
a.alekseev@postgrespro.ru
In reply to: Anthony Bykov (#6)
Re: Jsonb transform for pl/python

The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: tested, passed
Spec compliant: tested, passed
Documentation: tested, passed

Hi Anthony,

Thank you for the new version of the patch! Here is my code review.

1. In jsonb_plpython2.out:

+CREATE FUNCTION back(val jsonb) RETURNS jsonb
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+as $$
+return val
+$$;
+SELECT back('null'::jsonb);
+  back  
+--------
+ [null]
+(1 row)
+
+SELECT back('1'::jsonb);
+ back 
+------
+ [1]
+(1 row)
+
+SELECT back('true'::jsonb);
+  back  
+--------
+ [true]
+(1 row)

Maybe I'm missing something, but why exactly all JSONB values turn into arrays?

2. Could you please also add tests for some negative and real numbers? Also
could you check that your code handles numbers like MAX_INT, MIN_INT, +/-
infinity and NaN properly in both (Python <-> JSONB) directions?

3. Handling unicode strings properly is another thing that is worth checking.

4. I think we also need some tests that check the behavior of Python -> JSONB
conversion when the object contains data that is not representable in JSON
format, e.g. set(), some custom objects, etc.

5. PyObject_FromJsonbValue - I realize it's unlikely that the new
jsonbValue->type will be introduced any time soon. Still I believe it's a good
practice to add "it should never happen" default case that just does
elog(ERROR, ...) in case it happens nevertheless. Otherwise in this scenario
instead of reporting the error the code will silently do the wrong thing.

6. Well, you decided to make the extension non-relocatable. Could you at least
describe what prevents it to be relocatable or why it's meaningless is a
comment in .control file? Please note that almost all contrib/ extensions are
relocatable. I believe your extension should be relocatable as well unless
there is a good reason why it can't.

The new status of this patch is: Waiting on Author

#8Anthony Bykov
a.bykov@postgrespro.ru
In reply to: Aleksander Alekseev (#7)
1 attachment(s)
Re: Jsonb transform for pl/python

On Mon, 13 Nov 2017 15:08:16 +0000
Aleksander Alekseev <a.alekseev@postgrespro.ru> wrote:

The following review has been posted through the commitfest
application: make installcheck-world: tested, passed
Implements feature: tested, passed
Spec compliant: tested, passed
Documentation: tested, passed

Hi Anthony,

Thank you for the new version of the patch! Here is my code review.

1. In jsonb_plpython2.out:

+CREATE FUNCTION back(val jsonb) RETURNS jsonb
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+as $$
+return val
+$$;
+SELECT back('null'::jsonb);
+  back  
+--------
+ [null]
+(1 row)
+
+SELECT back('1'::jsonb);
+ back 
+------
+ [1]
+(1 row)
+
+SELECT back('true'::jsonb);
+  back  
+--------
+ [true]
+(1 row)

Maybe I'm missing something, but why exactly all JSONB values turn
into arrays?

2. Could you please also add tests for some negative and real
numbers? Also could you check that your code handles numbers like
MAX_INT, MIN_INT, +/- infinity and NaN properly in both (Python <->
JSONB) directions?

3. Handling unicode strings properly is another thing that is worth
checking.

4. I think we also need some tests that check the behavior of Python
-> JSONB conversion when the object contains data that is not
representable in JSON format, e.g. set(), some custom objects, etc.

5. PyObject_FromJsonbValue - I realize it's unlikely that the new
jsonbValue->type will be introduced any time soon. Still I believe
it's a good practice to add "it should never happen" default case
that just does elog(ERROR, ...) in case it happens nevertheless.
Otherwise in this scenario instead of reporting the error the code
will silently do the wrong thing.

6. Well, you decided to make the extension non-relocatable. Could you
at least describe what prevents it to be relocatable or why it's
meaningless is a comment in .control file? Please note that almost
all contrib/ extensions are relocatable. I believe your extension
should be relocatable as well unless there is a good reason why it
can't.

The new status of this patch is: Waiting on Author

Hi,
thank you for your review. I took your comments into account in the
third version of the patch. In the new version, I've added all the
tests you asked for. The interesting thing is that:
1. set or any other non-jsonb-transformable object is transformed into
string and added to jsonb as a string.
2. couldn't find a solution of working with "inf": this extension
troughs exception if python generates a number called "inf" and returns
it, but if we pass a very large integer into a function, it works fine
with the whole number. This situation can be seen in tests.

I've added tests of large numerics which weights quite heavy. So,
please find it in compressed format in attachments.

--
Anthony Bykov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attachments:

0001-jsonb_plpython-extension-v3.patch.gzapplication/gzipDownload
#9Aleksander Alekseev
a.alekseev@postgrespro.ru
In reply to: Anthony Bykov (#8)
Re: Jsonb transform for pl/python

Hi Anthony,

thank you for your review. I took your comments into account in the
third version of the patch. In the new version, I've added all the
tests you asked for. The interesting thing is that:
1. set or any other non-jsonb-transformable object is transformed into
string and added to jsonb as a string.

Well frankly I very much doubt that this:

```
+-- set -> jsonb
+CREATE FUNCTION test1set() RETURNS jsonb
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+x = set()
+x.add(1)
+x.add("string")
+x.add(None)
+return x
+$$;
+SELECT test1set();
+          test1set          
+----------------------------
+ "set([1, 'string', None])"
+(1 row)
```

... is an expected and valid behavior. If user tries to transform a
set() to JSONB this is most likely a mistake since there is no standard
representation of a set() in JSONB. I believe we should rise an error in
this case instead of generating a string. Besides user can expect that
such string can be transformed back to set() which doesn't sound like a
good idea either.

If necessary, user can just transform a set() to a list():

```

x = set([1,2,3,4])
x

{1, 2, 3, 4}

list(x)

[1, 2, 3, 4]
```

BTW I just recalled that Python supports complex numbers out-of-the box
and that range and xrange are a separate types too:

```

1 + 2j

(1+2j)

range(3)

range(0, 3)

type(range(3))

<class 'range'>
```

I think we should add all this to the tests as well. Naturally complex
numbers can't be represented in JSON so we should rise an error if user
tries to transform a complex number to JSON. I'm not that sure regarding
ranges though. Probably the best solution will be to rise and error in
this case as well just to keep things consistent.

+ERROR: jsonb doesn't support inf type yet

I would say this error message is too informal. How about something more
like "Infinity can't be represented in JSONB"?

2. couldn't find a solution of working with "inf": this extension
troughs exception if python generates a number called "inf" and returns
it, but if we pass a very large integer into a function, it works fine
with the whole number. This situation can be seen in tests.

I've added tests of large numerics which weights quite heavy. So,
please find it in compressed format in attachments.

I'm afraid that tests fail on Python 3:

```
SELECT test1set();
test1set
-----------------------
! "{None, 1, 'string'}"
(1 row)

  DROP EXTENSION plpython3u CASCADE;
--- 296,302 ----
  SELECT test1set();
         test1set        
  -----------------------
!  "{1, None, 'string'}"
  (1 row)

DROP EXTENSION plpython3u CASCADE
```

--
Best regards,
Aleksander Alekseev

#10Michael Paquier
michael.paquier@gmail.com
In reply to: Aleksander Alekseev (#9)
Re: Jsonb transform for pl/python

On Mon, Nov 20, 2017 at 10:48 PM, Aleksander Alekseev
<a.alekseev@postgrespro.ru> wrote:

Well frankly I very much doubt that this:
[snip]
I'm afraid that tests fail on Python 3:

So this still needs more work.. I am marking it as returned with
feedback as there has been no updates for more than 1 week.
--
Michael

#11Anthony Bykov
a.bykov@postgrespro.ru
In reply to: Aleksander Alekseev (#9)
1 attachment(s)
Re: Jsonb transform for pl/python

Hello,
fixed the issues:
1. Rising errors when invalid object being transformed.
2. We don't rise the exception when transforming range(3) only in
python 2. In third one it is an error.

Please, find the 4-th version of the patch in attachments to this
message. --
Anthony Bykov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attachments:

0001-jsonb_plpython-extension-v4.patch.tar.gzapplication/gzipDownload
#12Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Anthony Bykov (#11)
Re: Jsonb transform for pl/python

On 12/6/17 06:40, Anthony Bykov wrote:

Hello,
fixed the issues:
1. Rising errors when invalid object being transformed.
2. We don't rise the exception when transforming range(3) only in
python 2. In third one it is an error.

Please, find the 4-th version of the patch in attachments to this
message.

Why not make this part of the plpythonu extension? It doesn't have to
be a separate contrib module.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#13Anthony Bykov
a.bykov@postgrespro.ru
In reply to: Peter Eisentraut (#12)
Re: Jsonb transform for pl/python

On Sat, 9 Dec 2017 16:57:05 -0500
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote:

On 12/6/17 06:40, Anthony Bykov wrote:

Hello,
fixed the issues:
1. Rising errors when invalid object being transformed.
2. We don't rise the exception when transforming range(3) only in
python 2. In third one it is an error.

Please, find the 4-th version of the patch in attachments to this
message.

Why not make this part of the plpythonu extension? It doesn't have to
be a separate contrib module.

Hello,
I thought about that, but the problem is that there will be no
possibilities to create custom transform if we create this extension by
default. For example, it is easy to check if we install this extension
and try to create new transform:

# create extension jsonb_plperl cascade;
NOTICE: installing required extension "plperl"
CREATE EXTENSION

# CREATE TRANSFORM FOR jsonb LANGUAGE plperl (
# FROM SQL WITH FUNCTION jsonb_to_plperl(internal),
# TO SQL WITH FUNCTION plperl_to_jsonb(internal)
# );
2017-12-11 10:23:07.507 MSK [19149] ERROR: transform for type jsonb
language "plperl" already exists 2017-12-11 10:23:07.507 MSK [19149]
STATEMENT: CREATE TRANSFORM FOR jsonb LANGUAGE plperl ( FROM SQL WITH
FUNCTION jsonb_to_plperl(internal), TO SQL WITH FUNCTION
plperl_to_jsonb(internal) );
ERROR: transform for type jsonb language "plperl" already exists

Other types of transforms may be interesting for people when they want
to transform the jsonb to certain structures. For example, what if the
user wants the parameter to be always array inside the function, while
this extension can return integers or strings in some cases.

--
Anthony Bykov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

#14Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Anthony Bykov (#13)
Re: Jsonb transform for pl/python

On 12/11/17 03:22, Anthony Bykov wrote:

Why not make this part of the plpythonu extension? It doesn't have to
be a separate contrib module.

Hello,
I thought about that, but the problem is that there will be no
possibilities to create custom transform if we create this extension by
default.

OK, could it be a separate extension, but part of the same code directory?

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#15Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#14)
Re: Jsonb transform for pl/python

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

On 12/11/17 03:22, Anthony Bykov wrote:

Why not make this part of the plpythonu extension? It doesn't have to
be a separate contrib module.

I thought about that, but the problem is that there will be no
possibilities to create custom transform if we create this extension by
default.

OK, could it be a separate extension, but part of the same code directory?

I think our makefile infrastructure only allows for one shlib to be
built per directory.

Admittedly, you could have two extensions sharing the same shlib
(and the same regression test suite), but on the whole it's not clear
to me why we should do that.

regards, tom lane

#16Thomas Munro
thomas.munro@enterprisedb.com
In reply to: Anthony Bykov (#11)
Re: Jsonb transform for pl/python

On Thu, Dec 7, 2017 at 12:40 AM, Anthony Bykov <a.bykov@postgrespro.ru> wrote:

Hello,
fixed the issues:
1. Rising errors when invalid object being transformed.
2. We don't rise the exception when transforming range(3) only in
python 2. In third one it is an error.

Please, find the 4-th version of the patch in attachments to this
message. --

Hi Anthony,

FYI make docs fails:

json.sgml:584: parser error : Opening and ending tag mismatch: xref
line 581 and para
</para>
^
json.sgml:585: parser error : Opening and ending tag mismatch: para
line 575 and sect2
</sect2>
^
json.sgml:588: parser error : Opening and ending tag mismatch: sect2
line 572 and sect1
</sect1>
^
json.sgml:589: parser error : Premature end of data in tag sect1 line 3
json.sgml:589: parser error : chunk is not well balanced
datatype.sgml:4354: parser error : Failure to process entity json
&json;
^
datatype.sgml:4354: parser error : Entity 'json' not defined
&json;
^
datatype.sgml:4955: parser error : chunk is not well balanced
postgres.sgml:104: parser error : Failure to process entity datatype
&datatype;
^
postgres.sgml:104: parser error : Entity 'datatype' not defined
&datatype;
^

--
Thomas Munro
http://www.enterprisedb.com

#17Anthony Bykov
a.bykov@postgrespro.ru
In reply to: Thomas Munro (#16)
1 attachment(s)
Re: Jsonb transform for pl/python

On Fri, 12 Jan 2018 13:33:56 +1300
Thomas Munro <thomas.munro@enterprisedb.com> wrote:

On Thu, Dec 7, 2017 at 12:40 AM, Anthony Bykov
<a.bykov@postgrespro.ru> wrote:

Hello,
fixed the issues:
1. Rising errors when invalid object being transformed.
2. We don't rise the exception when transforming range(3) only in
python 2. In third one it is an error.

Please, find the 4-th version of the patch in attachments to this
message. --

Hi Anthony,

FYI make docs fails:

json.sgml:584: parser error : Opening and ending tag mismatch: xref
line 581 and para
</para>
^
json.sgml:585: parser error : Opening and ending tag mismatch: para
line 575 and sect2
</sect2>
^
json.sgml:588: parser error : Opening and ending tag mismatch: sect2
line 572 and sect1
</sect1>
^
json.sgml:589: parser error : Premature end of data in tag sect1 line
3 json.sgml:589: parser error : chunk is not well balanced
datatype.sgml:4354: parser error : Failure to process entity json
&json;
^
datatype.sgml:4354: parser error : Entity 'json' not defined
&json;
^
datatype.sgml:4955: parser error : chunk is not well balanced
postgres.sgml:104: parser error : Failure to process entity datatype
&datatype;
^
postgres.sgml:104: parser error : Entity 'datatype' not defined
&datatype;
^

Hello, thank you for your message.

Fixed it. Here is a new version of the patch.

--
Anthony Bykov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attachments:

0001-jsonb_plpython-extension-v5.patch.tar.gzapplication/gzipDownload
�[~XZ��]oI~0����}�>+j(���y��sF#�^ml�G�ww01�j���I-I���c`7wAnr���2@d�$@�<w�|���������6;���HVWW������j����V�i7�Gg������j��7�������*�����F���?��z�_��������J�7��JQisc����7?�\��7J����+Z��.�����BQ*���/�����z*���G��vz��>�u���j��i�����|���qt<�V��8��G��_��Z���v������>�J�_��[]]��O�X,N���/��OJ+�F��gx�u��[_<�����AT�=nw��8i�q����F(q.�E����Q�j�Y���(k��+/��rnu��i���i������I��>
Ro���Y�����n1��^~���������Gqt�IuJ�\#~�I�4O����|�����6�?-�_��#~�$~z�q^�_8���O����zT,�l&�=W�I4o�b�xo�������������\-d�?zR���z���k�<W����c������/wv77������;�w��o��C��h�h5��*��i��$:�;���{L�J�$�%;�'%��h����w��$;]��Z�4�����I��|-���Y�Z?�~�k�j��$n����h������:�������/w���o.�g�3�������������X}�/�����f�������-WWKk�k�_����������T��;!���`�[,�Q+�7��N��V�u^��r�������;w����!��O����NO����A����_d�lo����P��GG���%o�� ���8�����	�!Z={�<�4���K?�U�:*������'7�\1i����I�lm����pn��g������������&��a���Y��'�����F1�k�=;���z��m�q�m����z�Z��_$a����{��I\^�������kkkQ;��v�����������?��%�:�Yh��%��e2��p6[�(��'�J�$����q�_��*�����d�q�~�)����h�y����TBT;�U����C���zR�[I��kg{���{u����fL7��W��7I:aV�E��p��j'>�v�_�5�;�����-4�fi'n\f�������G�d������a9�hc���������g�b(F'�9�W?�����V��s��`w�0�(IZ*<�tG��h�|�`� ����n4l�)G���pk������(����_��
������\��n��Q���8L&�8�%-@h2IVk��i3m��I��Z�?i�����;��p���'���WC��w�����.}���o�V�|5#��������7���;���nD������Jt+���T�����r����jxs)�ji�a���%�K�w�b�����QrvP������zh�Q2KkG�vjqO�|w��z�O�]��}�?��HD��/���K�����������4=�_?��A�+�V����D?t|��K+����7)o��V��
IF�:������Y0���u_O���{�_R���e1X$piA���c!���=}r��V�<�����m��Q���������+�%��J����\(��Y0ge3����q��(�����'�$����q��5�"���XL�p�������h���S��}�N#�B#T�(K!�x,���0���������B��lL����4���(MDZ����x���Da(�#�����W���>�T��T�n�KV:�7�Jw��4���a����%o��^��dd����J36,K����G3�(:�T�������\q�/W�J��\�P�Z�g��b�xP�[�>��+�oK��r��YZ���t��7�f�x��\1�(:�T�������\q` W�VK��\�P�Z�g��b�xP�[�>��+�oK��y�t��7�f�x��\1};�WS�u�`���v�Uk<��de�D�l�Y�O���|)�q^����a/��<�l���t�99�.f;��Y��y�%��|9������X�C���ad/�����$bG���>�8��%��M����>��q���&��M�����X%Y�7�����L'����J4����	5��ci%�W��������b��]��H/Mo�w9��)�;�l�*A�����|��~��o���(���o^����|���E��_�'�y�o�����y��W����W�_�:���������������?��.X����L���*�H@VW�N��Dg/:�����8���V?��
�������rt�������n�����������^��[�wl�)Gg�,���\�pk������(����_�%�:��^��G7�R�X�,���+��c�)<%k���V#z�+^��Y��
v�+�������;���h�6[O*�Zx�Z�V�x�|��s7��<�����y�C��Zi4���8l�D�f��{���_�-�=������C��GQ�li^������g�?=oT�=�{���xu�o���^2�o!���#�c�[������;j���^��t��&�D����B(�����;�j�������_�*''�R�����c���gw�4��hG����8��������(}���k�W����'g��y�>9�[�9���R�f��j�����7��{q�o�c���7����v���n���G��@�'i�f|��6g���N{]6+���j=��V���������9��WkO*��Y�d�����e�V'�.36>�p�N����G���w�
�m�����r�rw�pg�Z�I�yS�Z9��I�)}5C{kE��o�j�=�����;q��4���G�QK�o`iG.d�����
q���Vk���'�\<i)9o/�4�M�\k,�6�4/\����8-�*;i��rv���jt���>"�&;����T$�S�N��������h�}O�I�F�S���1����V�$��6q��}����)��B����A(-����Ti��V'���B7�q��J���$��p���v���O��j��I8����7�O�'L)�e�J�Fo��������wG�6vvo���}]X�,�D��n����������d�������`��^�mL�F���C�F���y����*���pb�{�{��i���7���'�������B���yit��K���cW�����������PBF���y!N��6�����������7�lw����$/.�z����]KQ��������c��B�H��r.e��"�1=b3����em�y����A��a�����/W�K����E]�P,7�X��f,����g���M�w�^�J��L��I	�d�3&5�tg����P��7�]�gs�w�p���%Y
��C����5�B�}���PD���iK7�l�FctU�4i')�R�s<i=.^ �@oep�ub:��P�iW�3�N�I�����V���v����������fl�]C+�YN���J����I�W�r��������|��+���������R����Z*K����KK7n|�n6��?�#���J����i�Wr�h�������|��+���������R����i��K7n|�n6��?����\q5^'	�o�owZ����HQ�I����U/�������7�����r�3�6�(J���s�b����eX��[���3�jN��@��>���F���a���N"�ptY����{Y����l?��'�X�i�����p���U�e�9Z)]�tb��H�D3�[Z�P��9�V�yUO>_�Z��-���h�������1q�#;�����/����������W������������������W�^�_�5|��W�������W�����z�����������������O��������u�h���������du5���Nt������j����h��(�P���_�:,G��n����iK!Qa9�/>��=�����{����rtV��:���vn�����������^������\�yt3*���������Y8d���P�V�9o5���������`��������������+��i�������q��l�����[;wC���8zZ���Q>d����F��������I�i����{X��a����Z=���8d�xu����+��{�~����F5�s�W��X|W	���
��%#��"��>B�11����y
�Qi�����I
7�k2D�^]�+��_����3�2!�[�L�[�u�rrR(��
��7v��xv?Msz�V������8Z
�/}�������}e
�0�J���-��w�F;$]��f��
�����0\��ol�\�._���\���o\9z2�ms��wm��[�{��P���Ar�������r�pw�pg�*|�j�����������t=j���������N{(�ci���y4�Z���C�F��Umn�j��Y=~�h�$��%�X<���IK���`��K,�:��-�6��,\����8-�*��W9Nj������Z'�\�6!��F����Yh|����7��Zq�Y�t*��z��b�y����%���E�I��<�����G���j���y��t][��~��\�|������������[*�_D��+�Q����}�E��E����q��p��5��J���`�>���h7�!?�8��A��8~RIS��R��`�N���&�|F�Z��F�,���x�����J��y�Q
r��Ct~d�w7y���g�r�����!�C{\*�
�������2m�Cu���v�������Q2�L�7�A��<�n�m�P���R=��a�-���N�:��G��'tg/����
�No�9]�$�u�'�jg�<h���fCa4�0�\d2Ti��V'�������I�+iB�K�Z#k��n�B��j��I���<���F�����Q�<���d���}p����������.,U�V�[q���ROz���JTX��7����������Ewf�1�ZorP����u�F���y����B����&F�w�� �4�����w��+��^�\(��B/�n;6���]��^���G����b=�q��T^��q2����}]�X)=\,���8����x`���
f������%���z�%�{�_R���e1X$p�uE�~,d�^@,���5O�.>9j�?p�f]]-�wz�6:���E9
�����;�h/f�A\�u��&X�����Qv4T���i��5���@8��P��,������j����F�����$�%������w������u��K���8���U���V:��Z����<m�jv��]S����U��7K��$(���4c���`i����s\�~�C��?	��*@_���2�b�����s�\��|��}W���VK�bi���qi���o�����g�bQt\�~�C��?
��*�@����2�b�����s�\��|��}W���:��x���o�����g�b�v�+����$�����N��x��"�2���t��%���w�R�F��^y!�^��yf��E�Vsr.]�vr9�K�rK"s�r�[�����������^�9L���I���!�}�qx/K6#�����'|>c�$�!M6#�^uN;��J��o2'B+���N����hf}K+j<=��J4����+S�?�����-�^��7&�rd�S�;v ���U�������^����~��Q�������W����������O~���^��������#z�_�~��u�����w�������~]�.-�5�R��U�����F�����^t7Q�q�y�~�����[�����������4m)$*,G�������w�v�<��S���Y^������������{Q�~u��K�u]��+>�nF�r��YZ�x#W<��SxJ��;��F�<W�z���q�>�W������w�w��>m��T:��2n���\�V�pk�nH�yGO+��8��,�Q��h4;�q<�0>�:�~��v��:[>{\��Q�����n���r��w����z���{������"�x�^�d�7�B�W�G�7&�x��;�	7*�w��{Q�6����MF������P����w��&$y��I���UNN
��o�����n����iN����Q�W��h)d��y>��8��y
���R����?��'�q�='��Y��,�[mUQ1��fV�/����l�T��f��w�nx���B��(���$���o�����&������t7���;�b����tNHo����J�m5�����w�O�7&\�l|���`xk�~bS�=H*0(Z��9[��!������r��I�y���VN�v���Q34�V�<�&�v�C�j,m��7�IP�

Ga,q���H!�
�M���&��Z&�X<���IK��a���yb�����i�)��e=-���B��S��*g����F�����in��^����������*���F8��F[����Yh�����-�����J�r\��:/���a�z�����4��4O���������Z�$�b�kJkk��'��R)z~t�$~z-i�b����������zT,�l��/����5���{Uj�v�R�g����k�0vr�6��w��>��T�#r��;a���d�7�I7��c���(�sA��r����
���t�+��:<r�O8Q������+�t��b��v;���q�4r�����hJ#�u�l��2)�cM{�}��YeFF�Af�Z�"=k��Zuz���r�������������������i7�I�Q����������V�^{�y�����GI�5�O�?;����l���>9}��u�jv�����������n�����������6k'W���j�Zh���r�����{������m��3N���p����B�/N�������>(|0��Qg9*���}pVo=M�zz�V������7�������;&����j'���*�fx�j>�]��k��J�ETi�D���$�"�����.;��C-Cs�n��<%�L��6(v�>��u,a�����l?O�^!��ZW�����~��n�~����a��as��P\C�^��$�l���V;>�$�[��Q�m����N��_������_���9��7���!����Z5�����A�j����8�t��4���^	�����Q����>#��'
�'q#C
�^��<���)�*����nD�q��x�2���'��I�������l�j8�����9���h���
d���[�v�4n�N_���J'�F-�����0�T��6[�G�FhA���+��R���%���V�r\�w�{��!��O�����N�������
7HR�Y99J�V(�Q/(����INj�k����_�lo������[��+Q~<�|��%�-��w���jF��U��G|eNs�bF���d^~��L�wf��m�ew�=|���NN�P���������i2�IVzM1m=?��M�'����M��wN�������p����9�>?�O��F�tp�m�6mH
g�����o��w;f?�(� �h���t���~VK�ba����IoM�y���0	s������#=b�����1,Y��,}�8L��lx�/C�n��fl��u��������Y����w������e�]��nU:�O�\	�8�VW��6i[�sOV�+��������t��24�����2�I�&���/�
s�j�v�+l��T�nx&�+�*�r/��$�6.L8S&�$�W�so/�A���FV��Hw�X[�mvz_��djJ�z��zd�l6/������F��$-�0t��F���J�=�1m�Z��-�����V�W���H�������������{��P����v��W	-���m��Y����W�~v1]�A��������'���_^�k~�6�$�#��$g����J�����|Q�i-���>��^uZ���/�����;;�G[��[_e�6�1O���VI��Z��� �F*�������'���O�.u�+qh���
���$���i6_��n��9����{���S�EJ����F�|�|o����n���3c�{�fU`��� �{_���}8��n��Y���V������
�j�����cQ4��`�����mX��_�� ��\��#���l|^����F�{�������R�����0��]��.t��52~����%��N�0��.�j�
Lsh�ra�r$C���/\������0fnn��U������29O����$t�_�Q#�O�0B=m�C��a.�W�q�,N.k�O��b��x�������
��p��GOG���#=�
�
6T'=
�m	�V���H�;G��_e�=��:���8<�����L�����;p%�������&�4�����uK�nxv�~<���B�26,�����a��|��YT��"���V,��<�����������:���Z�\IF�;�~�~%�u[{o9LR�e;�z����OLV��(i9�
=�����j��<����djU�I��
���(-B�;2�e�b-]��9��}6�����s�����(��s2������vz�����I��I�W��k����l�g��7+��T��9�/�/���F�l���F��%��YV�m{�cx���6�]X�����V�4�Vz�-wO�s������n�4��pec����A�K3������������n���������.��/�l��������O��J56����;/t���a��aB�����i�`�+��'Yo��H�.���B�j�<����<�t�E��f��p�V��'mt�QO��;�tc>������>���j�����x����/��#_��c[idK��(mo�����<m�Q��3��s��K�'����JrV
]�{�X��?tr��n����9/��9��9�}w��=|>��������uOh�V}�ak���d���9��b��*��%������B�by��w�[f��im>���R�������0Y��rZ	������"��W�.4���}/4�L&M����Rw���6�kK���K���$�_�J���h%Z--�:��k�\%M9m$3�����i��[�Y��)��%u�R�[�di4ys{�V�hg7�wn�����W>,�g���n�(��I�Q!?���|?}��$��R��;��*���j�}c�U������d'3��KB��SI��8�Z�d�dd�����e�������������}qNH��y�q\�6m���)��K�t�G�d�ev�
�,����C^�U�)��4���
�2O/4�e1�D7�����K�dQ3-��{���r���jg6�$�p�K�V�/���.������|��py�r$������a����c��a�xN� LV�Z#������&��B��[���xw�������{���[oe��<�Y���%e!��rwL�^�L����w�i}3e�y���$������=�ax��w��Ra,��<����>+����`��K��l����1#�I�TGW1�����.�������Iwq���&
q�kz���p��h�Ew�tsq�������,���������,E�wR�����84c�:4c��;4W_�e���j��1����v�<��E6������;�;�7$��{7m4���Zt��
(��~�"�W��j���e�4n�7:5O���t�@Y������Y2�8o�W�������|/�,���������9����=���z/]����	�<I�����M�Az�F#{��Y�*����Vo�w�=
���|��P����r��]>C� L�:�Vw��`%���Wo����Ok���[�fgx�?k
�)g�A�m.�����jim}�����7�
m��]l�x��O���w�
�p�v�����l�>��e��I���G%t�����V������n�����v#:�W�V%�?��8����*Q����}���_�����A�3��^�S1�������]�W����~}^�$e�x���p������=��w�n����w���{�n}y���������������E�>����3���vO�o����1��F��a�<*�>p{�^t����~�s��Ya�5�B�M�f���mr�l����w��{���r'Tvp��q�y���U���������c�q���k^����X��n~X�^=��Io��o��~��y]L;�	7W��I�^�a�C/���o��z���i����l�=:=�Qy��K�;�.a)��{�xz���Y�]�K��YN|�c�y�1v��c���~rr�c�����i�}�Av��0�n��Av������������_��M���������og����(��Z����K��W?�������?�!��Z#��a�}������������j�M���3���0^�O�g���X��t������9bS�7�o�A�3�m�5���"��Z�|���)O������l��o,�G��t�)�����6���+i�����Z��Yh7[��$�p-�)���<x�Ar���tH��s�)���*���(_���'K������gf�w+>�u����U��&���W��3d���.o������o�?����F�W����Gcv!j_?��A������+���8V�������EC��8Sw�
k$D�]6�pjI���'>.��E%)�XT�z-���K��D�B�U�Z����b����;����76^�|�(�+����r�mf!�����)Tt(vi��<���N���DiP���#)����_�N�.�A��7�=*���]�+�{~�0Ka�Y5L��C�K������I!��n��*M�UZ�����Ju�1��;��L&���e�b������:�X��K�?�>�lz���I.>v�C_z���'�.����:�Y���a:��<zt�cD�>t��|^�����Y�������>�r��)���.�B�~����	Uxk�V|���=Zq����S.��=5q�s���?�G8���o��~x����<���?5o��N|j�����x�����s�i�!�q1�Z�e�B,��������Xyw���\�3]�3^�3Y�b
��[^��/���W@~, ?��G�6���d0�.��h��f���W�C���{���Q_���	,t�'��J����B1i��7�m.� ���i��3�7��3�7^���s��=������6/}��������~R����fmes�v������n��W��7,,1�����Ycz7���/2�IT��V��~,�<��.��:�k��G���u��\i�]KM��,5�������v�������V�%����������v�7��?���6o��i���m����q�������������3����G�����5����������7Z#���Gw��^k����d�g-�-��t������}����Qi���>�5��=L���<�|��?�t�zTL����sQ�O}�Y����gK��k'K�{W��0C9�WX�Z�b������p6,YW�v�����Ab��z�v>\������mh<'��r���a������N�[��u����g_P}M�b��0���&��I���W�z�z�QWF��`S�$93SLE���u��%�*^�|�9���K�r?�)�l�x�����[8KAh*"�za�s��dP�����PibO��I��0A�vd�;r�N�,Tr�sy2�������uu4��|f�&��������J�_;�"���,o���a���2)�!�
��F����B��R�\���$�����B����l�m�����@@�;>�T��6�s�vc��QFe��%?�1�5�$
#18Aleksander Alekseev
a.alekseev@postgrespro.ru
In reply to: Anthony Bykov (#17)
Re: Jsonb transform for pl/python

The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: tested, passed
Spec compliant: tested, passed
Documentation: tested, passed

LGTM.

The new status of this patch is: Ready for Committer

#19Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Aleksander Alekseev (#18)
Re: Jsonb transform for pl/python

On 1/12/18 10:43, Aleksander Alekseev wrote:

The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: tested, passed
Spec compliant: tested, passed
Documentation: tested, passed

LGTM.

The new status of this patch is: Ready for Committer

I've been working on polishing this a bit. I'll keep working on it. It
should be ready to commit soon.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#20Nikita Glukhov
n.gluhov@postgrespro.ru
In reply to: Peter Eisentraut (#19)
2 attachment(s)
Re: Jsonb transform for pl/python

On 01.02.2018 19:06, Peter Eisentraut wrote:

On 1/12/18 10:43, Aleksander Alekseev wrote:

The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: tested, passed
Spec compliant: tested, passed
Documentation: tested, passed

LGTM.

The new status of this patch is: Ready for Committer

I've been working on polishing this a bit. I'll keep working on it. It
should be ready to commit soon.

Hi.

I have reviewed this patch. Attached new 6th version of the patch with
v5-v6 delta-patch.

* Added out of memory checks after the following function calls:
- PyList_New()
- PyDict_New()
- PyString_FromStringAndSize() (added PyString_FromJsonbValue())

* Added Py_XDECREF() for key-value pairs and list elements after theirs
appending because PyDict_SetItem() and PyList_Append() do not steal
references (see also hstore_plpython code).

* Removed unnecessary JsonbValue heap-allocations in PyObject_ToJsonbValue().

* Added iterating to the end of iterator in PyObject_FromJsonb() for correct
freeing of JsonbIterators.

* Passed JsonbParseState ** to PyXxx_ToJsonbValue() functions.

* Added transformation of Python tuples into JSON arrays because standard
Python JSONEncoder encoder does the same.
(See https://docs.python.org/2/library/json.html#py-to-json-table)

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attachments:

0001-jsonb_plpython-extension-v6.patch.gzapplication/gzip; name=0001-jsonb_plpython-extension-v6.patch.gzDownload
0001-jsonb_plpython-extension-v5-v6-delta.patchtext/x-patch; name=0001-jsonb_plpython-extension-v5-v6-delta.patchDownload
diff --git a/contrib/jsonb_plpython/expected/jsonb_plpython2u.out b/contrib/jsonb_plpython/expected/jsonb_plpython2u.out
index 7073d52..7d29589 100644
--- a/contrib/jsonb_plpython/expected/jsonb_plpython2u.out
+++ b/contrib/jsonb_plpython/expected/jsonb_plpython2u.out
@@ -341,8 +341,22 @@ SELECT testDecimal();
  255
 (1 row)
 
+-- tuple -> jsonb
+CREATE FUNCTION testTuple() RETURNS jsonb
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+x = (1, 'String', None)
+return x
+$$;
+SELECT testTuple();
+      testtuple      
+---------------------
+ [1, "String", null]
+(1 row)
+
 DROP EXTENSION plpython2u CASCADE;
-NOTICE:  drop cascades to 17 other objects
+NOTICE:  drop cascades to 18 other objects
 DETAIL:  drop cascades to extension jsonb_plpython2u
 drop cascades to function test1(jsonb)
 drop cascades to function test1complex(jsonb)
@@ -360,6 +374,7 @@ drop cascades to function test1set()
 drop cascades to function testcomplexnumbers()
 drop cascades to function testrange()
 drop cascades to function testdecimal()
+drop cascades to function testtuple()
 -- test jsonb int -> python int
 CREATE EXTENSION jsonb_plpythonu CASCADE;
 NOTICE:  installing required extension "plpythonu"
diff --git a/contrib/jsonb_plpython/expected/jsonb_plpython3u.out b/contrib/jsonb_plpython/expected/jsonb_plpython3u.out
index 5acd45c..2492858 100644
--- a/contrib/jsonb_plpython/expected/jsonb_plpython3u.out
+++ b/contrib/jsonb_plpython/expected/jsonb_plpython3u.out
@@ -340,8 +340,22 @@ SELECT testDecimal();
  255
 (1 row)
 
+-- tuple -> jsonb
+CREATE FUNCTION testTuple() RETURNS jsonb
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+x = (1, 'String', None)
+return x
+$$;
+SELECT testTuple();
+      testtuple      
+---------------------
+ [1, "String", null]
+(1 row)
+
 DROP EXTENSION plpython3u CASCADE;
-NOTICE:  drop cascades to 17 other objects
+NOTICE:  drop cascades to 18 other objects
 DETAIL:  drop cascades to extension jsonb_plpython3u
 drop cascades to function test1(jsonb)
 drop cascades to function test1complex(jsonb)
@@ -359,3 +373,4 @@ drop cascades to function test1set()
 drop cascades to function testcomplexnumbers()
 drop cascades to function testrange()
 drop cascades to function testdecimal()
+drop cascades to function testtuple()
diff --git a/contrib/jsonb_plpython/jsonb_plpython.c b/contrib/jsonb_plpython/jsonb_plpython.c
index 28f7a3f..705e82f 100644
--- a/contrib/jsonb_plpython/jsonb_plpython.c
+++ b/contrib/jsonb_plpython/jsonb_plpython.c
@@ -21,7 +21,7 @@ static PyObject *decimal_constructor;
 
 static PyObject *PyObject_FromJsonb(JsonbContainer *jsonb);
 static JsonbValue *PyObject_ToJsonbValue(PyObject *obj,
-					  JsonbParseState *jsonb_state);
+					  JsonbParseState **jsonb_state, bool is_elem);
 
 #if PY_MAJOR_VERSION >= 3
 typedef PyObject *(*PLyUnicode_FromStringAndSize_t)
@@ -53,6 +53,43 @@ _PG_init(void)
 #define PLyUnicode_FromStringAndSize (PLyUnicode_FromStringAndSize_p)
 
 /*
+ * PyString_FromJsonbValue
+ *
+ * Transform string JsonbValue into python string
+ */
+static PyObject *
+PyString_FromJsonbValue(JsonbValue *jbv)
+{
+	PyObject   *str;
+
+	Assert(jbv->type == jbvString);
+
+	str = PyString_FromStringAndSize(jbv->val.string.val, jbv->val.string.len);
+
+	if (!str)
+		ereport(ERROR,
+				(errcode(ERRCODE_OUT_OF_MEMORY),
+				 errmsg("out of memory")));
+
+	return str;
+}
+
+/*
+ * PyString_ToJsonbValue
+ *
+ * Transform python string to JsonbValue
+ */
+static JsonbValue *
+PyString_ToJsonbValue(PyObject *obj, JsonbValue *jbvElem)
+{
+	jbvElem->type = jbvString;
+	jbvElem->val.string.val = PLyObject_AsString(obj);
+	jbvElem->val.string.len = strlen(jbvElem->val.string.val);
+
+	return jbvElem;
+}
+
+/*
  * PyObject_FromJsonbValue
  *
  * Transform JsonbValue into PyObject.
@@ -60,16 +97,14 @@ _PG_init(void)
 static PyObject *
 PyObject_FromJsonbValue(JsonbValue *jsonbValue)
 {
-	PyObject   *result;
-
 	switch (jsonbValue->type)
 	{
 		case jbvNull:
-			result = Py_None;
-			break;
+			return Py_None;
+
 		case jbvBinary:
-			result = PyObject_FromJsonb(jsonbValue->val.binary.data);
-			break;
+			return PyObject_FromJsonb(jsonbValue->val.binary.data);
+
 		case jbvNumeric:
 			{
 				Datum		num;
@@ -77,22 +112,20 @@ PyObject_FromJsonbValue(JsonbValue *jsonbValue)
 
 				num = NumericGetDatum(jsonbValue->val.numeric);
 				str = DatumGetCString(DirectFunctionCall1(numeric_out, num));
-				result = PyObject_CallFunction(decimal_constructor, "s", str);
-				break;
+
+				return PyObject_CallFunction(decimal_constructor, "s", str);
 			}
+
 		case jbvString:
-			result = PyString_FromStringAndSize(jsonbValue->val.string.val,
-												jsonbValue->val.string.len);
-			break;
+			return PyString_FromJsonbValue(jsonbValue);
+
 		case jbvBool:
-			result = jsonbValue->val.boolean ? Py_True : Py_False;
-			break;
-		case jbvArray:
-		case jbvObject:
-			result = PyObject_FromJsonb(jsonbValue->val.binary.data);
-			break;
+			return jsonbValue->val.boolean ? Py_True : Py_False;
+
+		default:
+			elog(ERROR, "unexpected jsonb value type: %d", jsonbValue->type);
+			return NULL;
 	}
-	return result;
 }
 
 /*
@@ -106,10 +139,7 @@ PyObject_FromJsonb(JsonbContainer *jsonb)
 	JsonbIteratorToken r;
 	JsonbValue	v;
 	JsonbIterator *it;
-
-	PyObject   *result,
-			   *key,
-			   *value;
+	PyObject   *result;
 
 	it = JsonbIteratorInit(jsonb);
 	r = JsonbIteratorNext(&it, &v, true);
@@ -119,47 +149,80 @@ PyObject_FromJsonb(JsonbContainer *jsonb)
 		case WJB_BEGIN_ARRAY:
 			if (v.val.array.rawScalar)
 			{
-				r = JsonbIteratorNext(&it, &v, true);
+				JsonbValue	tmp;
+
+				if ((r = JsonbIteratorNext(&it, &v, true)) != WJB_ELEM ||
+					(r = JsonbIteratorNext(&it, &tmp, true)) != WJB_END_ARRAY ||
+					(r = JsonbIteratorNext(&it, &tmp, true)) != WJB_DONE)
+					elog(ERROR, "unexpected jsonb token: %d", r);
+
 				result = PyObject_FromJsonbValue(&v);
 			}
 			else
 			{
 				/* array in v */
 				result = PyList_New(0);
-				while ((r = JsonbIteratorNext(&it, &v, true)) == WJB_ELEM)
-					PyList_Append(result, PyObject_FromJsonbValue(&v));
+				if (!result)
+					ereport(ERROR,
+							(errcode(ERRCODE_OUT_OF_MEMORY),
+							 errmsg("out of memory")));
+
+				while ((r = JsonbIteratorNext(&it, &v, true)) != WJB_DONE)
+				{
+					if (r == WJB_ELEM)
+					{
+						PyObject   *elem = PyObject_FromJsonbValue(&v);
+
+						PyList_Append(result, elem);
+						Py_XDECREF(elem);
+					}
+				}
 			}
 			break;
+
 		case WJB_BEGIN_OBJECT:
 			result = PyDict_New();
-			while ((r = JsonbIteratorNext(&it, &v, true)) == WJB_KEY)
+			if (!result)
+				ereport(ERROR,
+						(errcode(ERRCODE_OUT_OF_MEMORY),
+						 errmsg("out of memory")));
+
+			while ((r = JsonbIteratorNext(&it, &v, true)) != WJB_DONE)
 			{
-				key = PyString_FromStringAndSize(v.val.string.val,
-												 v.val.string.len);
-				r = JsonbIteratorNext(&it, &v, true);
-				value = PyObject_FromJsonbValue(&v);
-				PyDict_SetItem(result, key, value);
+				if (r == WJB_KEY)
+				{
+					PyObject   *key = PyString_FromJsonbValue(&v);
+
+					r = JsonbIteratorNext(&it, &v, true);
+
+					if (r == WJB_VALUE)
+					{
+						PyObject   *value = PyObject_FromJsonbValue(&v);
+
+						PyDict_SetItem(result, key, value);
+						Py_XDECREF(value);
+					}
+
+					Py_XDECREF(key);
+				}
 			}
 			break;
-		case WJB_END_OBJECT:
-			pg_unreachable();
-			break;
+
 		default:
-			result = PyObject_FromJsonbValue(&v);
-			break;
+			elog(ERROR, "unexpected jsonb token: %d", r);
+			return NULL;
 	}
+
 	return result;
 }
 
-
-
 /*
  * PyMapping_ToJsonbValue
  *
  * Transform python dict to JsonbValue.
  */
 static JsonbValue *
-PyMapping_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+PyMapping_ToJsonbValue(PyObject *obj, JsonbParseState **jsonb_state)
 {
 	int32		pcount;
 	JsonbValue *out = NULL;
@@ -174,21 +237,17 @@ PyMapping_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
 	{
 		int32		i;
 		PyObject   *items;
-		JsonbValue *jbvValue;
-		JsonbValue	jbvKey;
 
 		items = (PyObject *) items_v;
-		pushJsonbValue(&jsonb_state, WJB_BEGIN_OBJECT, NULL);
+
+		pushJsonbValue(jsonb_state, WJB_BEGIN_OBJECT, NULL);
 
 		for (i = 0; i < pcount; i++)
 		{
-			PyObject   *tuple,
-					   *key,
-					   *value;
-
-			tuple = PyList_GetItem(items, i);
-			key = PyTuple_GetItem(tuple, 0);
-			value = PyTuple_GetItem(tuple, 1);
+			JsonbValue	jbvKey;
+			PyObject   *tuple = PyList_GetItem(items, i),
+					   *key = PyTuple_GetItem(tuple, 0),
+					   *value = PyTuple_GetItem(tuple, 1);
 
 			/* Python dictionary can have None as key */
 			if (key == Py_None)
@@ -200,17 +259,14 @@ PyMapping_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
 			else
 			{
 				/* All others types of keys we serialize to string */
-				jbvKey.type = jbvString;
-				jbvKey.val.string.val = PLyObject_AsString(key);
-				jbvKey.val.string.len = strlen(jbvKey.val.string.val);
+				(void) PyString_ToJsonbValue(key, &jbvKey);
 			}
 
-			pushJsonbValue(&jsonb_state, WJB_KEY, &jbvKey);
-			jbvValue = PyObject_ToJsonbValue(value, jsonb_state);
-			if (IsAJsonbScalar(jbvValue))
-				pushJsonbValue(&jsonb_state, WJB_VALUE, jbvValue);
+			(void) pushJsonbValue(jsonb_state, WJB_KEY, &jbvKey);
+			(void) PyObject_ToJsonbValue(value, jsonb_state, false);
 		}
-		out = pushJsonbValue(&jsonb_state, WJB_END_OBJECT, NULL);
+
+		out = pushJsonbValue(jsonb_state, WJB_END_OBJECT, NULL);
 	}
 	PG_CATCH();
 	{
@@ -218,25 +274,8 @@ PyMapping_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
 		PG_RE_THROW();
 	}
 	PG_END_TRY();
-	return out;
-}
-
-/*
- * PyString_ToJsonbValue
- *
- * Transform python string to JsonbValue
- */
-static JsonbValue *
-PyString_ToJsonbValue(PyObject *obj)
-{
-	JsonbValue *jbvElem;
 
-	jbvElem = palloc(sizeof(JsonbValue));
-	jbvElem->type = jbvString;
-	jbvElem->val.string.val = PLyObject_AsString(obj);
-	jbvElem->val.string.len = strlen(jbvElem->val.string.val);
-
-	return jbvElem;
+	return out;
 }
 
 /*
@@ -246,27 +285,23 @@ PyString_ToJsonbValue(PyObject *obj)
  * a state required for jsonb construction.
  */
 static JsonbValue *
-PySequence_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+PySequence_ToJsonbValue(PyObject *obj, JsonbParseState **jsonb_state)
 {
-	JsonbValue *jbvElem;
 	Size		i,
 				pcount;
-	JsonbValue *out = NULL;
 
 	pcount = PySequence_Size(obj);
-	pushJsonbValue(&jsonb_state, WJB_BEGIN_ARRAY, NULL);
+
+	pushJsonbValue(jsonb_state, WJB_BEGIN_ARRAY, NULL);
 
 	for (i = 0; i < pcount; i++)
 	{
-		PyObject   *value;
+		PyObject   *value = PySequence_GetItem(obj, i);
 
-		value = PySequence_GetItem(obj, i);
-		jbvElem = PyObject_ToJsonbValue(value, jsonb_state);
-		if (IsAJsonbScalar(jbvElem))
-			pushJsonbValue(&jsonb_state, WJB_ELEM, jbvElem);
+		(void) PyObject_ToJsonbValue(value, jsonb_state, true);
 	}
-	out = pushJsonbValue(&jsonb_state, WJB_END_ARRAY, NULL);
-	return (out);
+
+	return pushJsonbValue(jsonb_state, WJB_END_ARRAY, NULL);
 }
 
 /*
@@ -275,11 +310,9 @@ PySequence_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
  * Transform python number to JsonbValue.
  */
 static JsonbValue *
-PyNumeric_ToJsonbValue(PyObject *obj)
+PyNumeric_ToJsonbValue(PyObject *obj, JsonbValue *jbvNum)
 {
-	volatile bool	failed = false;
 	Numeric		num;
-	JsonbValue *jbvInt;
 	char	   *str = PLyObject_AsString(obj);
 
 	PG_TRY();
@@ -289,21 +322,20 @@ PyNumeric_ToJsonbValue(PyObject *obj)
 	}
 	PG_CATCH();
 	{
-		failed = true;
-	}
-	PG_END_TRY();
-
-	if (failed)
 		ereport(ERROR,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
 				 (errmsg("plpython transformation error"),
-				  errdetail("the value \"%s\" cannot be transformed to jsonb", str))));
+				  errdetail("the value \"%s\" cannot be transformed to jsonb",
+							str))));
+	}
+	PG_END_TRY();
+
+	pfree(str);
 
-	jbvInt = palloc(sizeof(JsonbValue));
-	jbvInt->type = jbvNumeric;
-	jbvInt->val.numeric = num;
+	jbvNum->type = jbvNumeric;
+	jbvNum->val.numeric = num;
 
-	return jbvInt;
+	return jbvNum;
 }
 
 /*
@@ -312,39 +344,54 @@ PyNumeric_ToJsonbValue(PyObject *obj)
  * Transform python object to JsonbValue.
  */
 static JsonbValue *
-PyObject_ToJsonbValue(PyObject *obj, JsonbParseState *jsonb_state)
+PyObject_ToJsonbValue(PyObject *obj, JsonbParseState **jsonb_state, bool is_elem)
 {
-	JsonbValue *out = NULL;
+	JsonbValue	buf;
+	JsonbValue *out;
 
 	if (PyDict_Check(obj))
-		out = PyMapping_ToJsonbValue(obj, jsonb_state);
-	else if (PyString_Check(obj) || PyUnicode_Check(obj))
-		out = PyString_ToJsonbValue(obj);
-	else if (PyList_Check(obj))
-		out = PySequence_ToJsonbValue(obj, jsonb_state);
-	else if ((obj == Py_True) || (obj == Py_False))
+		return PyMapping_ToJsonbValue(obj, jsonb_state);
+	else if (PyList_Check(obj) || PyTuple_Check(obj))
+		return PySequence_ToJsonbValue(obj, jsonb_state);
+
+	/* Allocate JsonbValue in heap only if it is raw scalar value. */
+	out = *jsonb_state ? &buf : palloc(sizeof(JsonbValue));
+
+	if (PyString_Check(obj) || PyUnicode_Check(obj))
 	{
-		out = palloc(sizeof(JsonbValue));
-		out->type = jbvBool;
-		out->val.boolean = (obj == Py_True);
+		PyString_ToJsonbValue(obj, out);
 	}
 	else if (obj == Py_None)
 	{
-		out = palloc(sizeof(JsonbValue));
 		out->type = jbvNull;
 	}
+	/*
+	 * PyNumber_Check() returns true for booleans, so boolean conversion
+	 * should be placed before numeric conversion.
+	 */
+	else if (obj == Py_True || obj == Py_False)
+	{
+		out->type = jbvBool;
+		out->val.boolean = (obj == Py_True);
+	}
 	else if (PyNumber_Check(obj))
-		out = PyNumeric_ToJsonbValue(obj);
+	{
+		out = PyNumeric_ToJsonbValue(obj, out);
+	}
 	else
 	{
-		PyObject* repr = PyObject_Type(obj);
+		PyObject *repr = PyObject_Type(obj);
+
 		ereport(ERROR,
 				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
 				 (errmsg("plpython transformation error"),
-				  errdetail("\"%s\" type cannot be transformed to jsonb", PLyObject_AsString(repr)))));
+				  errdetail("\"%s\" type cannot be transformed to jsonb",
+							PLyObject_AsString(repr)))));
 	}
 
-	return out;
+	/* Push result into 'jsonb_state' unless it is raw scalar value. */
+	return *jsonb_state ?
+		pushJsonbValue(jsonb_state, is_elem ? WJB_ELEM : WJB_VALUE, out) : out;
 }
 
 /*
@@ -361,7 +408,7 @@ plpython_to_jsonb(PG_FUNCTION_ARGS)
 	JsonbParseState *jsonb_state = NULL;
 
 	obj = (PyObject *) PG_GETARG_POINTER(0);
-	out = PyObject_ToJsonbValue(obj, jsonb_state);
+	out = PyObject_ToJsonbValue(obj, &jsonb_state, true);
 	PG_RETURN_POINTER(JsonbValueToJsonb(out));
 }
 
diff --git a/contrib/jsonb_plpython/sql/jsonb_plpython2u.sql b/contrib/jsonb_plpython/sql/jsonb_plpython2u.sql
index b6f5fb9..81d10b4 100644
--- a/contrib/jsonb_plpython/sql/jsonb_plpython2u.sql
+++ b/contrib/jsonb_plpython/sql/jsonb_plpython2u.sql
@@ -212,6 +212,16 @@ $$;
 
 SELECT testDecimal();
 
+-- tuple -> jsonb
+CREATE FUNCTION testTuple() RETURNS jsonb
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+x = (1, 'String', None)
+return x
+$$;
+
+SELECT testTuple();
 
 DROP EXTENSION plpython2u CASCADE;
 
diff --git a/contrib/jsonb_plpython/sql/jsonb_plpython3u.sql b/contrib/jsonb_plpython/sql/jsonb_plpython3u.sql
index ff8a28a..a467fa4 100644
--- a/contrib/jsonb_plpython/sql/jsonb_plpython3u.sql
+++ b/contrib/jsonb_plpython/sql/jsonb_plpython3u.sql
@@ -212,4 +212,15 @@ $$;
 
 SELECT testDecimal();
 
+-- tuple -> jsonb
+CREATE FUNCTION testTuple() RETURNS jsonb
+LANGUAGE plpython2u
+TRANSFORM FOR TYPE jsonb
+AS $$
+x = (1, 'String', None)
+return x
+$$;
+
+SELECT testTuple();
+
 DROP EXTENSION plpython3u CASCADE;
#21Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Nikita Glukhov (#20)
Re: Jsonb transform for pl/python

On 3/12/18 11:26, Nikita Glukhov wrote:

I have reviewed this patch. Attached new 6th version of the patch with
v5-v6 delta-patch.

Thanks for the update. I'm working on committing this.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#22Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Peter Eisentraut (#21)
Re: Jsonb transform for pl/python

On 3/21/18 18:50, Peter Eisentraut wrote:

On 3/12/18 11:26, Nikita Glukhov wrote:

I have reviewed this patch. Attached new 6th version of the patch with
v5-v6 delta-patch.

Thanks for the update. I'm working on committing this.

Committed.

Everything seemed to work well. I just did a bit of cosmetic cleanups.
I did a fair amount of tweaking on the naming of functions, the
extensions and library names, etc., to make it look like the existing
plpython transforms. I also changed it so that the transform would
support mappings and sequences other than dict and list. I removed the
non-ASCII test and the test with the huge numbers.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#23Nikita Glukhov
n.gluhov@postgrespro.ru
In reply to: Peter Eisentraut (#22)
1 attachment(s)
Re: Jsonb transform for pl/python

On 28.03.2018 15:43, Peter Eisentraut wrote:

On 3/21/18 18:50, Peter Eisentraut wrote:

On 3/12/18 11:26, Nikita Glukhov wrote:

I have reviewed this patch. Attached new 6th version of the patch with
v5-v6 delta-patch.

Thanks for the update. I'm working on committing this.

Committed.

Everything seemed to work well. I just did a bit of cosmetic cleanups.
I did a fair amount of tweaking on the naming of functions, the
extensions and library names, etc., to make it look like the existing
plpython transforms. I also changed it so that the transform would
support mappings and sequences other than dict and list. I removed the
non-ASCII test and the test with the huge numbers.

I found a memory leak in PLySequence_ToJsonbValue():
PyObject returned from PySequence_GetItem() is not released.

A bug can be easily reproduced using function roudtrip() from regression test:
SELECT roundtrip('[1,2,3]'::jsonb) FROM generate_series(1, 1000000);

Similar code in PLyMapping_ToJsonbValue() seems to be correct because
PyList_GetItem() and PyTuple_GetItem() return a borrowed reference.

Patch with fix is attached.

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attachments:

0001-Fix-memory-leak-in-contrib-jsonb_plpython.patchtext/x-patch; name=0001-Fix-memory-leak-in-contrib-jsonb_plpython.patchDownload
From 53ab2aa2f345d7c4f5924b2e327b2d94c6f2c765 Mon Sep 17 00:00:00 2001
From: Nikita Glukhov <n.gluhov@postgrespro.ru>
Date: Fri, 15 Jun 2018 02:58:40 +0300
Subject: [PATCH] Fix memory leak in contrib/jsonb_plpython

---
 contrib/jsonb_plpython/jsonb_plpython.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/contrib/jsonb_plpython/jsonb_plpython.c b/contrib/jsonb_plpython/jsonb_plpython.c
index f752d6c..d6d6eeb 100644
--- a/contrib/jsonb_plpython/jsonb_plpython.c
+++ b/contrib/jsonb_plpython/jsonb_plpython.c
@@ -308,6 +308,8 @@ PLySequence_ToJsonbValue(PyObject *obj, JsonbParseState **jsonb_state)
 		PyObject   *value = PySequence_GetItem(obj, i);
 
 		(void) PLyObject_ToJsonbValue(value, jsonb_state, true);
+
+		Py_XDECREF(value);
 	}
 
 	return pushJsonbValue(jsonb_state, WJB_END_ARRAY, NULL);
-- 
2.7.4

#24Alexander Korotkov
a.korotkov@postgrespro.ru
In reply to: Nikita Glukhov (#23)
Re: Jsonb transform for pl/python

Hi!

On Fri, Jun 15, 2018 at 2:11 PM Nikita Glukhov <n.gluhov@postgrespro.ru> wrote:

On 28.03.2018 15:43, Peter Eisentraut wrote:

On 3/21/18 18:50, Peter Eisentraut wrote:

On 3/12/18 11:26, Nikita Glukhov wrote:

I have reviewed this patch. Attached new 6th version of the patch with
v5-v6 delta-patch.

Thanks for the update. I'm working on committing this.

Committed.

Everything seemed to work well. I just did a bit of cosmetic cleanups.
I did a fair amount of tweaking on the naming of functions, the
extensions and library names, etc., to make it look like the existing
plpython transforms. I also changed it so that the transform would
support mappings and sequences other than dict and list. I removed the
non-ASCII test and the test with the huge numbers.

I found a memory leak in PLySequence_ToJsonbValue():
PyObject returned from PySequence_GetItem() is not released.

A bug can be easily reproduced using function roudtrip() from regression test:
SELECT roundtrip('[1,2,3]'::jsonb) FROM generate_series(1, 1000000);

Similar code in PLyMapping_ToJsonbValue() seems to be correct because
PyList_GetItem() and PyTuple_GetItem() return a borrowed reference.

Patch with fix is attached.

I'm going to check and commit this if everything is OK.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

#25Nikita Glukhov
n.gluhov@postgrespro.ru
In reply to: Alexander Korotkov (#24)
2 attachment(s)
Re: Jsonb transform for pl/python

Hi!

We have found another performance problem in this transform -- very slow
conversion via I/O from PostgreSQL numerics (which are used for the
representation of jsonb numbers) to Python Decimals.

Attached patch with fix.

We are simply trying first to convert numeric to int64 if is does not have
digits after the decimal point, and then construct Python Int instead of
Decimal. Standard Python json.loads() does the same for exact integers.

A special function numeric_to_exact_int64() was added to numeric.c. Existing
numeric_int8() can't be used here because it rounds input numeric.

Performance results (see the second attached file jsonb_plplython_tests.sql
for the function definitions):

- calculating the length of the passed jsonb object (input transformation):

py_jsonb_length_trans opt 2761,873 ms
py_jsonb_length_trans 10419,230 ms
py_jsonb_length_json 8691,201 ms

- returning integer arrays (output transformation):

py_jsonb_ret_int_array_trans opt 3284,810 ms
py_jsonb_ret_int_array_trans 4540,063 ms
py_jsonb_ret_int_array_raw 5100,793 ms
py_jsonb_ret_int_array_json 9887,821 ms

- returning float arrays (output transformation):

py_jsonb_ret_float_array_trans opt 5699,360 ms
py_jsonb_ret_float_array_trans 5735,854 ms
py_jsonb_ret_float_array_raw 6516,514 ms
py_jsonb_ret_float_array_json 10869,213 ms

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attachments:

0001-Optimize-numeric-conversions-in-jsonb_plpython.patchtext/x-patch; name=0001-Optimize-numeric-conversions-in-jsonb_plpython.patchDownload
From ad090568f4eedf4cf0177f6ea5b7bb282f583bfb Mon Sep 17 00:00:00 2001
From: Nikita Glukhov <n.gluhov@postgrespro.ru>
Date: Thu, 21 Jun 2018 18:11:57 +0300
Subject: [PATCH] Optimize numeric conversions in jsonb_plpython

---
 contrib/jsonb_plpython/jsonb_plpython.c | 29 +++++++++++++++++++++++++++--
 src/backend/utils/adt/numeric.c         | 21 +++++++++++++++++++++
 src/include/utils/numeric.h             |  1 +
 3 files changed, 49 insertions(+), 2 deletions(-)

diff --git a/contrib/jsonb_plpython/jsonb_plpython.c b/contrib/jsonb_plpython/jsonb_plpython.c
index d6d6eeb..33ef15a 100644
--- a/contrib/jsonb_plpython/jsonb_plpython.c
+++ b/contrib/jsonb_plpython/jsonb_plpython.c
@@ -110,6 +110,13 @@ PLyObject_FromJsonbValue(JsonbValue *jsonbValue)
 			{
 				Datum		num;
 				char	   *str;
+				int64		intval;
+
+				if (numeric_to_exact_int64(jsonbValue->val.numeric, &intval))
+#ifndef HAVE_LONG_INT_64
+					if ((long) intval == intval)
+#endif
+						return PyInt_FromLong((long) intval);
 
 				num = NumericGetDatum(jsonbValue->val.numeric);
 				str = DatumGetCString(DirectFunctionCall1(numeric_out, num));
@@ -324,7 +331,26 @@ static JsonbValue *
 PLyNumber_ToJsonbValue(PyObject *obj, JsonbValue *jbvNum)
 {
 	Numeric		num;
-	char	   *str = PLyObject_AsString(obj);
+	char	   *str;
+
+	jbvNum->type = jbvNumeric;
+
+	if (PyInt_Check(obj))
+	{
+		long		val = PyInt_AsLong(obj);
+
+		if (val != -1 || !PyErr_Occurred())
+		{
+			jbvNum->val.numeric =
+				DatumGetNumeric(DirectFunctionCall1(int8_numeric,
+													Int64GetDatum((int64) val)));
+			return jbvNum;
+		}
+
+		PyErr_Clear();
+	}
+
+	str = PLyObject_AsString(obj);
 
 	PG_TRY();
 	{
@@ -356,7 +382,6 @@ PLyNumber_ToJsonbValue(PyObject *obj, JsonbValue *jbvNum)
 				(errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
 				 (errmsg("cannot convert NaN to jsonb"))));
 
-	jbvNum->type = jbvNumeric;
 	jbvNum->val.numeric = num;
 
 	return jbvNum;
diff --git a/src/backend/utils/adt/numeric.c b/src/backend/utils/adt/numeric.c
index 82a1429..eefaa6d 100644
--- a/src/backend/utils/adt/numeric.c
+++ b/src/backend/utils/adt/numeric.c
@@ -3348,6 +3348,27 @@ numeric_float4(PG_FUNCTION_ARGS)
 	PG_RETURN_DATUM(result);
 }
 
+/*
+ * numeric_get_int64() -
+ *
+ *	Try to convert numeric to int64 if it is an exact integer (i.e. it does
+ *	not have digits after the decimal point).  Return true if okay.
+ */
+bool
+numeric_to_exact_int64(Numeric num, int64 *result)
+{
+	NumericVar	var;
+
+	if (NUMERIC_IS_NAN(num))
+		return false;
+
+	if (NUMERIC_DSCALE(num) != 0)
+		return false;	/* digits after the decimal point are not allowed */
+
+	init_var_from_num(num, &var);
+
+	return numericvar_to_int64(&var, result);
+}
 
 /* ----------------------------------------------------------------------
  *
diff --git a/src/include/utils/numeric.h b/src/include/utils/numeric.h
index cd8da8b..ab49c0e 100644
--- a/src/include/utils/numeric.h
+++ b/src/include/utils/numeric.h
@@ -60,5 +60,6 @@ extern bool numeric_is_nan(Numeric num);
 int32		numeric_maximum_size(int32 typmod);
 extern char *numeric_out_sci(Numeric num, int scale);
 extern char *numeric_normalize(Numeric num);
+extern bool numeric_to_exact_int64(Numeric value, int64 *out);
 
 #endif							/* _PG_NUMERIC_H_ */
-- 
2.7.4

jsonb_plpython_tests.sqlapplication/sql; name=jsonb_plpython_tests.sqlDownload
#26Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Nikita Glukhov (#25)
Re: Jsonb transform for pl/python

On 6/23/18 01:44, Nikita Glukhov wrote:

We are simply trying first to convert numeric to int64 if is does not have
digits after the decimal point, and then construct Python Int instead of
Decimal. Standard Python json.loads() does the same for exact integers.

We just had a very similar but not identical discussion in the thread
about the PL/Perl transforms, where we rejected the proposal. Other
comments on this one?

In any case, this might well be a consideration for PG12, not a bug in
the existing implementation.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#27Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#26)
Re: Jsonb transform for pl/python

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

On 6/23/18 01:44, Nikita Glukhov wrote:

We are simply trying first to convert numeric to int64 if is does not have
digits after the decimal point, and then construct Python Int instead of
Decimal. Standard Python json.loads() does the same for exact integers.

We just had a very similar but not identical discussion in the thread
about the PL/Perl transforms, where we rejected the proposal. Other
comments on this one?

I can sympathize with the speed concern, but the proposed patch produces
a functional change (ie, you get a different kind of Python object, with
different behaviors, in some cases). That seems to me to be a good reason
to reject it. The fact that it makes the behavior vary depending on the
local width of "long" is also a problem, although I think that could be
fixed.

More generally, I'd be happier with a patch that sped things up for
non-integers too. I don't know whether Python exposes enough internals
of type Decimal to make that practical, though. One idea for doing it at
arm's length is to compute the Decimal value using numeric-digit-at-a-time
arithmetic, roughly

x := 0;
foreach(digit, numeric)
x.fma(10000, digit);
x.scaleb(appropriate-exponent);

In principle that's probably faster than string conversion, but not
sure by how much.

regards, tom lane

#28Alexander Korotkov
a.korotkov@postgrespro.ru
In reply to: Tom Lane (#27)
Re: Jsonb transform for pl/python

On Wed, Jul 11, 2018 at 12:30 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

On 6/23/18 01:44, Nikita Glukhov wrote:

We are simply trying first to convert numeric to int64 if is does not have
digits after the decimal point, and then construct Python Int instead of
Decimal. Standard Python json.loads() does the same for exact integers.

We just had a very similar but not identical discussion in the thread
about the PL/Perl transforms, where we rejected the proposal. Other
comments on this one?

I can sympathize with the speed concern, but the proposed patch produces
a functional change (ie, you get a different kind of Python object, with
different behaviors, in some cases). That seems to me to be a good reason
to reject it.

Nevertheless, as Nikita pointed, json.loads() performs the same in
Python. See an example for python 2.7.10.

import json
type(json.loads('1'))

<type 'int'>

type(json.loads('1.0'))

<type 'float'>

So, from postgres point of view this behavior is wrong. But on
another hand why don't let python transform to behave like a python?

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

#29Nikita Glukhov
n.gluhov@postgrespro.ru
In reply to: Alexander Korotkov (#24)
1 attachment(s)
Re: Jsonb transform for pl/python

Working on the new lazy transform for jsonb I found another memory leak in
PLyObject_ToJsonbValue(): palloc() for output boolean JsonbValue is unnecessary,
'out' variable is already initialized.

Fix is attached.

On 15.06.2018 14:42, Alexander Korotkov wrote:

Hi!

On Fri, Jun 15, 2018 at 2:11 PM Nikita Glukhov <n.gluhov@postgrespro.ru> wrote:

I found a memory leak in PLySequence_ToJsonbValue():
PyObject returned from PySequence_GetItem() is not released.

A bug can be easily reproduced using function roudtrip() from regression test:
SELECT roundtrip('[1,2,3]'::jsonb) FROM generate_series(1, 1000000);

Similar code in PLyMapping_ToJsonbValue() seems to be correct because
PyList_GetItem() and PyTuple_GetItem() return a borrowed reference.

Patch with fix is attached.
I'm going to check and commit this if everything is OK.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

--
Nikita Glukhov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Attachments:

0001-Fix-memory-leak-in-PLyObject_ToJsonbValue.patchtext/x-patch; name=0001-Fix-memory-leak-in-PLyObject_ToJsonbValue.patchDownload
From 06e22b5ff12486917259270c8f2a95b1a5d7cee2 Mon Sep 17 00:00:00 2001
From: Nikita Glukhov <n.gluhov@postgrespro.ru>
Date: Tue, 7 Aug 2018 13:37:10 +0300
Subject: [PATCH] Fix memory leak in PLyObject_ToJsonbValue()

---
 contrib/jsonb_plpython/jsonb_plpython.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/contrib/jsonb_plpython/jsonb_plpython.c b/contrib/jsonb_plpython/jsonb_plpython.c
index d6d6eeb..f44d364 100644
--- a/contrib/jsonb_plpython/jsonb_plpython.c
+++ b/contrib/jsonb_plpython/jsonb_plpython.c
@@ -398,7 +398,6 @@ PLyObject_ToJsonbValue(PyObject *obj, JsonbParseState **jsonb_state, bool is_ele
 	 */
 	else if (PyBool_Check(obj))
 	{
-		out = palloc(sizeof(JsonbValue));
 		out->type = jbvBool;
 		out->val.boolean = (obj == Py_True);
 	}
-- 
2.7.4

#30Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Nikita Glukhov (#29)
Re: Jsonb transform for pl/python

On 27/09/2018 16:58, Nikita Glukhov wrote:

Working on the new lazy transform for jsonb I found another memory leak in
PLyObject_ToJsonbValue(): palloc() for output boolean JsonbValue is unnecessary,
'out' variable is already initialized.

Fix is attached.

Committed, thanks.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services