Patch for 'Not to stuff everything as files in a single directory, hash dirs'

Started by Denis Perchineover 25 years ago6 messages

dyp@perchine.com

over 25 years ago

1 attachment(s)

Hello all,

I did small patch for subj. I am sure that it is not perfect, but it works for me.
I will continue its testing. This is my first patch to pgsql. If you will find some
obvious mistakes, do not flame, just show the right way.

If you have further suggestion/ideas, do not hesistate to contact me.

--
Sincerely Yours,
Denis Perchine

----------------------------------
E-Mail: dyp@perchine.com
HomePage: http://www.perchine.com/dyp/
FidoNet: 2:5000/120.5
----------------------------------

Attachments:

pgsql.lo.patchtext/x-c; name=pgsql.lo.patchDownload

Index: src/backend/catalog/catalog.c
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/backend/catalog/catalog.c,v
retrieving revision 1.32
diff -u -b -r1.32 catalog.c
--- src/backend/catalog/catalog.c	2000/04/12 17:14:55	1.32
+++ src/backend/catalog/catalog.c	2000/06/10 22:53:55
@@ -45,6 +45,37 @@
 		return path;
 	}
 
+	if ((strncmp(relname, "xin", 3) == 0) && (strlen(relname) > 4))
+	  if ((relname[3] == 'v') || (relname[3] == 'x')) {
+		/* LO's are resided in Database dir */
+		char *err;
+		int4 oid;
+		size_t bufsize;
+
+		oid = strtol(relname + 4, &err, 10);
+		if (*err == '\0') {
+			int pw = 0;
+			int4 s_oid = oid;
+			while (s_oid != 0) {
+			  pw++;
+			  s_oid /= 100;
+			}
+			s_oid = oid / 100;
+			bufsize = strlen(relname) + pw * 3 + 4;
+			path = (char *) palloc(sizeof(char) * bufsize);
+			path[0] = relname[3];
+			path[1] = SEP_CHAR;
+			path[2] = 0;
+			pw = 2;
+			while (s_oid != 0) {
+			  snprintf(path + pw, 4, "%02d%c", s_oid % 100, SEP_CHAR);
+			  pw += 3;
+			  s_oid /= 100;
+			}
+			snprintf(path + pw, bufsize - pw, "%s", relname);
+			return path;
+		}
+	  }
 	/*
 	 * If it is in the current database, assume it is in current working
 	 * directory.  NB: this does not work during bootstrap!
Index: src/backend/storage/large_object/inv_api.c
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/backend/storage/large_object/inv_api.c,v
retrieving revision 1.69
diff -u -b -r1.69 inv_api.c
--- src/backend/storage/large_object/inv_api.c	2000/06/05 07:28:45	1.69
+++ src/backend/storage/large_object/inv_api.c	2000/06/10 22:54:06
@@ -34,6 +34,8 @@
 #include "utils/fmgroids.h"
 #include "utils/relcache.h"
 
+#include <errno.h>
+
 /*
  *	Warning, Will Robinson...  In order to pack data into an inversion
  *	file as densely as possible, we violate the class abstraction here.
@@ -96,6 +98,9 @@
 	Oid			classObjectId[1];
 	char		objname[NAMEDATALEN];
 	char		indname[NAMEDATALEN];
+	Oid		s_oid;
+	int		pw = 0;
+	size_t		dirsize;
 
 	/*
 	 * add one here since the pg_class tuple created will have the next
@@ -132,6 +137,44 @@
 					   BYTEAOID,
 					   -1, 0, false);
 
+	s_oid = file_oid;
+	while (s_oid != 0) {
+	        pw++;
+		s_oid /= 100;
+	}
+	dirsize = pw * 3 + 2;
+	
+	{
+		char		dirname[dirsize];
+		struct stat	st;
+		
+		dirname[1] = 0;
+		pw = 1;
+		s_oid = file_oid;
+		for(; ; ) {
+			dirname[0] = 'v';
+			if (stat(dirname, &st) == -1) {
+				if (errno == ENOENT) {
+					if (mkdir(dirname, S_IRWXU) != 0)
+				                elog(ERROR, "lo_create: unable to create large object directory '%s': %s", dirname, strerror(errno));
+				} else
+				        elog(ERROR, "lo_create: unable to stat large object directory '%s': %s", dirname, strerror(errno));
+			}
+			dirname[0] = 'x';
+			if (stat(dirname, &st) == -1) {
+				if (errno == ENOENT) {
+					if (mkdir(dirname, S_IRWXU) != 0)
+				                elog(ERROR, "lo_create: unable to create large object directory '%s': %s", dirname, strerror(errno));
+				} else
+				        elog(ERROR, "lo_create: unable to stat large object directory '%s': %s", dirname, strerror(errno));
+			}
+			s_oid /= 100;
+			if (s_oid == 0)
+				break;
+			snprintf(dirname + pw, 4, "%c%02d", SEP_CHAR, s_oid % 100);
+			pw += 3;
+		}
+	}
 	/*
 	 * First create the table to hold the inversion large object.  It will
 	 * be located on whatever storage manager the user requested.

Bruce Momjian

pgman@candle.pha.pa.us

over 25 years ago

In reply to: Denis Perchine (#1)

Re: Patch for 'Not to stuff everything as files in a single directory, hash dirs'

Seems the whole large object per file is going away in 7.1. Can someone
confirm this?

Hello all,

I did small patch for subj. I am sure that it is not perfect, but it works for me.
I will continue its testing. This is my first patch to pgsql. If you will find some
obvious mistakes, do not flame, just show the right way.

If you have further suggestion/ideas, do not hesistate to contact me.

--
Sincerely Yours,
Denis Perchine

----------------------------------
E-Mail: dyp@perchine.com
HomePage: http://www.perchine.com/dyp/
FidoNet: 2:5000/120.5
----------------------------------

[ Attachment, skipping... ]

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Denis Perchine

dyp@perchine.com

over 25 years ago

In reply to: Denis Perchine (#1)

Re: Patch for 'Not to stuff everything as files in a single directory, hash dirs'

I did small patch for subj. I am sure that it is not perfect, but it works for me.
I will continue its testing. This is my first patch to pgsql. If you will find some
obvious mistakes, do not flame, just show the right way.

If you have further suggestion/ideas, do not hesistate to contact me.

Forget about this. Better patch posted to pgsql-patches.

--
Sincerely Yours,
Denis Perchine

----------------------------------
E-Mail: dyp@perchine.com
HomePage: http://www.perchine.com/dyp/
FidoNet: 2:5000/120.5
----------------------------------

Noname

JanWieck@t-online.de

over 25 years ago

In reply to: Bruce Momjian (#2)

Re: Patch for 'Not to stuff everything as files in a single directory, hash dirs''

Bruce Momjian wrote:

Seems the whole large object per file is going away in 7.1. Can someone
confirm this?

Not the whole one in 7.1.

The TOAST stuff will lower the need for large objects alot,
but we already discovered the fact that it isn't a real
answer to LARGE objects.

First of all, the entire datum must be properly quoted to fit
into a querystring. Therefore the client needs to have the
original datum, the qouted copy, the querystring it built.
Then the querystring is sent to the backend, parsed (where a
CONST node is built from it), copied into a tuple to be split
up into TOAST items.

So on a central system, where client and DB are both running,
we have 6 copies of the object in memory! Not that optimal.

For 7.2 I'll work on real CLOB and BLOB data types. Requires
some more thinking though.

Jan

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Bruce Momjian

pgman@candle.pha.pa.us

over 25 years ago

In reply to: Noname (#4)

Re: Patch for 'Not to stuff everything as files in a single directory, hash dirs''

Bruce Momjian wrote:

Seems the whole large object per file is going away in 7.1. Can someone
confirm this?

Not the whole one in 7.1.

The TOAST stuff will lower the need for large objects alot,
but we already discovered the fact that it isn't a real
answer to LARGE objects.

First of all, the entire datum must be properly quoted to fit
into a querystring. Therefore the client needs to have the
original datum, the qouted copy, the querystring it built.
Then the querystring is sent to the backend, parsed (where a
CONST node is built from it), copied into a tuple to be split
up into TOAST items.

So on a central system, where client and DB are both running,
we have 6 copies of the object in memory! Not that optimal.

For 7.2 I'll work on real CLOB and BLOB data types. Requires
some more thinking though.

I thought we would keep the existing large object interface, but allow
storage of large object data directly in fields using TOAST.

-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian

pgman@candle.pha.pa.us

over 25 years ago

In reply to: Noname (#4)

Re: Patch for 'Not to stuff everything as files in a single directory, hash dirs''

My idea was to implement the large object API on top of TOAST.

Bruce Momjian wrote:

Seems the whole large object per file is going away in 7.1. Can someone
confirm this?

Not the whole one in 7.1.

The TOAST stuff will lower the need for large objects alot,
but we already discovered the fact that it isn't a real
answer to LARGE objects.

First of all, the entire datum must be properly quoted to fit
into a querystring. Therefore the client needs to have the
original datum, the qouted copy, the querystring it built.
Then the querystring is sent to the backend, parsed (where a
CONST node is built from it), copied into a tuple to be split
up into TOAST items.

So on a central system, where client and DB are both running,
we have 6 copies of the object in memory! Not that optimal.

For 7.2 I'll work on real CLOB and BLOB data types. Requires
some more thinking though.

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026