Removing unreferenced files
Here is a cleaned-up version of the unreference file patch that was
discussed extensively in May of 2005. I want to get it into the
archives in case someone else want to work on it.
Here is a reference to the work still needed on the patch:
http://archives.postgresql.org/pgsql-patches/2005-05/msg00024.php
--
Bruce Momjian http://candle.pha.pa.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Attachments:
/pgpatches/checkfiles.difftext/x-diffDownload
Index: doc/src/sgml/maintenance.sgml
===================================================================
RCS file: /cvsroot/pgsql/doc/src/sgml/maintenance.sgml,v
retrieving revision 1.41
retrieving revision 1.42
diff -c -r1.41 -r1.42
*** doc/src/sgml/maintenance.sgml 20 Feb 2005 02:21:26 -0000 1.41
--- doc/src/sgml/maintenance.sgml 2 May 2005 18:26:52 -0000 1.42
***************
*** 474,479 ****
--- 474,496 ----
</para>
</sect1>
+ <sect1 id="check-files-after-crash">
+ <title>Check files after crash</title>
+
+ <indexterm zone="check-files-after-crash">
+ <primary>stale file</primary>
+ </indexterm>
+
+ <para>
+ <productname>PostgreSQL</productname> recovers automatically after crash
+ using the write-ahead log (see <xref linkend="wal">) and no manual
+ operations are normally needed. However, if there was a transaction running
+ when the crash occured that created or dropped a relation, the
+ transaction might have left a stale file in the data directory. If this
+ happens, you will get a notice in the log file stating which files can be
+ deleted.
+ </para>
+ </sect1>
<sect1 id="logfile-maintenance">
<title>Log File Maintenance</title>
Index: src/backend/access/transam/xlog.c
===================================================================
RCS file: /cvsroot/pgsql/src/backend/access/transam/xlog.c,v
retrieving revision 1.189
retrieving revision 1.190
diff -c -r1.189 -r1.190
*** src/backend/access/transam/xlog.c 28 Apr 2005 21:47:10 -0000 1.189
--- src/backend/access/transam/xlog.c 2 May 2005 18:26:52 -0000 1.190
***************
*** 43,48 ****
--- 43,49 ----
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/relcache.h"
+ #include "utils/flatfiles.h"
/*
***************
*** 4525,4530 ****
--- 4526,4533 ----
CreateCheckPoint(true, true);
+ CheckStaleRelFiles();
+
/*
* Close down recovery environment
*/
***************
*** 4536,4541 ****
--- 4539,4550 ----
*/
remove_backup_label();
}
+ else
+ {
+ XLogInitRelationCache();
+ CheckStaleRelFiles();
+ XLogCloseRelationCache();
+ }
/*
* Preallocate additional log files, if wanted.
Index: src/backend/catalog/catalog.c
===================================================================
RCS file: /cvsroot/pgsql/src/backend/catalog/catalog.c,v
retrieving revision 1.59
retrieving revision 1.60
diff -c -r1.59 -r1.60
*** src/backend/catalog/catalog.c 14 Apr 2005 20:03:23 -0000 1.59
--- src/backend/catalog/catalog.c 2 May 2005 18:26:53 -0000 1.60
***************
*** 106,111 ****
--- 106,144 ----
return path;
}
+ /*
+ * GetTablespacePath - construct path to a tablespace symbolic link
+ *
+ * Result is a palloc'd string.
+ *
+ * XXX this must agree with relpath and GetDatabasePath!
+ */
+ char *
+ GetTablespacePath(Oid spcNode)
+ {
+ int pathlen;
+ char *path;
+
+ Assert(spcNode != GLOBALTABLESPACE_OID);
+
+ if (spcNode == DEFAULTTABLESPACE_OID)
+ {
+ /* The default tablespace is {datadir}/base */
+ pathlen = strlen(DataDir) + 5 + 1;
+ path = (char *) palloc(pathlen);
+ snprintf(path, pathlen, "%s/base",
+ DataDir);
+ }
+ else
+ {
+ /* All other tablespaces have symlinks in pg_tblspc */
+ pathlen = strlen(DataDir) + 11 + OIDCHARS + 1;
+ path = (char *) palloc(pathlen);
+ snprintf(path, pathlen, "%s/pg_tblspc/%u",
+ DataDir, spcNode);
+ }
+ return path;
+ }
/*
* IsSystemRelation
Index: src/backend/commands/tablespace.c
===================================================================
RCS file: /cvsroot/pgsql/src/backend/commands/tablespace.c,v
retrieving revision 1.17
retrieving revision 1.18
diff -c -r1.17 -r1.18
*** src/backend/commands/tablespace.c 14 Apr 2005 20:03:24 -0000 1.17
--- src/backend/commands/tablespace.c 2 May 2005 18:26:53 -0000 1.18
***************
*** 341,348 ****
/*
* All seems well, create the symlink
*/
! linkloc = (char *) palloc(strlen(DataDir) + 11 + 10 + 1);
! sprintf(linkloc, "%s/pg_tblspc/%u", DataDir, tablespaceoid);
if (symlink(location, linkloc) < 0)
ereport(ERROR,
--- 341,347 ----
/*
* All seems well, create the symlink
*/
! linkloc = GetTablespacePath(tablespaceoid);
if (symlink(location, linkloc) < 0)
ereport(ERROR,
***************
*** 495,502 ****
char *subfile;
struct stat st;
! location = (char *) palloc(strlen(DataDir) + 11 + 10 + 1);
! sprintf(location, "%s/pg_tblspc/%u", DataDir, tablespaceoid);
/*
* Check if the tablespace still contains any files. We try to rmdir
--- 494,500 ----
char *subfile;
struct stat st;
! location = GetTablespacePath(tablespaceoid);
/*
* Check if the tablespace still contains any files. We try to rmdir
***************
*** 1036,1043 ****
set_short_version(location);
/* Create the symlink if not already present */
! linkloc = (char *) palloc(strlen(DataDir) + 11 + 10 + 1);
! sprintf(linkloc, "%s/pg_tblspc/%u", DataDir, xlrec->ts_id);
if (symlink(location, linkloc) < 0)
{
--- 1034,1040 ----
set_short_version(location);
/* Create the symlink if not already present */
! linkloc = GetTablespacePath(xlrec->ts_id);
if (symlink(location, linkloc) < 0)
{
Index: src/backend/utils/adt/misc.c
===================================================================
RCS file: /cvsroot/pgsql/src/backend/utils/adt/misc.c,v
retrieving revision 1.40
retrieving revision 1.41
diff -c -r1.40 -r1.41
*** src/backend/utils/adt/misc.c 31 Dec 2004 22:01:22 -0000 1.40
--- src/backend/utils/adt/misc.c 2 May 2005 18:26:53 -0000 1.41
***************
*** 26,31 ****
--- 26,32 ----
#include "funcapi.h"
#include "catalog/pg_type.h"
#include "catalog/pg_tablespace.h"
+ #include "catalog/catalog.h"
#define atooid(x) ((Oid) strtoul((x), NULL, 10))
***************
*** 144,154 ****
fctx = palloc(sizeof(ts_db_fctx));
- /*
- * size = path length + tablespace dirname length + 2 dir sep
- * chars + oid + terminator
- */
- fctx->location = (char *) palloc(strlen(DataDir) + 11 + 10 + 1);
if (tablespaceOid == GLOBALTABLESPACE_OID)
{
fctx->dirdesc = NULL;
--- 145,150 ----
***************
*** 157,168 ****
}
else
{
! if (tablespaceOid == DEFAULTTABLESPACE_OID)
! sprintf(fctx->location, "%s/base", DataDir);
! else
! sprintf(fctx->location, "%s/pg_tblspc/%u", DataDir,
! tablespaceOid);
!
fctx->dirdesc = AllocateDir(fctx->location);
if (!fctx->dirdesc)
--- 153,159 ----
}
else
{
! fctx->location = GetTablespacePath(tablespaceOid);
fctx->dirdesc = AllocateDir(fctx->location);
if (!fctx->dirdesc)
Index: src/backend/utils/init/Makefile
===================================================================
RCS file: /cvsroot/pgsql/src/backend/utils/init/Makefile,v
retrieving revision 1.18
retrieving revision 1.19
diff -c -r1.18 -r1.19
*** src/backend/utils/init/Makefile 20 Feb 2005 02:22:00 -0000 1.18
--- src/backend/utils/init/Makefile 2 May 2005 18:26:53 -0000 1.19
***************
*** 12,18 ****
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
! OBJS = flatfiles.o globals.o miscinit.o postinit.o
all: SUBSYS.o
--- 12,18 ----
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
! OBJS = flatfiles.o globals.o miscinit.o postinit.o checkfiles.o
all: SUBSYS.o
Index: src/include/catalog/catalog.h
===================================================================
RCS file: /cvsroot/pgsql/src/include/catalog/catalog.h,v
retrieving revision 1.30
retrieving revision 1.31
diff -c -r1.30 -r1.31
*** src/include/catalog/catalog.h 31 Dec 2004 22:03:24 -0000 1.30
--- src/include/catalog/catalog.h 2 May 2005 18:26:54 -0000 1.31
***************
*** 19,24 ****
--- 19,25 ----
extern char *relpath(RelFileNode rnode);
extern char *GetDatabasePath(Oid dbNode, Oid spcNode);
+ extern char *GetTablespacePath(Oid spcNode);
extern bool IsSystemRelation(Relation relation);
extern bool IsToastRelation(Relation relation);
Index: src/include/utils/flatfiles.h
===================================================================
RCS file: /cvsroot/pgsql/src/include/utils/flatfiles.h,v
retrieving revision 1.1
retrieving revision 1.2
diff -c -r1.1 -r1.2
*** src/include/utils/flatfiles.h 20 Feb 2005 02:22:07 -0000 1.1
--- src/include/utils/flatfiles.h 2 May 2005 18:26:54 -0000 1.2
***************
*** 30,33 ****
--- 30,36 ----
extern Datum flatfile_update_trigger(PG_FUNCTION_ARGS);
+ /* from checkfiles.c */
+ extern void CheckStaleRelFiles(void);
+
#endif /* FLATFILES_H */
Bruce Momjian <pgman@candle.pha.pa.us> writes:
Here is a cleaned-up version of the unreference file patch that was
discussed extensively in May of 2005. I want to get it into the
archives in case someone else want to work on it.Here is a reference to the work still needed on the patch:
http://archives.postgresql.org/pgsql-patches/2005-05/msg00024.php
Hello,
This is a TODO item, but I'd like to know if there's general interest in
this feature at the moment.
I think it can be useful in a situation when the DBA knows that the
database was shutdown or crashed during extensive data load and has a
way to proceed without the need to start over with the clean database.
In such a case having a tool to report or remove any stale files makes
perfect sense.
My other concern is that if at some point there is a bug in DROP
TABLE/INDEX that leaves the files it's supposed to remove, we would
likely want to know it.
Regardless of the outcome of this discussion it is interesting to know
if such functionality can only be programmed in backend/startup or
making it a separate tool is feasible (the tool will only run with the
stopped server)? There was once a tool named pgfsck[1]http://svana.org/kleptog/pgsql/pgfsck.html, which was never
updated after 8.2, so the name is already taken...
Yet another point is that one might be interested not only in reporting
stale files, but any *missing* ones too. Currently, you would only know
if a relation data file is missing when actually trying to read from it
and no attempt is made to verify this on startup. This might be not a
very useful check since if the file is not missing, but corrupted the
check doesn't buy you much. (I am likely kicking a dead horse here.)
Thank you.
--
Alex
[1]: http://svana.org/kleptog/pgsql/pgfsck.html
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers