"End-to-end" paper
For those of you who have missed it, here
is the paper some of us mention, "END-TO-END ARGUMENTS IN SYSTEM DESIGN"
by Saltzer, Reed, and Clark.
The abstract is:
This paper presents a design principle that helps guide placement of
functions among the modules of a distributed computer system. The
principle, called the end-to-end argument, suggests that functions
placed at low levels of a system may be redundant or of little value
when compared with the cost of providing them at that low level.
Examples discussed in the paper include bit error recovery, security
using encryption, duplicate message suppression, recovery from
system crashes, and delivery acknowledgement. Low level mechanisms
to support these functions are justified only as performance
enhancements.
It was written in 1981 and is undiminished by the subsequent decades.
Nathan Myers
ncm@zembu.com
At 12:24 AM 17-05-2001 -0700, Nathan Myers wrote:
For those of you who have missed it, here
http://www.google.com/search?q=cache:web.mit.edu/Saltzer/www/publications/e
ndtoend/endtoend.pdf+clark+end+to+end&hl=en
is the paper some of us mention, "END-TO-END ARGUMENTS IN SYSTEM DESIGN"
by Saltzer, Reed, and Clark.The abstract is:
This paper presents a design principle that helps guide placement of
functions among the modules of a distributed computer system. The
principle, called the end-to-end argument, suggests that functions
placed at low levels of a system may be redundant or of little value
when compared with the cost of providing them at that low level.
Examples discussed in the paper include bit error recovery, security
using encryption, duplicate message suppression, recovery from
system crashes, and delivery acknowledgement. Low level mechanisms
to support these functions are justified only as performance
enhancements.It was written in 1981 and is undiminished by the subsequent decades.
Maybe I don't understand the paper.
The end-to-end argument might be true if taking the monolithic approach. I
find more useful ideas gleaned from the RFCs, TCP/IP and the OSI 7 layer
model: modularity, "useful standard interfaces", "Be liberal in what you
accept, and conservative in what you send" and so on.
Within a module I figure the end to end argument might hold, but the author
keeps talking about networks and networking.
SSL and TCP are useful. The various CRC checks down the IP stack to the
datalink layer have their uses too.
By splitting stuff up at appropriate points, adding or substituting objects
at various layers becomes so much easier. People can download Postgresql
over token ring, Gigabit ethernet, X.25 and so on.
Splitting stuff up does mean that the bits and pieces now do have a certain
responsibility. If those responsibilities involve some redundancies in
error checking or encryption or whatever, so be it, because if done well
people can use those bits and pieces in interesting ways never dreamed of
initially.
For example SSL over TCP over IPSEC over encrypted WAP works (even though
IPSEC is way too complicated :)). There's so much redundancy there, but at
the same time it's not a far fetched scenario - just someone ordering
online on a notebook pc.
But if a low level module never bothered with error
correction/detection/handling or whatever and was optimized for an
application specific purpose, it's harder to use it for other purposes. And
if you do, some chap could post an article to Bugtraq on it, mentioning
exploit, DoS or buffer overflow.
Cheerio,
Link.
On Thu, May 17, 2001 at 06:04:54PM +0800, Lincoln Yeoh wrote:
At 12:24 AM 17-05-2001 -0700, Nathan Myers wrote:
For those of you who have missed it, here
is the paper some of us mention, "END-TO-END ARGUMENTS IN SYSTEM DESIGN"
by Saltzer, Reed, and Clark.The abstract is:
This paper presents a design principle that helps guide placement
of functions among the modules of a distributed computer system.
The principle, called the end-to-end argument, suggests that
functions placed at low levels of a system may be redundant or
of little value when compared with the cost of providing them
at that low level. Examples discussed in the paper include
bit error recovery, security using encryption, duplicate
message suppression, recovery from system crashes, and delivery
acknowledgement. Low level mechanisms to support these functions
are justified only as performance enhancements.It was written in 1981 and is undiminished by the subsequent decades.
Maybe I don't understand the paper.
Yes. It bears re-reading.
The end-to-end argument might be true if taking the monolithic
approach. I find more useful ideas gleaned from the RFCs, TCP/IP and
the OSI 7 layer model: modularity, "useful standard interfaces", "Be
liberal in what you accept, and conservative in what you send" and so
on.
The end-to-end principle has had profound effects on the design of
Internet protocols, perhaps most importantly in keeping them simpler
than OSI's.
Within a module I figure the end to end argument might hold,
The end-to-end principle isn't particularly applicable within a module.
It's a system-design principle. Its prescription for individual modules
is: don't imagine that anybody else gets much value from your complex
error recovery shenanigans; they have to do their own error recovery
anyway. You provide more value by making a good effort.
but the author keeps talking about networks and networking.
Of course networking is just an example, but it's a particularly
good example. Data storage (e.g. disk) is another good example; in
the context of the paper it may be thought of as a mechanism for
communicating with other (later) times. The point there is that the CRCs
and ECC performed by the disk are not sufficient to ensure reliability
for the system (e.g. database service); for that, end-to-end measures
such as hot-failover, backups, redo logs, and block- or record-level
CRCs are needed. The purpose of the disk CRCs is not reliability, a job
they cannot do alone, but performance: they help make the need to use
the backups and redo logs infrequent enough to be tolerable.
SSL and TCP are useful. The various CRC checks down the IP stack to
the datalink layer have their uses too.
Yes, of course they are useful. The authors say so in the paper, and
they say precisely how (and how not).
By splitting stuff up at appropriate points, adding or substituting
objects at various layers becomes so much easier. People can download
Postgresql over token ring, Gigabit ethernet, X.25 and so on.
As noted in the paper, the principle is most useful in helping to decide
what goes in each layer.
Splitting stuff up does mean that the bits and pieces now do have
a certain responsibility. If those responsibilities involve some
redundancies in error checking or encryption or whatever, so be
it, because if done well people can use those bits and pieces in
interesting ways never dreamed of initially.For example SSL over TCP over IPSEC over encrypted WAP works (even
though IPSEC is way too complicated :)). There's so much redundancy
there, but at the same time it's not a far fetched scenario - just
someone ordering online on a notebook pc.
The authors quote a similar example in the paper, even though it was
written twenty years ago.
But if a low level module never bothered with error
correction/detection/handling or whatever and was optimized for
an application specific purpose, it's harder to use it for other
purposes. And if you do, some chap could post an article to Bugtraq on
it, mentioning exploit, DoS or buffer overflow.
The point is that leaving that stuff _out_ is how you keep low-level
mechanisms useful for a variety of purposes. Putting in complicated
error-recovery stuff might suit it better for a particular application,
but make it less suitable for others.
This is why, at the IP layer, packets get tossed at the first sign of
congestion. It's why TCP connections often get dropped at the first sign
of a data-format violation. This is a very deep principle; understanding
it thoroughly will make you a much better system designer.
Nathan Myers
ncm@zembu.com
On OBSD from cvs source, clean checkout:
gcc -O2 -pipe -Wall -Wmissing-prototypes -Wmissing-declarations
-I../../../../src/include -DLIBDIR=\"/usr/local/pgsql/lib\
" -DDLSUFFIX=\".so\" -c -o dfmgr.o dfmgr.c
dfmgr.c: In function `load_external_function':
dfmgr.c:118: `RTLD_GLOBAL' undeclared (first use in this function)
dfmgr.c:118: (Each undeclared identifier is reported only once
dfmgr.c:118: for each function it appears in.)
gmake[4]: *** [dfmgr.o] Error 1
gmake[4]: Leaving directory
`/home/bpalmer/APPS/pgsql/src/backend/utils/fmgr'
?? RTLD_GLOBAL problems?
- b
b. palmer, bpalmer@crimelabs.net
pgp: www.crimelabs.net/bpalmer.pgp5
This seems to have been broken for a few days ~monday night
- brandon
b. palmer, bpalmer@crimelabs.net
pgp: www.crimelabs.net/bpalmer.pgp5
On Sat, May 19, 2001 at 08:03:50PM -0400, bpalmer wrote:
On OBSD from cvs source, clean checkout:
gcc -O2 -pipe -Wall -Wmissing-prototypes -Wmissing-declarations
-I../../../../src/include -DLIBDIR=\"/usr/local/pgsql/lib\
" -DDLSUFFIX=\".so\" -c -o dfmgr.o dfmgr.c
dfmgr.c: In function `load_external_function':
dfmgr.c:118: `RTLD_GLOBAL' undeclared (first use in this function)
dfmgr.c:118: (Each undeclared identifier is reported only once
dfmgr.c:118: for each function it appears in.)
gmake[4]: *** [dfmgr.o] Error 1
gmake[4]: Leaving directory
`/home/bpalmer/APPS/pgsql/src/backend/utils/fmgr'?? RTLD_GLOBAL problems?
Not a solution, but a few data points: I had a successful build from cvs of
May 19 16:21 GMT under NetBSD/i386, and for me RTLD_GLOBAL is defined in
/usr/include/dlfcn.h ie., system header file, not postgresql.
Hope that helps,
Patrick
This was added by Peter E to allow PL/Perl to compile.
On Sat, May 19, 2001 at 08:03:50PM -0400, bpalmer wrote:
On OBSD from cvs source, clean checkout:
gcc -O2 -pipe -Wall -Wmissing-prototypes -Wmissing-declarations
-I../../../../src/include -DLIBDIR=\"/usr/local/pgsql/lib\
" -DDLSUFFIX=\".so\" -c -o dfmgr.o dfmgr.c
dfmgr.c: In function `load_external_function':
dfmgr.c:118: `RTLD_GLOBAL' undeclared (first use in this function)
dfmgr.c:118: (Each undeclared identifier is reported only once
dfmgr.c:118: for each function it appears in.)
gmake[4]: *** [dfmgr.o] Error 1
gmake[4]: Leaving directory
`/home/bpalmer/APPS/pgsql/src/backend/utils/fmgr'?? RTLD_GLOBAL problems?
Not a solution, but a few data points: I had a successful build from cvs of
May 19 16:21 GMT under NetBSD/i386, and for me RTLD_GLOBAL is defined in
/usr/include/dlfcn.h ie., system header file, not postgresql.Hope that helps,
Patrick
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026