Large scale reliable software system
Dear all,
After greeting,
I taught PostgreSQL myself and developed a small scale experimental
software system using PostgreSQL in the back-end.
I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i can share
the storage with the different system users where they login to the system
through the web application front-end with different passwords and
usernames , save the privacy of each user data, improve overall system
security and performance, achieve fast response, make backups and save the
stored data from loss. The system will be hosted on a cloud.
Thank you in advance.
On 6/26/23 16:48, B M wrote:
Dear all,
After greeting,
I taught PostgreSQL myself and developed a small scale
experimentalsoftware system using PostgreSQL in the back-end.I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i can
share the storage with the different system users where they login to
the system through the web application front-end with different
passwords and usernames , save the privacy of each user data, improve
overall system security and performance, achieve fast response, make
backups and save the stored data from loss. The system will be hosted on
a cloud.
https://www.djangoproject.com/
Thank you in advance.
--
Adrian Klaver
adrian.klaver@aklaver.com
This is a reasonable answer, but I want to offer a caveat.
Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development community that the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby or Python or whatever back end code.
This heuristic is nearly always mostly wrong.
Guyren G Howe
Show quoted text
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:
On 6/26/23 16:48, B M wrote:
Dear all,
After greeting,
I taught PostgreSQL myself and developed a small scale
experimentalsoftware system using PostgreSQL in the back-end.I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i can
share the storage with the different system users where they login to
the system through the web application front-end with different
passwords and usernames , save the privacy of each user data, improve
overall system security and performance, achieve fast response, make
backups and save the stored data from loss. The system will be hosted on
a cloud.https://www.djangoproject.com/
Thank you in advance.
--
Adrian Klaver
adrian.klaver@aklaver.com
It's not just Ruby, dumb databases are preferred in projects like
WordPress, Drupal and Joomla, too.
Now, if it's because they're used to using MySQL, well maybe that's
not so hard to understand. :-)
Show quoted text
On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:
This is a reasonable answer, but I want to offer a caveat.
Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development community that the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby or Python or whatever back end code.
This heuristic is nearly always mostly wrong.
Guyren G Howe
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:On 6/26/23 16:48, B M wrote:
Dear all,
After greeting,
I taught PostgreSQL myself and developed a small scale
experimentalsoftware system using PostgreSQL in the back-end.I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i can
share the storage with the different system users where they login to
the system through the web application front-end with different
passwords and usernames , save the privacy of each user data, improve
overall system security and performance, achieve fast response, make
backups and save the stored data from loss. The system will be hosted on
a cloud.https://www.djangoproject.com/
Thank you in advance.
--
Adrian Klaver
adrian.klaver@aklaver.com
Correct. It’s a tragically wrong piece of folk wisdom that’s pretty general across web development communities.
Show quoted text
On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:
It's not just Ruby, dumb databases are preferred in projects like
WordPress, Drupal and Joomla, too.Now, if it's because they're used to using MySQL, well maybe that's
not so hard to understand. :-)On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:
This is a reasonable answer, but I want to offer a caveat.
Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development community that the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby or Python or whatever back end code.
This heuristic is nearly always mostly wrong.
Guyren G Howe
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:On 6/26/23 16:48, B M wrote:
Dear all,
After greeting,
I taught PostgreSQL myself and developed a small scale
experimentalsoftware system using PostgreSQL in the back-end.I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i can
share the storage with the different system users where they login to
the system through the web application front-end with different
passwords and usernames , save the privacy of each user data, improve
overall system security and performance, achieve fast response, make
backups and save the stored data from loss. The system will be hosted on
a cloud.https://www.djangoproject.com/
Thank you in advance.
--
Adrian Klaver
adrian.klaver@aklaver.com
Correct. It’s a tragically wrong piece of folk wisdom that’s pretty
general across web development communities.
So, what's your advice, and is there some book / resource to get upto speed?
Thanks!
On Tue, Jun 27, 2023 at 10:38 AM Guyren Howe <guyren@gmail.com> wrote:
Show quoted text
Correct. It’s a tragically wrong piece of folk wisdom that’s pretty
general across web development communities.On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:
It's not just Ruby, dumb databases are preferred in projects like
WordPress, Drupal and Joomla, too.Now, if it's because they're used to using MySQL, well maybe that's
not so hard to understand. :-)On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:
This is a reasonable answer, but I want to offer a caveat.
Likely because of the influence of the originator of Ruby on Rails, it is
close to holy writ in the web development community that the database must
be treated as a dumb data bucket and all business logic must be implemented
in the Ruby or Python or whatever back end code.This heuristic is nearly always mostly wrong.
Guyren G Howe
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>,
wrote:On 6/26/23 16:48, B M wrote:
Dear all,
After greeting,
I taught PostgreSQL myself and developed a small scale
experimentalsoftware system using PostgreSQL in the back-end.I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i can
share the storage with the different system users where they login to
the system through the web application front-end with different
passwords and usernames , save the privacy of each user data, improve
overall system security and performance, achieve fast response, make
backups and save the stored data from loss. The system will be hosted on
a cloud.https://www.djangoproject.com/
Thank you in advance.
--
Adrian Klaver
adrian.klaver@aklaver.com
On Tue, 27 Jun 2023 at 07:08, Guyren Howe <guyren@gmail.com> wrote:
Correct. It’s a tragically wrong piece of folk wisdom that’s pretty
general across web development communities.On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:
It's not just Ruby, dumb databases are preferred in projects like
WordPress, Drupal and Joomla, too.Now, if it's because they're used to using MySQL, well maybe that's
not so hard to understand. :-)On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:
This is a reasonable answer, but I want to offer a caveat.
Likely because of the influence of the originator of Ruby on Rails, it is
close to holy writ in the web development community that the database must
be treated as a dumb data bucket and all business logic must be implemented
in the Ruby or Python or whatever back end code.This heuristic is nearly always mostly wrong.
Guyren G Howe
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>,
wrote:On 6/26/23 16:48, B M wrote:
--
Adrian Klaver
adrian.klaver@aklaver.comThe accepted front-end developer wisdom of treating the DB as a dumb data
store works under conditions, for example the DB will never be accessed
from a different ORM / framework, and where the performance attributes of
using an ORM with 'standard' datastructures are acceptable.
The moment you need to plug in something like reporting tools, or access
from a different system / API / framework / language / ORM or whatever, the
approach not having rules / views / procedures / whatever built into the
database falls apart.
Other things to consider are performance / load / overhead: we have one
system that involves processing through large amounts of data for reports /
queries. Shipping all that back through the ORM / db interface (ODBC /
JDBC / psycopg2 / whatever for resolution / filtering on the front end
application where SQL / procedures / views could do that in the DB and just
ship back the required data seems counterproductive.
Tony Shelver
Show quoted text
Well, seeing as postgres isn't designed to serve http requests or to run
general purpose code that doesn't involve databases, which you can express
elegantly in python, to answer OP's question - my vote is on the original
answer - Django. It's got everything out of the box - authentication. file
storage. etc etc.
Once you get the application running you can enhance the database as
necessary.
On Tue, Jun 27, 2023 at 4:19 PM Tony Shelver <tshelver@gmail.com> wrote:
Show quoted text
On Tue, 27 Jun 2023 at 07:08, Guyren Howe <guyren@gmail.com> wrote:
Correct. It’s a tragically wrong piece of folk wisdom that’s pretty
general across web development communities.On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:
It's not just Ruby, dumb databases are preferred in projects like
WordPress, Drupal and Joomla, too.Now, if it's because they're used to using MySQL, well maybe that's
not so hard to understand. :-)On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:
This is a reasonable answer, but I want to offer a caveat.
Likely because of the influence of the originator of Ruby on Rails, it is
close to holy writ in the web development community that the database must
be treated as a dumb data bucket and all business logic must be implemented
in the Ruby or Python or whatever back end code.This heuristic is nearly always mostly wrong.
Guyren G Howe
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>,
wrote:On 6/26/23 16:48, B M wrote:
--
Adrian Klaver
adrian.klaver@aklaver.comThe accepted front-end developer wisdom of treating the DB as a dumb
data store works under conditions, for example the DB will never be
accessed from a different ORM / framework, and where the performance
attributes of using an ORM with 'standard' datastructures are acceptable.The moment you need to plug in something like reporting tools, or access
from a different system / API / framework / language / ORM or whatever, the
approach not having rules / views / procedures / whatever built into the
database falls apart.Other things to consider are performance / load / overhead: we have one
system that involves processing through large amounts of data for reports /
queries. Shipping all that back through the ORM / db interface (ODBC /
JDBC / psycopg2 / whatever for resolution / filtering on the front end
application where SQL / procedures / views could do that in the DB and just
ship back the required data seems counterproductive.Tony Shelver
On Mon, Jun 26, 2023 at 6:49 PM B M <bmmasprep@gmail.com> wrote:
Dear all,
After greeting,
I taught PostgreSQL myself and developed a small scale experimental
software system using PostgreSQL in the back-end.I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i can share
the storage with the different system users where they login to the system
through the web application front-end with different passwords and
usernames , save the privacy of each user data, improve overall system
security and performance, achieve fast response, make backups and save the
stored data from loss. The system will be hosted on a cloud.Thank you in advance.
* your sql is code, and treat it as such, check it into git etc
* robust deployment strategy is essential to scaling team
* write a lot of tests
* become intimate with pg_stat_statements
* keep your transactions as short as possible while preserving safety
* avoid developers who advocate for keeping business logic out of the
database religiously (controversial)
* try to avoid assumptions that only one technical stack interacts with
your database
* do not waste time making ERDs use a tool that generates them (i like
schemaspy)
* test your DR strategy before disaster strikes
* think about security model up front
merlin
On 6/27/23 07:58, Merlin Moncure wrote:
On Mon, Jun 26, 2023 at 6:49 PM B M <bmmasprep@gmail.com
<mailto:bmmasprep@gmail.com>> wrote:Dear all,
After greeting,
I taught PostgreSQL myself and developed a small scale
experimentalsoftware system using PostgreSQL in the back-end.I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i
can share the storage with the different system users where they
login to the system through the web application front-end with
different passwords and usernames , save the privacy of each user
data, improve overall system security and performance, achieve fast
response, make backups and save the stored data from loss. The
system will be hosted on a cloud.Thank you in advance.
* your sql is code, and treat it as such, check it into git etc
* robust deployment strategy is essential to scaling team
* write a lot of tests
* become intimate with pg_stat_statements
* keep your transactions as short as possible while preserving safety
* avoid developers who advocate for keeping business logic out of the
database religiously (controversial)
* try to avoid assumptions that only one technical stack interacts with
your database
* do not waste time making ERDs use a tool that generates them (i like
schemaspy)
* test your DR strategy before disaster strikes
* think about security model up front
+1
merlin
--
Adrian Klaver
adrian.klaver@aklaver.com