[GSOC 18] Performance Farm Project——Initialization Project

Hi

Maybe I’m missing something (I’ve been offline a lot recently for
unavoidable reasons), but the perf farm project already has a Django
backend initialised and configured to work with community auth, on
community infrastructure.

https://git.postgresql.org/gitweb/?p=pgperffarm.git;a=summary

On Sunday, March 11, 2018, Hongyuan Ma <cs_maleicacid@163.com> wrote:

Hello, mark
I initialized a Django project and imported the Django REST Framework. Its
github address is: https://github.com/PGPerfFarm/server-code
I created some model classes and also wrote scripts in the dbtools folder
to import simulation data for development. I am hesitant to use admin or
xadmin as the administrator's backend for the site (I prefer to use xadmin).

I also initialized the website's front-end application. Here is its
address: https://github.com/PGPerfFarm/front-end-code.git
I wrote the header component as shown:

I hope this effect can enhance the future user experience:).
This application uses vue.js and element-ui, but if you insist on using
other js libraries or not using js, please let me know. I will empty this
project and then rewrite it as you wish.

My next step is to determine the necessary api interface and the
corresponding components of the front-end application. Then implement them
one by one.
Finally, as you can see, I created an organization named PGPerfFarm on
github without permission. I sent you an invitation letter, and after you
join, I am willing to hand over the administrator of the organization to
you.

Regards,
Hongyuan Ma (CS_MaleicAcid@163.com)

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi Dave,
Thank you for your reminder. This is indeed my negligence.
In fact, I have browsed the code in the pgperffarm.git repository, including the client folder and the web folder. However, I found that the web folder has some unfinished html files and does not contain model class files. And the project used Django 1.8 without importing the Django REST Framework (When I talked to Mark about the PerfFarm project, he told me he insisted on using Django 1.11 and considered using the REST framework). So I mistakenly thought that the code in the web folder had been shelved.

In the newly initialized Django application, I upgraded its version and assigned the directory to make the project structure clearer. I want to use a separate front-end development approach (The front-end applications will use vue.js for programming.). I hope you can allow me to use it instead of the old one. I am willing to integrate the auth module into this new application as soon as possible.

Regards,
Hongyuan Ma (CS_MaleicAcid@163.com)

在 2018-03-12 02:26:43，"Dave Page" <dpage@pgadmin.org> 写道：
Hi

Maybe I’m missing something (I’ve been offline a lot recently for unavoidable reasons), but the perf farm project already has a Django backend initialised and configured to work with community auth, on community infrastructure.

https://git.postgresql.org/gitweb/?p=pgperffarm.git;a=summary

On Sunday, March 11, 2018, Hongyuan Ma <cs_maleicacid@163.com> wrote:

Hello, mark
I initialized a Django project and imported the Django REST Framework. Its github address is: https://github.com/PGPerfFarm/server-code
I created some model classes and also wrote scripts in the dbtools folder to import simulation data for development. I am hesitant to use admin or xadmin as the administrator's backend for the site (I prefer to use xadmin).

I also initialized the website's front-end application. Here is its address: https://github.com/PGPerfFarm/front-end-code.git
I wrote the header component as shown:

I hope this effect can enhance the future user experience:).
This application uses vue.js and element-ui, but if you insist on using other js libraries or not using js, please let me know. I will empty this project and then rewrite it as you wish.

My next step is to determine the necessary api interface and the corresponding components of the front-end application. Then implement them one by one.
Finally, as you can see, I created an organization named PGPerfFarm on github without permission. I sent you an invitation letter, and after you join, I am willing to hand over the administrator of the organization to you.

Regards,
Hongyuan Ma (CS_MaleicAcid@163.com)

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi

On Mon, Mar 12, 2018 at 9:57 AM, Hongyuan Ma <cs_maleicacid@163.com> wrote:

Hi Dave,
Thank you for your reminder. This is indeed my negligence.
In fact, I have browsed the code in the pgperffarm.git
<https://git.postgresql.org/gitweb/?p=pgperffarm.git;a=summary&gt; repository,
including the client folder and the web folder. However, I found that the
web folder has some unfinished html files and does not contain model class
files. And the project used Django 1.8 without importing the Django REST
Framework (When I talked to Mark about the PerfFarm project, he told me he
insisted on using Django 1.11 and considered using the REST framework). So
I mistakenly thought that the code in the web folder had been shelved.

Nope, not at all. It just wasn't updated to meet the latest PG
infrastructure requirements yet (basically just the update to Django 11).
The rest should be fine as-is.

In the newly initialized Django application, I upgraded its version and
assigned the directory to make the project structure clearer. I want to use
a separate front-end development approach (The front-end applications will
use vue.js for programming.). I hope you can allow me to use it instead of
the old one. I am willing to integrate the auth module into this new
application as soon as possible.

I would much prefer to see jQuery + React, purely because there are likely
more PostgreSQL developers (particularly from the pgAdmin team) that know
those technologies. It is important to consider long term maintenance as
well as ease of initial development with any project.

Thanks.

Regards,
Hongyuan Ma (CS_MaleicAcid@163.com)

在 2018-03-12 02:26:43，"Dave Page" <dpage@pgadmin.org> 写道：

Hi

Maybe I’m missing something (I’ve been offline a lot recently for
unavoidable reasons), but the perf farm project already has a Django
backend initialised and configured to work with community auth, on
community infrastructure.

https://git.postgresql.org/gitweb/?p=pgperffarm.git;a=summary

On Sunday, March 11, 2018, Hongyuan Ma <cs_maleicacid@163.com> wrote:

Hello, mark
I initialized a Django project and imported the Django REST Framework.
Its github address is: https://github.com/PGPerfFarm/server-code
I created some model classes and also wrote scripts in the dbtools folder
to import simulation data for development. I am hesitant to use admin or
xadmin as the administrator's backend for the site (I prefer to use xadmin).

I also initialized the website's front-end application. Here is its
address: https://github.com/PGPerfFarm/front-end-code.git
I wrote the header component as shown:

I hope this effect can enhance the future user experience:).
This application uses vue.js and element-ui, but if you insist on using
other js libraries or not using js, please let me know. I will empty this
project and then rewrite it as you wish.

My next step is to determine the necessary api interface and the
corresponding components of the front-end application. Then implement them
one by one.
Finally, as you can see, I created an organization named PGPerfFarm on
github without permission. I sent you an invitation letter, and after you
join, I am willing to hand over the administrator of the organization to
you.

Regards,
Hongyuan Ma (CS_MaleicAcid@163.com)

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi Dave,
I am willing to use React to re-create the front-end application. Since I plan to use separate front-end and back-end development methods, this means that there will be no html files in Django applications. Front-end and back-end applications will interact via restful api. I will use react to rewrite some existing html files if needed. Before initializing the react application, I want to learn more about your tendency toward front-end technology.

- About React: Which version of React (15.x or 16) and react-router (2.x or 4) do you tend to use?
- About Package Manager: Do you tend to use yarn or npm?
- About the UI library: I guess you might prefer Bootstrap or Material-UI.

At the same time I hope you can help me understand the functional requirements of this project more clearly. Here are some of my thoughts on PerfFarm:

- I see this comment in the client folder (In line 15 of the "pgperffarm \ client \ benchmarks \ pgbench.py" file):
'''
# TODO allow running custom scripts, not just the default
'''
Will PerfFarm have many test items so that the test item search function or even the test item sort function is expected to be provided?

- What value will be used to determine if a machine's performance is improving or declining?
- After the user logs in to the site, I think it may be necessary to provide a page for the user to browse or manage his own machine. Is there any function that requires users to log in before they can use it?
- When I registered a machine on BuildFarm, I did not receive the confirmation email immediately but several days later. In PerfFarm, when a user registers a machine, will the administrator review it before sending a confirmation email?
- I see BuildFarm assigning an animal name to each registered machine. Will PerfFarm also have this interesting feature?

My postgresql.org community account is:
- Username: maleicacid
- Email: cs_maleicacid@163.com

I hope to get the commit permission of the pgperffarm.git repository. I am willing to continue coding and complete the project on the existing code.Looking forward to your reply.

Best Regards,
Hongyuan Ma (CS_MaleicAcid@163.com)

At 2018-03-13 03:03:08, "Dave Page" <dpage@pgadmin.org> wrote:

Hi

On Mon, Mar 12, 2018 at 9:57 AM, Hongyuan Ma <cs_maleicacid@163.com> wrote:

Hi Dave,
Thank you for your reminder. This is indeed my negligence.
In fact, I have browsed the code in the pgperffarm.git repository, including the client folder and the web folder. However, I found that the web folder has some unfinished html files and does not contain model class files. And the project used Django 1.8 without importing the Django REST Framework (When I talked to Mark about the PerfFarm project, he told me he insisted on using Django 1.11 and considered using the REST framework). So I mistakenly thought that the code in the web folder had been shelved.

Nope, not at all. It just wasn't updated to meet the latest PG infrastructure requirements yet (basically just the update to Django 11). The rest should be fine as-is.

In the newly initialized Django application, I upgraded its version and assigned the directory to make the project structure clearer. I want to use a separate front-end development approach (The front-end applications will use vue.js for programming.). I hope you can allow me to use it instead of the old one. I am willing to integrate the auth module into this new application as soon as possible.

I would much prefer to see jQuery + React, purely because there are likely more PostgreSQL developers (particularly from the pgAdmin team) that know those technologies. It is important to consider long term maintenance as well as ease of initial development with any project.

Thanks.

Regards,
Hongyuan Ma (CS_MaleicAcid@163.com)

在 2018-03-12 02:26:43，"Dave Page" <dpage@pgadmin.org> 写道：
Hi

Maybe I’m missing something (I’ve been offline a lot recently for unavoidable reasons), but the perf farm project already has a Django backend initialised and configured to work with community auth, on community infrastructure.

https://git.postgresql.org/gitweb/?p=pgperffarm.git;a=summary

On Sunday, March 11, 2018, Hongyuan Ma <cs_maleicacid@163.com> wrote:

Hello, mark
I initialized a Django project and imported the Django REST Framework. Its github address is: https://github.com/PGPerfFarm/server-code
I created some model classes and also wrote scripts in the dbtools folder to import simulation data for development. I am hesitant to use admin or xadmin as the administrator's backend for the site (I prefer to use xadmin).

I also initialized the website's front-end application. Here is its address: https://github.com/PGPerfFarm/front-end-code.git
I wrote the header component as shown:

I hope this effect can enhance the future user experience:).
This application uses vue.js and element-ui, but if you insist on using other js libraries or not using js, please let me know. I will empty this project and then rewrite it as you wish.

My next step is to determine the necessary api interface and the corresponding components of the front-end application. Then implement them one by one.
Finally, as you can see, I created an organization named PGPerfFarm on github without permission. I sent you an invitation letter, and after you join, I am willing to hand over the administrator of the organization to you.

Regards,
Hongyuan Ma (CS_MaleicAcid@163.com)

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi

On Tue, Mar 13, 2018 at 11:31 PM, Hongyuan Ma <cs_maleicacid@163.com> wrote:

Hi Dave,
I am willing to use React to re-create the front-end application. Since I
plan to use separate front-end and back-end development methods, this means
that there will be no html files in Django applications. Front-end and
back-end applications will interact via restful api. I will use react to
rewrite some existing html files if needed. Before initializing the react
application, I want to learn more about your tendency toward front-end
technology.

- About React: Which version of React (15.x or 16) and react-router (2.x
or 4) do you tend to use?

pgAdmin is using React 16.2.

- About Package Manager: Do you tend to use yarn or npm?

Yarn for pgAdmin, though npm would probably be a little easier for a
project running on pginfra.

- About the UI library: I guess you might prefer Bootstrap or
Material-UI.

Bootstrap.

At the same time I hope you can help me understand the functional
requirements of this project more clearly. Here are some of my thoughts
on PerfFarm:

- I see this comment in the client folder (In line 15 of the "pgperffarm
\ client \ benchmarks \ pgbench.py" file):
'''
# TODO allow running custom scripts, not just the default
'''
Will PerfFarm have many test items so that the test item search function
or even the test item sort function is expected to be provided?

I don't know - Tomas or Mark are probably the best folks to ask about that.
I spent some time on the initial web app work, but then ran out of spare
cycles.

- What value will be used to determine if a machine's performance is
improving or declining?

Good question - I think that needs discussion, as it's not necessarily
clear-cut when you think that performance may vary at different numbers of
clients.

- After the user logs in to the site, I think it may be necessary to
provide a page for the user to browse or manage his own machine. Is there
any function that requires users to log in before they can use it?

Yes, that! We want the system to be self-service for registered users. The
only interaction required by admins should be to approve new machines in my
opinion.

- When I registered a machine on BuildFarm, I did not receive the
confirmation email immediately but several days later. In PerfFarm, when
a user registers a machine, will the administrator review it before sending
a confirmation email?

See above. It should be automated though - i.e. if an admin approves the
registration of a new machine, the system sends a welcome email with any
required info.

- I see BuildFarm assigning an animal name to each registered machine. Will
PerfFarm also have this interesting feature?

It was going to, but I forget what we decided on for a naming scheme!
Another discussion item I think - in the code, we can just use "Name" or
similar.

My postgresql.org community account is:
- Username: maleicacid
- Email: cs_maleicacid@163.com

I hope to get the commit permission of the pgperffarm.git
<https://git.postgresql.org/gitweb/?p=pgperffarm.git;a=summary&gt;
repository. I am willing to continue coding and complete the project on the
existing code.Looking forward to your reply.

Typically PostgreSQL projects only give commit rights to developers who
have submitted a number of good quality patches (in the case of PostgreSQL
and pgAdmin, that can be over the course of years!). For the perffarm, I
expect the bar to be a little lower than that, but for now you should
assume you'll be sending patches for review.

Thanks.

Best Regards,
Hongyuan Ma (CS_MaleicAcid@163.com)

At 2018-03-13 03:03:08, "Dave Page" <dpage@pgadmin.org> wrote:

Hi

On Mon, Mar 12, 2018 at 9:57 AM, Hongyuan Ma <cs_maleicacid@163.com>
wrote:

Hi Dave,
Thank you for your reminder. This is indeed my negligence.
In fact, I have browsed the code in the pgperffarm.git
<https://git.postgresql.org/gitweb/?p=pgperffarm.git;a=summary&gt; repository,
including the client folder and the web folder. However, I found that the
web folder has some unfinished html files and does not contain model class
files. And the project used Django 1.8 without importing the Django REST
Framework (When I talked to Mark about the PerfFarm project, he told me he
insisted on using Django 1.11 and considered using the REST framework). So
I mistakenly thought that the code in the web folder had been shelved.

Nope, not at all. It just wasn't updated to meet the latest PG
infrastructure requirements yet (basically just the update to Django 11).
The rest should be fine as-is.

In the newly initialized Django application, I upgraded its version and
assigned the directory to make the project structure clearer. I want to use
a separate front-end development approach (The front-end applications will
use vue.js for programming.). I hope you can allow me to use it instead of
the old one. I am willing to integrate the auth module into this new
application as soon as possible.

I would much prefer to see jQuery + React, purely because there are likely
more PostgreSQL developers (particularly from the pgAdmin team) that know
those technologies. It is important to consider long term maintenance as
well as ease of initial development with any project.

Thanks.

Regards,
Hongyuan Ma (CS_MaleicAcid@163.com)

在 2018-03-12 02:26:43，"Dave Page" <dpage@pgadmin.org> 写道：

Hi

Maybe I’m missing something (I’ve been offline a lot recently for
unavoidable reasons), but the perf farm project already has a Django
backend initialised and configured to work with community auth, on
community infrastructure.

https://git.postgresql.org/gitweb/?p=pgperffarm.git;a=summary

On Sunday, March 11, 2018, Hongyuan Ma <cs_maleicacid@163.com> wrote:

Hello, mark
I initialized a Django project and imported the Django REST Framework.
Its github address is: https://github.com/PGPerfFarm/server-code
I created some model classes and also wrote scripts in the dbtools
folder to import simulation data for development. I am hesitant to use
admin or xadmin as the administrator's backend for the site (I prefer to
use xadmin).

I also initialized the website's front-end application. Here is its
address: https://github.com/PGPerfFarm/front-end-code.git
I wrote the header component as shown:

I hope this effect can enhance the future user experience:).
This application uses vue.js and element-ui, but if you insist on using
other js libraries or not using js, please let me know. I will empty this
project and then rewrite it as you wish.

My next step is to determine the necessary api interface and the
corresponding components of the front-end application. Then implement them
one by one.
Finally, as you can see, I created an organization named PGPerfFarm on
github without permission. I sent you an invitation letter, and after you
join, I am willing to hand over the administrator of the organization to
you.

Regards,
Hongyuan Ma (CS_MaleicAcid@163.com)

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi,

I have some additional comments to a couple areas...

On Wed, Mar 14, 2018 at 05:33:00PM -0400, Dave Page wrote:

On Tue, Mar 13, 2018 at 11:31 PM, Hongyuan Ma <cs_maleicacid@163.com> wrote:

At the same time I hope you can help me understand the functional
requirements of this project more clearly. Here are some of my thoughts
on PerfFarm:

- I see this comment in the client folder (In line 15 of the "pgperffarm
\ client \ benchmarks \ pgbench.py" file):
'''
# TODO allow running custom scripts, not just the default
'''
Will PerfFarm have many test items so that the test item search function
or even the test item sort function is expected to be provided?

I don't know - Tomas or Mark are probably the best folks to ask about that.
I spent some time on the initial web app work, but then ran out of spare
cycles.

Yes, there is potential for the number of tests to change over time.
(Maybe a test is less relevant, maybe there is a more useful test down
he road.)

I'm haven't thought too much about sorting and searching functions. But
right now there is just pgbench with 6 basic configurations: read/write,
read-only, and a low, medium, and high data set size for each of those.

- What value will be used to determine if a machine's performance is
improving or declining?

Good question - I think that needs discussion, as it's not necessarily
clear-cut when you think that performance may vary at different numbers of
clients.

My thought here is that each test should provide a single metric,
defined by the test itself, so expect this to be define in the test
results sent to the web site.

I do expect that off of the test results should be archived somehow and
retrievable. But the idea I have is to have a generic high level view
that all of the different tests can share.

- I see BuildFarm assigning an animal name to each registered machine. Will
PerfFarm also have this interesting feature?

It was going to, but I forget what we decided on for a naming scheme!
Another discussion item I think - in the code, we can just use "Name" or
similar.

Since the buildfarm is composed of animals, I thought plants would be a
complimentary scheme? I'm also not sure if this was discussed
previously...

Regards,
Mark

On 14 Mar 2018, at 19:14, Mark Wong <mark@2ndQuadrant.com> wrote:

Hi,

I have some additional comments to a couple areas...

On Wed, Mar 14, 2018 at 05:33:00PM -0400, Dave Page wrote:

On Tue, Mar 13, 2018 at 11:31 PM, Hongyuan Ma <cs_maleicacid@163.com> wrote:
At the same time I hope you can help me understand the functional
requirements of this project more clearly. Here are some of my thoughts
on PerfFarm:

- I see this comment in the client folder (In line 15 of the "pgperffarm
\ client \ benchmarks \ pgbench.py" file):
'''
# TODO allow running custom scripts, not just the default
'''
Will PerfFarm have many test items so that the test item search function
or even the test item sort function is expected to be provided?

I don't know - Tomas or Mark are probably the best folks to ask about that.
I spent some time on the initial web app work, but then ran out of spare
cycles.

Yes, there is potential for the number of tests to change over time.
(Maybe a test is less relevant, maybe there is a more useful test down
he road.)

I'm haven't thought too much about sorting and searching functions. But
right now there is just pgbench with 6 basic configurations: read/write,
read-only, and a low, medium, and high data set size for each of those.

- What value will be used to determine if a machine's performance is
improving or declining?

Good question - I think that needs discussion, as it's not necessarily
clear-cut when you think that performance may vary at different numbers of
clients.

My thought here is that each test should provide a single metric,
defined by the test itself, so expect this to be define in the test
results sent to the web site.

I do expect that off of the test results should be archived somehow and
retrievable. But the idea I have is to have a generic high level view
that all of the different tests can share.

- I see BuildFarm assigning an animal name to each registered machine. Will
PerfFarm also have this interesting feature?

It was going to, but I forget what we decided on for a naming scheme!
Another discussion item I think - in the code, we can just use "Name" or
similar.

Since the buildfarm is composed of animals, I thought plants would be a
complimentary scheme? I'm also not sure if this was discussed
previously...

For some reason I was thinking fish!

Dave Page <dpage@pgadmin.org> writes:

On 14 Mar 2018, at 19:14, Mark Wong <mark@2ndQuadrant.com> wrote:
Since the buildfarm is composed of animals, I thought plants would be a
complimentary scheme? I'm also not sure if this was discussed
previously...

For some reason I was thinking fish!

There's already a fair number of fish in the buildfarm. I like the
plant idea ...

regards, tom lane

On Wed, Mar 14, 2018 at 10:33 PM, Dave Page <dpage@pgadmin.org> wrote:

Hi

On Tue, Mar 13, 2018 at 11:31 PM, Hongyuan Ma <cs_maleicacid@163.com>
wrote:

Hi Dave,
I am willing to use React to re-create the front-end application. Since
I plan to use separate front-end and back-end development methods, this
means that there will be no html files in Django applications. Front-end
and back-end applications will interact via restful api. I will use
react to rewrite some existing html files if needed. Before initializing
the react application, I want to learn more about your tendency toward
front-end technology.

- About React: Which version of React (15.x or 16) and react-router
(2.x or 4) do you tend to use?

pgAdmin is using React 16.2.

- About Package Manager: Do you tend to use yarn or npm?

Yarn for pgAdmin, though npm would probably be a little easier for a
project running on pginfra.

If at all possible, please stick to things that are packaged in debian
stable (currently stretch).

Anything else will increase the maintenance burden by orders of magnitude,
and make it a lot less likely to be accepted.

(That is for anything that needs to run on the deploy of course -- if a
tool is needed to update the contents of the repo before deploy or such,
then it's of course no problem)

//Magnus