Experimental tool to explore commitfest patches
Hi all,
I have created an experimental tool [0]https://patchwork-three.vercel.app/ to help explore the vast depths of
the upcoming commitfest, and it's designed to help each contributor find
actually useful and relevant patches to review. Please have a look!
Under the hood, it does two things:
1. Use a good LLM [1]https://deepmind.google/technologies/gemini/flash/ to analyze all the mailing threads tied to the
commitfest. This gives us a summary of the thread, a summary of the main
blocker, if any, and a gut-check on whether we actually need a new reviewer
in the thread. It also gives a first-principles read on the actual status:
are we waiting for the author to make changes, for a reviewer to respond,
etc.
2. Cross-reference the files in the patches to the personal commit history
of everyone in the postgres project. In this way, using a variant of the
classic TF-IDF algorithm [2]https://en.wikipedia.org/wiki/Tf%E2%80%93idf, we can score how close the patch lies to each
contributor's usual territory. It's only a heuristic, but seemed well worth
trying.
The data pipeline for this is run on my personal laptop at the moment. I've
just refreshed everything but, of course, the various statuses and analyses
need to be re-run reasonably often to remain useful. It only costs a dollar
or two to run everything through the LLM, and I can probably optimize what
really needs to be processed, but this is worth considering if there is
broader interest. If we like this, I'm happy to help port a version of it
over to the commitfest app.
Here is the source code for the whole app [3]https://github.com/polkerty/patchwork.
Regards,
Jacob Brazeal
[0]: https://patchwork-three.vercel.app/
[1]: https://deepmind.google/technologies/gemini/flash/
[2]: https://en.wikipedia.org/wiki/Tf%E2%80%93idf
[3]: https://github.com/polkerty/patchwork
Hi Jacob,
Thanks a lot for this! We have been trying to come up with a similar
feature for the new commitfest app [0]https://github.com/postgres/pgcommitfest/issues/18.
If we like this, I'm happy to help port a version of it over to the
commitfest app.
I would love to help you out in porting this to the commitfest app.
Regards,
Akshat Jaimini
[0]: https://github.com/postgres/pgcommitfest/issues/18
On Mon, Feb 24, 2025 at 9:01 AM Jacob Brazeal <jacob.brazeal@gmail.com>
wrote:
Show quoted text
Hi all,
I have created an experimental tool [0] to help explore the vast depths of
the upcoming commitfest, and it's designed to help each contributor find
actually useful and relevant patches to review. Please have a look!Under the hood, it does two things:
1. Use a good LLM [1] to analyze all the mailing threads tied to the
commitfest. This gives us a summary of the thread, a summary of the main
blocker, if any, and a gut-check on whether we actually need a new reviewer
in the thread. It also gives a first-principles read on the actual status:
are we waiting for the author to make changes, for a reviewer to respond,
etc.
2. Cross-reference the files in the patches to the personal commit history
of everyone in the postgres project. In this way, using a variant of the
classic TF-IDF algorithm [2], we can score how close the patch lies to each
contributor's usual territory. It's only a heuristic, but seemed well worth
trying.The data pipeline for this is run on my personal laptop at the moment.
I've just refreshed everything but, of course, the various statuses and
analyses need to be re-run reasonably often to remain useful. It only costs
a dollar or two to run everything through the LLM, and I can probably
optimize what really needs to be processed, but this is worth considering
if there is broader interest. If we like this, I'm happy to help port a
version of it over to the commitfest app.Here is the source code for the whole app [3].
Regards,
Jacob Brazeal[0] https://patchwork-three.vercel.app/
[1] https://deepmind.google/technologies/gemini/flash/
[2] https://en.wikipedia.org/wiki/Tf%E2%80%93idf
[3] https://github.com/polkerty/patchwork
We have been trying to come up with a similar feature for the new
commitfest app
I would love to help you out in porting this to the commitfest app.
Great! There is plenty of work to do, there is some discussion in the
discord [0]https://discord.com/channels/1258108670710124574/1328362897189113867.
I think it would also be nice to set up a proper data pipeline using
something like Dagster [1]https://dagster.io/ or Airflow [2]https://airflow.apache.org/ to manage the scraping, LLM
calls, postprocessing, etc.
[0]: https://discord.com/channels/1258108670710124574/1328362897189113867
[1]: https://dagster.io/
[2]: https://airflow.apache.org/
On Sun, Feb 23, 2025 at 10:31 PM Jacob Brazeal <jacob.brazeal@gmail.com> wrote:
I have created an experimental tool [0] to help explore the vast depths of the upcoming commitfest, and it's designed to help each contributor find actually useful and relevant patches to review. Please have a look!
As I also mentioned in the Discord, I really like the auto-generated
summaries. No doubt they are not perfect, but they seem pretty useful
on first look. The patch ranking seems a bit odd, though -- it thinks
I should be super-interested in postgres_fdw patches. So that part
might need some more work.
--
Robert Haas
EDB: http://www.enterprisedb.com
I wanted to provide a quick update on the app [0]https://patchwork-three.vercel.app/. Here are the main issues
I've seen flagged so far:
1. The ranking system needs improvement. Ideally it should promote *relevant,
important, ready-for-review* patches.
2. We should display contributor names as they appear in the commitfest app
(this is relevant because we have to correlate names from several different
systems.)
I will be working on all of these, but tonight I want to provide an update
on the ranking system. The app now predicts which committers might be a
good fit for a patch, and displays this information in the app. If you are
committer and select your name in the queue, those patches will float to
the top. As a quick sanity check, most of the cases I've seen flagged so
far are correctly handled by the new system. Here are some more details on
how it works and how
The new recommendation system is based on keywords. I used an LLM to
extract technical keywords from the mailing list threads associated with
the last 10,000 git commits, and then trained a logistic regression model
to match the keywords to committers. I'm no expert at this, but I did some
basic statistical validation of the result on a training/test split and got
decent results: around 44% of the top choices of the model were correct,
and just to be safe, I show the top 3 predicted committer for each patch in
the UX. When looking at specific folks like Robert, in our test dataset,
about 77% of the results matched to him he actually committed (precision)
and we overall identify about 45% of his commits (recall.) So, not perfect,
but actually pretty likely to tag a mailing list thread to the person who
will commit it.
In the UX, if you are one of the top 3 identified committers, you will also
see a list of the top keywords from the mailing thread that were associated
with you.