Postgres forums ... take 2

On 15 November 2010 08:34, Elliot Chance <elliotchance@gmail.com> wrote:

Hi again,

I've taken in all the feedback about http://forums.postgresql.com.au and
the general consensus is that nobody wants a separate entity - a few people
mentioned that if it was interoperable with the mailing list that it would
be better. So I did.

The concept goes like this;
1. Any posts to the general mailing list will be picked up by the forum,
the email data is converted and posted on the forum, for example;
http://forums.postgresql.com.au/viewtopic.php?f=30&amp;t=39
2. Any reply to the forum will do the reverse and send the post back to the
mailing list as a reply.

This means the forum can be fully controlled through the mailing list
without the need to visit the forums directly. However those people who
prefer to use a forum interface can, and those messages are relayed back
through the mailing list to get answered.

Step 1 is complete (might need a little tweaking, i've only tried it with a
couple of topics.) Step 2 I haven't begun - wanted to get some more
feedback.

All the forum topics and posts are back-dated to match the emails, which
means it would be *theoretically* possible to load in the entire postgres
mailing list archive but I wouldn't do that on a server that couldn't handle
that much data.

Disclaimer about user names:
User names are registered automatically based on the unique email address
of the person emailing the response. Each user is given a random 8 character
password. You can use the recover password page to login to your account and
change your user name to anything you want, the only important thing is that
your email address matches.

I know this is a sensitive issue with some people, i've made sure no
information is posted thats not already currently being indexed by google.

The only maintenance I can see is that all new topics are pushed into the
General > Other category as the script can't differentiate what category it
should in fact belong to, once the topic is moved it will stay there. This
shouldn't be a real problem as theres not many new topics being created on
any given day.

Elliot,

That's actually some good work you've done there! I didn't know phpBB
supported bidirectional mailing list support.

I think, however, that having such a forum at a .com.au address isn't
particularly desirable, as it implies it's regional. If others are happy
for you to work on this, it might be an idea to speak to the existing web
team to see if they are able to provide you with pointers and possibly
resources to get such a thing up and running. It would be nice, for
example, to have forums.postgresql.org set up.

A few points though. I think we'd need to disable smileys, bbcode, any form
of rich text formatting, flash or embedded images. In short, plain text
only, which is the policy on the mailing list. I think it would be more
useful if each forum directly corresponded to a mailing list too. What I
mean is that if there was a forum on the site which didn't match to a
mailing list, only forum users could use it.

Also, if someone registers on the forum, do they get a major domo
registration email? And if so, would this be set to receive no emails upon
registration? I'm not clear as to how this step would work because, at the
moment, mailing list subscribers have to subscribe on a list-by-list basis.
So registration to the forum site wouldn't necessarily mean they'd want to
join any particular mailing list. Similarly, could they unregister easily?
And anyone who attempts to post to a mailing list they aren't subscribed to
requires moderation, so we don't wish to exacerbate this.

But that's a nice start. :)

Cheers

--
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935

On 15/11/2010, at 8:37 PM, Thom Brown wrote:

On 15 November 2010 08:34, Elliot Chance <elliotchance@gmail.com> wrote:
Hi again,

I've taken in all the feedback about http://forums.postgresql.com.au and the general consensus is that nobody wants a separate entity - a few people mentioned that if it was interoperable with the mailing list that it would be better. So I did.

The concept goes like this;
1. Any posts to the general mailing list will be picked up by the forum, the email data is converted and posted on the forum, for example;
http://forums.postgresql.com.au/viewtopic.php?f=30&amp;t=39
2. Any reply to the forum will do the reverse and send the post back to the mailing list as a reply.

This means the forum can be fully controlled through the mailing list without the need to visit the forums directly. However those people who prefer to use a forum interface can, and those messages are relayed back through the mailing list to get answered.

Step 1 is complete (might need a little tweaking, i've only tried it with a couple of topics.) Step 2 I haven't begun - wanted to get some more feedback.

All the forum topics and posts are back-dated to match the emails, which means it would be *theoretically* possible to load in the entire postgres mailing list archive but I wouldn't do that on a server that couldn't handle that much data.

Disclaimer about user names:
User names are registered automatically based on the unique email address of the person emailing the response. Each user is given a random 8 character password. You can use the recover password page to login to your account and change your user name to anything you want, the only important thing is that your email address matches.

I know this is a sensitive issue with some people, i've made sure no information is posted thats not already currently being indexed by google.

The only maintenance I can see is that all new topics are pushed into the General > Other category as the script can't differentiate what category it should in fact belong to, once the topic is moved it will stay there. This shouldn't be a real problem as theres not many new topics being created on any given day.

Elliot,

That's actually some good work you've done there! I didn't know phpBB supported bidirectional mailing list support.

It doesn't. I have a subscription address that is piped into a PHP script that uses the phpBB3 APIs to do all you see.

I think, however, that having such a forum at a .com.au address isn't particularly desirable, as it implies it's regional. If others are happy for you to work on this, it might be an idea to speak to the existing web team to see if they are able to provide you with pointers and possibly resources to get such a thing up and running. It would be nice, for example, to have forums.postgresql.org set up.

I was just amazed that postgresql.com.au was available (in australia you need a registered company to get a .com.au address so that's why.)

At the moment its running on mysql (I know, but they don't support postgres) but it will work with postgres. The forum software, database and scripts I've written are all portable so theres no reason why it couldn't be moved to another domain any time in the future. Obviously at the time I couldn't use forums.postgresql.org.

Lets see how it goes, if it does turn out to be useful then we'll have a chat to the developers.

A few points though. I think we'd need to disable smileys, bbcode, any form of rich text formatting, flash or embedded images. In short, plain text only, which is the policy on the mailing list. I think it would be more useful if each forum directly corresponded to a mailing list too. What I mean is that if there was a forum on the site which didn't match to a mailing list, only forum users could use it.

If someone were to send a reply on the forum all the bbcode would be stripped before emailing it to the mailing list to keep the mailing list "pure." Is that what you mean?

Also, if someone registers on the forum, do they get a major domo registration email? And if so, would this be set to receive no emails upon registration? I'm not clear as to how this step would work because, at the moment, mailing list subscribers have to subscribe on a list-by-list basis. So registration to the forum site wouldn't necessarily mean they'd want to join any particular mailing list. Similarly, could they unregister easily? And anyone who attempts to post to a mailing list they aren't subscribed to requires moderation, so we don't wish to exacerbate this.

No they are not registered on the mailing list, but they actually don't need to be, let me explain:
1. John Smith has a postgres related question and finds the forums, he signs up and posts his question.
2. His post is then emailed to the mailing list under a generic registered address like "mailinglist@postgresql.com.au"
3. Bob House reads Johns question on the mailing list and simply sends an email reply.
4. The email reply is piped into the forum and matches the topic based on the email subject (thats how it currently does it.)
5. John gets an email from phpBB along the lines of "Bob House has replied to your post, click here" (all forums do this) he reads the response and is happy.

This is the best balance of no-fuss and expert response, keeping in mind that:
* John can still sign up to the mailing list like anyone else if he wants to.
* All of John's forums communications are in the postgres mailing list archive now.

But that's a nice start. :)

I thought smilies were banned ...... :D

Nothing is set in stone. There almost definitely will be forum categories added/merged/removed, lets call this a beta.

On Mon, Nov 15, 2010 at 11:08, Elliot Chance <elliotchance@gmail.com> wrote:

On 15/11/2010, at 8:37 PM, Thom Brown wrote:

I know this is a sensitive issue with some people, i've made sure no
information is posted thats not already currently being indexed by google.

The only maintenance I can see is that all new topics are pushed into the
General > Other category as the script can't differentiate what category it
should in fact belong to, once the topic is moved it will stay there. This
shouldn't be a real problem as theres not many new topics being created on
any given day.

Elliot,

That's actually some good work you've done there!  I didn't know phpBB
supported bidirectional mailing list support.

It doesn't. I have a subscription address that is piped into a PHP script
that uses the phpBB3 APIs to do all you see.

That sounds scary :-) Particularly given attachments and such. but if
it works....

A few points though.  I think we'd need to disable smileys, bbcode, any form
of rich text formatting, flash or embedded images.  In short, plain text
only, which is the policy on the mailing list.  I think it would be more
useful if each forum directly corresponded to a mailing list too.  What I
mean is that if there was a forum on the site which didn't match to a
mailing list, only forum users could use it.

If someone were to send a reply on the forum all the bbcode would be
stripped before emailing it to the mailing list to keep the mailing list
"pure." Is that what you mean?

Personally, my thoughts are that if we want lists mirrored to a forum,
they should look the same in both cases. Which means they should be
stripped in the forums *as well*. but since I wouldn't be using the
forums, my view should perhaps not be paid attention to around that.
But there should *definitely* not be any bbcode going to the
mailinglists.

Also, if someone registers on the forum, do they get a major domo
registration email?  And if so, would this be set to receive no emails upon
registration?  I'm not clear as to how this step would work because, at the
moment, mailing list subscribers have to subscribe on a list-by-list basis.
So registration to the forum site wouldn't necessarily mean they'd want to
join any particular mailing list.  Similarly, could they unregister easily?
And anyone who attempts to post to a mailing list they aren't subscribed to
requires moderation, so we don't wish to exacerbate this.

No they are not registered on the mailing list, but they actually don't need
to be, let me explain:
1. John Smith has a postgres related question and finds the forums, he signs
up and posts his question.
2. His post is then emailed to the mailing list under a generic registered
address like "mailinglist@postgresql.com.au"

This part I really don't like. It should at least be posted with some
kind of uniquely identifiable pass-through address, if not the users
own address (make that an option?). Like
magnus-hagander-net@forums.whatever

3. Bob House reads Johns question on the mailing list and simply sends an
email reply.
4. The email reply is piped into the forum and matches the topic based on
the email subject (thats how it currently does it.)

You really should be matching on the response headers rather than
subject... Or at least both.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

I have made some major changes "beta2"

1. For now the forums have been set to read only, this is to prevent anyone posting a response (as it doesn't send emails back to the mailing list yet.)

2. Added a bunch of new forums to match the mailing lists, also have subscribed to all main mailing list with the following map:

pgsql-admin => General > Server Administration & Maintenance
pgsql-advocacy => News > Advocacy & Media
pgsql-announce => News > News & Announcements
pgsql-bugs => Development > Bugs & Testing
pgsql-docs => Development > Documentation
pgsql-general => General > Other
pgsql-jdbc => Languages > Java
pgsql-jobs => Other > Commercial & Jobs
pgsql-novice => General > Newbie Section
pgsql-odbc => Languages > ODBC / Other
pgsql-performance => General > Performance & Benchmarking
pgsql-php => Languages > PHP
pgsql-sql => Languages > SQL
pgsql-students => Other > Education & Certification

This is visa-vers so forum topics posted on Languages > SQL will be posted to pgsql-sql, etc.

3. A rewrite of the mail to forums script so that it uses a MIME parser which handles messy emails, quotations and multipart emails as it should now.

4. The mail parser uses the correct "in-reply-to" to match up discussion threads rather than simply stripping the email subject.

Extra thoughts;

It would not be practical for the forums to create a dummy mailing list email address per person or forum, however theres needs to be a robust way to make sure the topics/threads and posts match up with the threads and emails in the mailing list. The problem I see is that replies to the forum are not technically replies via email and so they will not carry the unique "in-reply-to" identifier. I believe this can be fixed by spoofing the in-reply-to from the forum, so that each forum reply will drop in the in-reply-to manually. I will do some testing on my own address before any messages are sent to the real mailing lists of course.

Stripping bbcode, smilies, HTML or whatever is very simple, nothing to worry about there.

Since the php piping script had so many changes, I wiped the contents of the forum to run it all again. So any URLs you had that point to specific thread IDs probably wont work any more, but as you can see the forums seems to be doing what it should:
http://forums.postgresql.com.au

There needs to be more forum mapping from specific forums to mailing lists, for example "Languages > Perl" to the closest mailing list which might be pgsql-general. However once the topic is created in a forum all the responses will stay in that forum, so even though people reply on the pgsql-general mailing list the replies appear under Languages > Perl.

The infrastructure exists to create as many forum mappings as needed, and I could add post processing. So for example an email to pgsql-general with the title "perl won't connect" will recognise "perl" and move it to the Languages > Perl.

On 15/11/2010, at 9:42 PM, Magnus Hagander wrote:

On 15/11/10 17:37, Thom Brown wrote:

That's actually some good work you've done there! I didn't know phpBB
supported bidirectional mailing list support.

Yikes. Neither did I. I've always seen phpBB as the barren wasteland of
web forums - forums full of half-page animated GIF signatures separating
single lines of text, some kind of content-free zone of minimum
information density. Maybe it can be configured to be better than that
after all.

How does it handle threading? Will forum threads be properly threaded?
And will replies have the correct In-Reply-To: <msgid> header so they
get threaded correctly?

Have you been in touch with the Pg list admins to make sure they're cool
with this?

A few points though. I think we'd need to disable smileys, bbcode, any
form of rich text formatting, flash or embedded images.

Mostly agreed. Limiting signatures to 4/5 lines would be nice too.

Limited HTML is really useful on web forums, though, as it allows you to
delineate code from other text. Unless the whole forum is set to
monospaced text with preserved whitespace, that's necessary to ensure
that code samples are readable.

--
Craig Ringer

Tech-related writing: http://soapyfrogs.blogspot.com/

On 16/11/2010, at 2:01 PM, Craig Ringer wrote:

On 15/11/10 17:37, Thom Brown wrote:

That's actually some good work you've done there! I didn't know phpBB
supported bidirectional mailing list support.

Yikes. Neither did I. I've always seen phpBB as the barren wasteland of
web forums - forums full of half-page animated GIF signatures separating
single lines of text, some kind of content-free zone of minimum
information density. Maybe it can be configured to be better than that
after all.

How does it handle threading? Will forum threads be properly threaded?
And will replies have the correct In-Reply-To: <msgid> header so they
get threaded correctly?

It uses the message-id and in-reply-to header information, so a change in email subject will not effect the flow of the thread. However not all emails stick by those rules, a few emails do not carry the header information in which case the script strips the subject (to remove "Re:", "[GENERAL]") and then matches the topic by name as a fall back.

Email header information can be sketchy at the best of times, but this does a pretty good job at making sure almost all of the messages are handled correctly. In some rare cases when the in-reply-to is missing and the subject has changed it will create a new thread, but a forum moderator can click a button to merge the threads and all is fixed.

Have you been in touch with the Pg list admins to make sure they're cool
with this?

At this point its a good idea, who is the best person(s) to contact? I want to make sure anything I do does not in any way reflect badly on the community or seem like i'm doing anything dishonest.

A few points though. I think we'd need to disable smileys, bbcode, any
form of rich text formatting, flash or embedded images.

Mostly agreed. Limiting signatures to 4/5 lines would be nice too.

All forum software limits the size of a signature to stop people abusing it. As the signature is a separate entity the mail that comes from the forum can contain or ignore the signature.

Limited HTML is really useful on web forums, though, as it allows you to
delineate code from other text. Unless the whole forum is set to
monospaced text with preserved whitespace, that's necessary to ensure
that code samples are readable.

That's one thing that can't be fixed when incoming emails are converted to forums posts the code blocks appear as normal text.

There is no use importing huge archives seeing as most of it is either too old to be relevant, bloat the forum and most people don't bother using a search anyway before posting. But it may be alright to say import the last few months of data (that include product release announcements, active bugs etc.) I have written another script which imports the mbox files just as a proof of concept:
http://forums.postgresql.com.au/viewforum.php?f=34
Those 7 threads were imported from a single months mbox file.

On Tue, Nov 16, 2010 at 07:00, Elliot Chance <elliotchance@gmail.com> wrote:

On 16/11/2010, at 2:01 PM, Craig Ringer wrote:

Have you been in touch with the Pg list admins to make sure they're cool
with this?

At this point its a good idea, who is the best person(s) to contact? I want to make sure anything I do does not in any way reflect badly on the community or seem like i'm doing anything dishonest.

Some of us are already reading this thread. But the correct forum to
use is the pgsql-www mailinglist.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tue, Nov 16, 2010 at 03:45, Elliot Chance <elliotchance@gmail.com> wrote:

I have made some major changes "beta2"

<snip>

Extra thoughts;

It would not be practical for the forums to create a dummy mailing list email address per person or forum,

Why? It doesn't have to be actual mailboxes, but it needs to be a
deliverable email address.

The other option is, of course, to send the email using the email
address the forum user uses to register with the forum. That might
cause issues with some antispam solutions, but as long as it's done
right, I think that would work.

Personally, I find the lack of this a show-stopper issue. We do *not*
want what's basically going to be anonymous posts on the lists.

however theres needs to be a robust way to make sure the topics/threads and posts match up with the threads and emails in the mailing list. The problem I see is that replies to the forum are not technically replies via email and so they will not carry the unique "in-reply-to" identifier.

The email generated is a reply via email, and carries a message id. It
should be perfectly possible to chain those together using
in-reply-to, as long as all posts are mirrored between the two media.

There needs to be more forum mapping from specific forums to mailing lists, for example "Languages > Perl" to the closest mailing list which might be pgsql-general. However once the topic is created in a forum all the responses will stay in that forum, so even though people reply on the pgsql-general mailing list the replies appear under Languages > Perl.

IMHO, there needs to be a one-to-one mapping, and nothing else.

The infrastructure exists to create as many forum mappings as needed, and I could add post processing. So for example an email to pgsql-general with the title "perl won't connect" will recognise "perl" and move it to the Languages > Perl.

That sounds like a really bad idea - it's going to cause nothing but confusion.

I'm not a big user of web forums (I use them when I have to, but it's
certainly not a medium I consider efficient so I don't choose it), so
here's a question that may be obvious, but still required: quoting.
Can the forum software be set up to always quote responses properly?
And somehow discourage top-posting in said responses? We absolutely do
*not* want a forum to start feeding non-quoted responses back to the
mailinglists, and non-quoted responses is unfortunately pretty common
on most forums where I usually end up - but again, that is hopefully
just a setting :-)

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tuesday 16 November 2010 10:30:05 Magnus Hagander wrote:

On Tue, Nov 16, 2010 at 03:45, Elliot Chance <elliotchance@gmail.com> wrote:

I have made some major changes "beta2"

<snip>

Extra thoughts;

It would not be practical for the forums to create a dummy mailing list
email address per person or forum,

Why? It doesn't have to be actual mailboxes, but it needs to be a
deliverable email address.

The other option is, of course, to send the email using the email
address the forum user uses to register with the forum. That might
cause issues with some antispam solutions, but as long as it's done
right, I think that would work.

If this is done in cooperation with the list admins, they could whitelist the
forum-server for this?

Personally, I find the lack of this a show-stopper issue. We do *not*
want what's basically going to be anonymous posts on the lists.

I agree, as it's an easy way for spammers to start spamming the whole list.
Has anyone thought about what would happen if someone does an unsubscribe for
the forum email? :)

however theres needs to be a robust way to make sure the topics/threads
and posts match up with the threads and emails in the mailing list. The
problem I see is that replies to the forum are not technically replies
via email and so they will not carry the unique "in-reply-to"
identifier.

The email generated is a reply via email, and carries a message id. It
should be perfectly possible to chain those together using
in-reply-to, as long as all posts are mirrored between the two media.

I think the only way to correctly mirror these 2 is to use one as the master
and have the other populated by the master.
As the mailing list already exists and is used a lot already, I would think
the following would work:
- user posts on forum, email is generated. When email comes from list, it is
entered into the forum

I believe this is how gmane works.

There needs to be more forum mapping from specific forums to mailing
lists, for example "Languages > Perl" to the closest mailing list which
might be pgsql-general. However once the topic is created in a forum all
the responses will stay in that forum, so even though people reply on
the pgsql-general mailing list the replies appear under Languages >
Perl.

IMHO, there needs to be a one-to-one mapping, and nothing else.

Agreed

The infrastructure exists to create as many forum mappings as needed, and
I could add post processing. So for example an email to pgsql-general
with the title "perl won't connect" will recognise "perl" and move it to
the Languages > Perl.

That sounds like a really bad idea - it's going to cause nothing but
confusion.

How will it be done if a subject contains more then one "keywords"?
Eg. "Porting C-code to Perl causes performance issue"
I see "C", "Perl" and "Performance"
Where will it then be moved to?

I'm not a big user of web forums (I use them when I have to, but it's
certainly not a medium I consider efficient so I don't choose it), so
here's a question that may be obvious, but still required: quoting.
Can the forum software be set up to always quote responses properly?
And somehow discourage top-posting in said responses? We absolutely do
*not* want a forum to start feeding non-quoted responses back to the
mailinglists, and non-quoted responses is unfortunately pretty common
on most forums where I usually end up - but again, that is hopefully
just a setting :-)

I doubt that, I am also on a mailing list where a similar link is already set
up.
I occasionally get emails there without quotes. It's ok if the thread isn't
too old. But if someone replies to a thread that's more then a year old, it
doesn't get linked up. (I move older posts into subfolders to keep my mail
client responsive)

If I feel like it, I can then click on the link to the forum to read the
actual thread. At least that way it is possible to make sense of it.

--
Joost

On 16 November 2010 09:30, Magnus Hagander <magnus@hagander.net> wrote:

On Tue, Nov 16, 2010 at 03:45, Elliot Chance <elliotchance@gmail.com> wrote:

I have made some major changes "beta2"

<snip>

Extra thoughts;

It would not be practical for the forums to create a dummy mailing list email address per person or forum,

Why? It doesn't have to be actual mailboxes, but it needs to be a
deliverable email address.

The other option is, of course, to send the email using the email
address the forum user uses to register with the forum. That might
cause issues with some antispam solutions, but as long as it's done
right, I think that would work.

Won't the vast majority of those require moderation if users aren't
signed up to a mailing list?  I mean we could have the "reply-to"
header value contain the forum's email address, but the "from" address
would be rejected by the list surely?  If that could somehow be made
to work, that would be ideal though.

Personally, I find the lack of this a show-stopper issue. We do *not*
want what's basically going to be anonymous posts on the lists.

Definitely.

IMHO, there needs to be a one-to-one mapping, and nothing else.

Agreed.  The justification for a forum, from my perspective, is
another method of interacting with the mailing list to open it up to a
wider audience.  I don't like the idea of additional forums which
don't match a mailing list as it would not only create community
fragmentation, but most of the people with the answers won't be
reading the forums.

The infrastructure exists to create as many forum mappings as needed, and I could add post processing. So for example an email to pgsql-general with the title "perl won't connect" will recognise "perl" and move it to the Languages > Perl.

That sounds like a really bad idea - it's going to cause nothing but confusion.

Yes, we wouldn't want any clever logic to automagically file the posts
into certain categories.  We don't have that on the mailing lists, so
the forum also shouldn't have it.

I'm not a big user of web forums (I use them when I have to, but it's
certainly not a medium I consider efficient so I don't choose it), so
here's a question that may be obvious, but still required: quoting.
Can the forum software be set up to always quote responses properly?
And somehow discourage top-posting in said responses? We absolutely do
*not* want a forum to start feeding non-quoted responses back to the
mailinglists, and non-quoted responses is unfortunately pretty common
on most forums where I usually end up - but again, that is hopefully
just a setting :-)

Have you seen Elliot's prototype?  From what I've seen, quoting comes
through fine.  It would just have to work correctly the other way, in
that it sends plain text emails with correct levels of chevrons.

A test mailing list will no doubt need to be set up for testing such
functionality. But before too much work commences on this, will this
have the backing of the team? I personally think, even though I don't
want a forum myself, others will, and it would reduce barriers to the
community. Obviously it will need to work seamlessly too, so that it
doesn't cause any issues on the mailing lists themselves.

--
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935

On Tue, Nov 16, 2010 at 11:11, Thom Brown <thom@linux.com> wrote:

On 16 November 2010 09:30, Magnus Hagander <magnus@hagander.net> wrote:

On Tue, Nov 16, 2010 at 03:45, Elliot Chance <elliotchance@gmail.com> wrote:

I have made some major changes "beta2"

<snip>

Extra thoughts;

It would not be practical for the forums to create a dummy mailing list email address per person or forum,

Why? It doesn't have to be actual mailboxes, but it needs to be a
deliverable email address.

The other option is, of course, to send the email using the email
address the forum user uses to register with the forum. That might
cause issues with some antispam solutions, but as long as it's done
right, I think that would work.

Won't the vast majority of those require moderation if users aren't
signed up to a mailing list?  I mean we could have the "reply-to"

It would. The best way around that would be to auto-subscribe them and
set them to NOMAIL, as has been previously suggested.

Personally, I find the lack of this a show-stopper issue. We do *not*
want what's basically going to be anonymous posts on the lists.

Definitely.

IMHO, there needs to be a one-to-one mapping, and nothing else.

Agreed.  The justification for a forum, from my perspective, is
another method of interacting with the mailing list to open it up to a
wider audience.  I don't like the idea of additional forums which
don't match a mailing list as it would not only create community
fragmentation, but most of the people with the answers won't be
reading the forums.

Exactly my point.

I'm not a big user of web forums (I use them when I have to, but it's
certainly not a medium I consider efficient so I don't choose it), so
here's a question that may be obvious, but still required: quoting.
Can the forum software be set up to always quote responses properly?
And somehow discourage top-posting in said responses? We absolutely do
*not* want a forum to start feeding non-quoted responses back to the
mailinglists, and non-quoted responses is unfortunately pretty common
on most forums where I usually end up - but again, that is hopefully
just a setting :-)

Have you seen Elliot's prototype?  From what I've seen, quoting comes
through fine.  It would just have to work correctly the other way, in
that it sends plain text emails with correct levels of chevrons.

Yes, I'm only talking about the forum->mail direction here.

A test mailing list will no doubt need to be set up for testing such
functionality.  But before too much work commences on this, will this
have the backing of the team?  I personally think, even though I don't
want a forum myself, others will, and it would reduce barriers to the
community.  Obviously it will need to work seamlessly too, so that it
doesn't cause any issues on the mailing lists themselves.

Personally, I would Ok with doing this, *IF* all the issues raised are
dealt with. Particularly the posting side *must* be fixed. Without
that, it definitely won't have the backing of the
sysadmin/infrastructure team. With it, probably, but I can't speak for
others than myself.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tue, Nov 16, 2010 at 10:59, J. Roeleveld <joost@antarean.org> wrote:

On Tuesday 16 November 2010 10:30:05 Magnus Hagander wrote:

On Tue, Nov 16, 2010 at 03:45, Elliot Chance <elliotchance@gmail.com> wrote:

I have made some major changes "beta2"

<snip>

Extra thoughts;

It would not be practical for the forums to create a dummy mailing list
email address per person or forum,

Why? It doesn't have to be actual mailboxes, but it needs to be a
deliverable email address.

The other option is, of course, to send the email using the email
address the forum user uses to register with the forum. That might
cause issues with some antispam solutions, but as long as it's done
right, I think that would work.

If this is done in cooperation with the list admins, they could whitelist the
forum-server for this?

There's actually no way to whitelist a server in mj2. We've tried this
for other things, and it just doesn't work.

however theres needs to be a robust way to make sure the topics/threads
and posts match up with the threads and emails in the mailing list. The
problem I see is that replies to the forum are not technically replies
via email and so they will not carry the unique "in-reply-to"
identifier.

The email generated is a reply via email, and carries a message id. It
should be perfectly possible to chain those together using
in-reply-to, as long as all posts are mirrored between the two media.

I think the only way to correctly mirror these 2 is to use one as the master
and have the other populated by the master.
As the mailing list already exists and is used a lot already, I would think
the following would work:
- user posts on forum, email is generated. When email comes from list, it is
entered into the forum

Yes, that's pretty much how it would have to work.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Hi,

Something isn't going right:
http://forums.postgresql.com.au/viewtopic.php?f=30&amp;t=96
Contains 2 issues.
I participated in both, but my reply to the second issue is not included in
the forum.
If you need me to show you the emails or whatever, just ask.

btw: great stuff! :)

Cheers,

WBL

On Tue, Nov 16, 2010 at 11:29 AM, Magnus Hagander <magnus@hagander.net>wrote:

On Tue, Nov 16, 2010 at 10:59, J. Roeleveld <joost@antarean.org> wrote:

On Tuesday 16 November 2010 10:30:05 Magnus Hagander wrote:

On Tue, Nov 16, 2010 at 03:45, Elliot Chance <elliotchance@gmail.com>

wrote:

I have made some major changes "beta2"

<snip>

Extra thoughts;

It would not be practical for the forums to create a dummy mailing

list

email address per person or forum,

Why? It doesn't have to be actual mailboxes, but it needs to be a
deliverable email address.

The other option is, of course, to send the email using the email
address the forum user uses to register with the forum. That might
cause issues with some antispam solutions, but as long as it's done
right, I think that would work.

If this is done in cooperation with the list admins, they could whitelist

the

forum-server for this?

There's actually no way to whitelist a server in mj2. We've tried this
for other things, and it just doesn't work.

however theres needs to be a robust way to make sure the

topics/threads

and posts match up with the threads and emails in the mailing list.

The

problem I see is that replies to the forum are not technically replies
via email and so they will not carry the unique "in-reply-to"
identifier.

The email generated is a reply via email, and carries a message id. It
should be perfectly possible to chain those together using
in-reply-to, as long as all posts are mirrored between the two media.

I think the only way to correctly mirror these 2 is to use one as the

master

and have the other populated by the master.
As the mailing list already exists and is used a lot already, I would

think

the following would work:
- user posts on forum, email is generated. When email comes from list, it

is

entered into the forum

Yes, that's pretty much how it would have to work.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
"Patriotism is the conviction that your country is superior to all others
because you were born in it." -- George Bernard Shaw

Alrighty, here are the revised plans for beta3:

--- Changes / Fixes

* Without a doubt everyone one wants the forums to match the mailing lists. I personally think that this defeats the whole purpose of a forum, but i'm here to do what the community thinks is best. The extra forums that are not connected with a mailing list will be removed. And some of the forums renamed - i'll leave that part up to you to decide amongst yourselves.

* Obviously the above point means no filing script is needed. But for JRoeleveld; I had already through about that - I was thinking of a point keyword system. A title that has 3 words that belong to 3 different forums goes nowhere because a single forum location is no stronger than the rest but at best it's machine guesswork and we all know how different theoretical plans can be from practicality.

* The second largest issue is that of the email address to assign to emails created by forum posts. I don't know the internals of how the mailing list software works, but some thoughts;
- Does the mailing allow a range of email addresses? For example using the persons clean username like: "forums-chancey@postgresql.com.au" or "forums-bobsmith@postgresql.com.au" hence we allow the "forums-*@postgresql.com.au" range? This way I can use a catch all address and filter.
- If theres an API to the mailing list that lets me register the persons real email address without then having to get separate copies in their inbox and also does not require a URL to be clicked to enable that email?

* Quotation. Emails have a higher depth of quotation - it is not uncommon for a single email to contain levels for the past 4 emails and this works well in emails but not in a forum, this makes posts much longer then they need to be in most cases and bloats the search with loads of repeated information.
For example phpBB3 by default limits the max quote level to 3 to stop extraneous information. I'm still thinking of a solution for this.

* Quotation ownership. You will notice that emails take the form something like:
Bob Smith <bob@smith.com> wrote:

1 + 1 = 2?

Yes.

The above gets translated only so that the ">" gets converted into the [quote] tags for the forums display. But quoting in forums adds the ownership into the quote block, so the above would look like (forums do this automatically when you quote someone):
| Quote by [Bob Smith]:
| 1 + 1 = 2?
Yes.

Where [Bob Smith] is a link to the forum user. I originally wrote this into beta1 but saw there was no consistency and emails came in with "wrote:" "writes:" or something that just didn't make sense at all so I disabled the code until I was ready to work on it properly. I will institute it again - wish me luck...

--- Testing

* Don't be alarmed if your some posts don't show up. The parser script is not on a cron for now and I may be deleting or playing with posts while I tweak it, the idea is that things will get messy while its being tested and then wipe the forums completely and use the mbox importer which I wrote a few hours ago to add the real posts.

* I added user back dating in beta2. This means that if a mbox from 2 years ago is imported (hypothetically) the user is automatically back dated to the earliest post so it will show then as signed up 2 years ago with X posts per day since then. Anyone want to guess who's had the most posts overall?

--- Other notes

I didn't quote all the appropriate people above to keep the plan for beta3 clean. beta3 is when concept turns into a clear(er) plan. Lets get all this worked before I start coding.

Nobody has made much comment on the permission of containing the mailing list information and member email addresses in another container like a forum. I understand that I can broadcast this information on other mailing lists (www was mentioned) but if theres no absolute authority figure would it make any serious difference from what we're already doing?

Cheers,
Elliot

On 11/16/2010 08:43 PM, Elliot Chance wrote:

Alrighty, here are the revised plans for beta3:

--- Changes / Fixes

* Without a doubt everyone one wants the forums to match the mailing lists. I personally think that this defeats the whole purpose of a forum, but i'm here to do what the community thinks is best. The extra forums that are not connected with a mailing list will be removed. And some of the forums renamed - i'll leave that part up to you to decide amongst yourselves.

On a bit of a side-note, I'm increasingly wishing Stack Overflow had a
mail interface. I take Greg's point that it's an increasingly key place
for people (especially people not otherwise engaged in the community) to
seek information and help. It's a pity that there doesn't seem to be a
good way to connect the StackOverflow postgresql discussion in some way
that makes it more visible on the main community resources.

I think Greg may have a point when questioning whether adding a forum
interface is overly useful, given that Stack Overflow already exists and
doesn't see all that much attention. OTOH, maybe forums will draw people
who otherwise ask on S.O. to the community, providing a bit of a bridge.
It's worth a try.

--
Craig Ringer

On Tue, Nov 16, 2010 at 13:43, Elliot Chance <elliotchance@gmail.com> wrote:

Alrighty, here are the revised plans for beta3:

* The second largest issue is that of the email address to assign to emails created by forum posts. I don't know the internals of how the mailing list software works, but some thoughts;
 - Does the mailing allow a range of email addresses? For example using the persons clean username like: "forums-chancey@postgresql.com.au" or "forums-bobsmith@postgresql.com.au" hence we allow the "forums-*@postgresql.com.au" range? This way I can use a catch all address and filter.

It does allow regular expressions in some ways, but I'm not sure it
does it in this case - nor that it's the best idea.

 - If theres an API to the mailing list that lets me register the persons real email address without then having to get separate copies in their inbox and also does not require a URL to be clicked to enable that email?

There's no actual API, but it can be done - and has been done before -
by screen-scraping the CGI interface. See for example
https://github.com/mhagander/hamn/blob/master/listsync.py

* Quotation. Emails have a higher depth of quotation - it is not uncommon for a single email to contain levels for the past 4 emails and this works well in emails but not in a forum, this makes posts much longer then they need to be in most cases and bloats the search with loads of repeated information.
For example phpBB3 by default limits the max quote level to 3 to stop extraneous information. I'm still thinking of a solution for this.

If you can find a way to represent it "the email way" in email and
"the forum way" in the forums, that's obviously the best...

* Quotation ownership. You will notice that emails take the form something like:
Bob Smith <bob@smith.com> wrote:

1 + 1 = 2?

Yes.

The above gets translated only so that the ">" gets converted into the [quote] tags for the forums display. But quoting in forums adds the ownership into the quote block, so the above would look like (forums do this automatically when you quote someone):
| Quote by [Bob Smith]:
| 1 + 1 = 2?
Yes.

Where [Bob Smith] is a link to the forum user. I originally wrote this into beta1 but saw there was no consistency and emails came in with "wrote:" "writes:" or something that just didn't make sense at all so I disabled the code until I was ready to work on it properly. I will institute it again - wish me luck...

You can't safely rely on that format of the quoting header. "Proper"
quoting will always have > (top-posting often doesn't, but if we can't
parse that into proper quotes, I don't think it's a problem), but you
can't rely on the format of the row(s) before it.

OTOH, if it gets wrong here and there it's not a big problem. it just
mustn't get wrong too often :)

Nobody has made much comment on the permission of containing the mailing list information and member email addresses in another container like a forum. I understand that I can broadcast this information on other mailing lists (www was mentioned) but if theres no absolute authority figure would it make any serious difference from what we're already doing?

We don't deal in authority figures, we deal in authority teams :-)
Just like with the source to the database.

There are some people who read -www that don't read -general. That are
fairly critical. So I suggest moving the thread over there.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tue, Nov 16, 2010 at 16:40, Craig Ringer <craig@postnewspapers.com.au> wrote:

On 11/16/2010 08:43 PM, Elliot Chance wrote:

Alrighty, here are the revised plans for beta3:

--- Changes / Fixes

* Without a doubt everyone one wants the forums to match the mailing
lists. I personally think that this defeats the whole purpose of a forum,
but i'm here to do what the community thinks is best. The extra forums that
are not connected with a mailing list will be removed. And some of the
forums renamed - i'll leave that part up to you to decide amongst
yourselves.

On a bit of a side-note, I'm increasingly wishing Stack Overflow had a mail
interface. I take Greg's point that it's an increasingly key place for
people (especially people not otherwise engaged in the community) to seek
information and help. It's a pity that there doesn't seem to be a good way
to connect the StackOverflow postgresql discussion in some way that makes it
more visible on the main community resources.

I think Greg may have a point when questioning whether adding a forum
interface is overly useful, given that Stack Overflow already exists and
doesn't see all that much attention. OTOH, maybe forums will draw people who
otherwise ask on S.O. to the community, providing a bit of a bridge. It's
worth a try.

I think stack overflow is something different, though, with it's
rating systems and such.

What I'm more interested in is still a word from the people who would
actually *use* a forum on how this would be better than sites like
Nabble and Gmane.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Personally I don't care what kind of "forum" interface is used. I just
don't like the email because while I like to follow the forum, I spend a lot
of time out of the office and I don't like to have to download all of that
mail just to keep up. I'd much rather use something that I can access from
my phone browser. I do this even with my other company email because I
don't want to use up the space on my phone.

Best Regards
Michael Gould

On 17/11/2010 12:01 AM, Michael Gould wrote:

Personally I don't care what kind of "forum" interface is used. I just
don't like the email because while I like to follow the forum, I spend a lot
of time out of the office and I don't like to have to download all of that
mail just to keep up. I'd much rather use something that I can access from
my phone browser. I do this even with my other company email because I
don't want to use up the space on my phone.

Which phone, out of interest? And which server backend?

Even my ancient Nokia E71 (a Symbian Series 60 phone) is capable of
downloading only the most recent "n" messages from a mailbox. Unless
you're stuck with some incredibly brain-dead phone and/or IMAP server
setup, I find it hard to imagine this being a problem.

I'm also unsure how this issue is better addressed by a dedicated forum
than by gmane/nabble/etc.

I now find myself wondering if the existing mail/web gateways like
nabble or gmane offer "skinned" versions under an org's own domain.
That'd be an awfully nice option - have a "forum.postgresql.org" that's
really just gmane/nabble/etc under the hood.

--
Craig Ringer