From: Johannes Schindelin <hidden> Date: 2016-08-24 15:41:57
Hi Arif,
On Tue, 23 Aug 2016, Arif Khokar wrote:
On 08/20/2016 03:57 PM, Jakub Narębski wrote:
quoted
But perhaps the problem is current lack of tooling in the opposite
direction, namely getting patches from mailing list and applying them
to GitHub repo, or Bitbucket, or GitLab. Though with working Git, it
is something easier than sending patches via email; it is enough that
email client can save email to a file (or better, whole sub-thread to
file or files).
Given that public-inbox provides an NNTP interface, couldn't the ARTICLE
<message-id> NNTP command be used to easily retrieve the messages in a
given patch series (at least compared to POP or IMAP). Perhaps
git-send-email could be modified to include the message-id value of each
patch in the series that it sends to the mailing list and include it in
the cover letter.
I am no expert in the NNTP protocol (I abandoned News long ago), but if
you go from HTML, you can automate the process without requiring changes
in format-patch.
Then a script could be written (i.e., git-download-patch) which could
parse the cover letter message (specified using its message-id), and
download all the patches in series, which can then be applied using
git-am. This would in fact take the email client out of the equation in
terms of saving patches.
From: Eric Wong <hidden> Date: 2016-08-24 18:57:51
Johannes Schindelin [off-list ref] wrote:
Hi Arif,
On Tue, 23 Aug 2016, Arif Khokar wrote:
quoted
On 08/20/2016 03:57 PM, Jakub Narębski wrote:
quoted
But perhaps the problem is current lack of tooling in the opposite
direction, namely getting patches from mailing list and applying them
to GitHub repo, or Bitbucket, or GitLab. Though with working Git, it
is something easier than sending patches via email; it is enough that
email client can save email to a file (or better, whole sub-thread to
file or files).
Given that public-inbox provides an NNTP interface, couldn't the ARTICLE
<message-id> NNTP command be used to easily retrieve the messages in a
given patch series (at least compared to POP or IMAP). Perhaps
git-send-email could be modified to include the message-id value of each
patch in the series that it sends to the mailing list and include it in
the cover letter.
I think that makes sense; perhaps an X-Git-Followups: header
from send-email which lists the child Message-IDs the same way
References: does for ancestors. (perhaps there's already a
standardized header for listing children)
I thought about allowing a giant MIME message with all the
patches attached, too but that won't work for a large patch
series due to size limits along various SMTP hops.
Compression might make spam filters unhappy, too.
I am no expert in the NNTP protocol (I abandoned News long ago), but if
you go from HTML, you can automate the process without requiring changes
in format-patch.
quoted
Then a script could be written (i.e., git-download-patch) which could
parse the cover letter message (specified using its message-id), and
download all the patches in series, which can then be applied using
git-am. This would in fact take the email client out of the equation in
terms of saving patches.
w3m -dump -dump_source nntp://<NNTP-server>/<Message-ID>
ought to already work for news.gmane.org and news.public-inbox.org
The Net::NNTP Perl module is a standard part of the Perl distro
for many years, now (along with Net::SMTP), so that would not
be a roadblock for implementing a custom downloader distributed
with git.
I would be hesitant to depend too much on public-inbox.org until
more mirrors appear. Even then, NNTP is a better-established
protocol and a fallback to news.gmane still works.
(public-inbox.org is powered by hamsters running on wheels,
sometimes I let them rest :)
From: Jeff King <hidden> Date: 2016-08-24 19:12:46
On Wed, Aug 24, 2016 at 06:49:38PM +0000, Eric Wong wrote:
quoted
quoted
Given that public-inbox provides an NNTP interface, couldn't the ARTICLE
<message-id> NNTP command be used to easily retrieve the messages in a
given patch series (at least compared to POP or IMAP). Perhaps
git-send-email could be modified to include the message-id value of each
patch in the series that it sends to the mailing list and include it in
the cover letter.
I think that makes sense; perhaps an X-Git-Followups: header
from send-email which lists the child Message-IDs the same way
References: does for ancestors. (perhaps there's already a
standardized header for listing children)
I think that's harder to adapt to some workflows, since it implies
generating all of the message-ids ahead of time (whereas if you are
feeding the messages into an existing MUA, it may generate them on the
fly as it sends).
I thought about allowing a giant MIME message with all the
patches attached, too but that won't work for a large patch
series due to size limits along various SMTP hops.
Compression might make spam filters unhappy, too.
This was a problem faced by binary groups on Usenet, which had to split
large files across many messages.
It has been a long time since I've dealt with those, but I think the
state of the art involved using "1/20", "2/20", etc in the subjects to
piece together the original. There may also have been header or body
content that included a unique id, so you always knew which messages
were part of a set.
They also used things like forward error correction to handle dropped
messages, but I don't think we need to go that far.
So parsing the "PATCH 1/20" headers sounds hacky, but I think it has
worked for years in other communities.
-Peff
From: Eric Wong <hidden> Date: 2016-08-24 19:40:20
Jeff King [off-list ref] wrote:
On Wed, Aug 24, 2016 at 06:49:38PM +0000, Eric Wong wrote:
quoted
quoted
quoted
Given that public-inbox provides an NNTP interface, couldn't the ARTICLE
<message-id> NNTP command be used to easily retrieve the messages in a
given patch series (at least compared to POP or IMAP). Perhaps
git-send-email could be modified to include the message-id value of each
patch in the series that it sends to the mailing list and include it in
the cover letter.
I think that makes sense; perhaps an X-Git-Followups: header
from send-email which lists the child Message-IDs the same way
References: does for ancestors. (perhaps there's already a
standardized header for listing children)
I think that's harder to adapt to some workflows, since it implies
generating all of the message-ids ahead of time (whereas if you are
feeding the messages into an existing MUA, it may generate them on the
fly as it sends).
Yeah, it would be limited to git send-email users, only :<
quoted
I thought about allowing a giant MIME message with all the
patches attached, too but that won't work for a large patch
series due to size limits along various SMTP hops.
Compression might make spam filters unhappy, too.
This was a problem faced by binary groups on Usenet, which had to split
large files across many messages.
It has been a long time since I've dealt with those, but I think the
state of the art involved using "1/20", "2/20", etc in the subjects to
piece together the original. There may also have been header or body
content that included a unique id, so you always knew which messages
were part of a set.
They also used things like forward error correction to handle dropped
messages, but I don't think we need to go that far.
So parsing the "PATCH 1/20" headers sounds hacky, but I think it has
worked for years in other communities.
nzb (an XML format) seems to be the thing for Usenet binaries,
nowadays. Maybe it's workable for git, maybe it's overkill or
not worth it for the two (non-.onion) NNTP servers we have.
nzb seems widely supported enough (on a Debian jessie system):
$ apt-cache search nzb
sabnzbdplus - web-based binary newsgrabber with nzb support
sabnzbdplus-theme-classic - classic interface templates for the SABnzbd+ binary newsgrabber
sabnzbdplus-theme-iphone - transitional package for migration to sabnzbdplus-theme-mobile
sabnzbdplus-theme-mobile - mobile interface templates for the SABnzbd+ binary newsgrabber
sabnzbdplus-theme-plush - plush interface templates for the SABnzbd+ binary newsgrabber
sabnzbdplus-theme-smpl - smpl interface templates for the SABnzbd+ binary newsgrabber
libnzb-dev - An nzb based Usenet binary grabber (development files)
libnzb0c2a - An nzb based Usenet binary grabber (runtime library)
lottanzb - simple and automated Usenet downloader for Newzbin (NZB) files
nzb - Usenet binary grabber
nzbget - command-line based binary newsgrabber for nzb files
python-pynzb - unified API for parsing NZB files from NNTP (Usenet) servers
spotweb - web interface to search and filter Usenet spots
From: Johannes Schindelin <hidden> Date: 2017-02-10 16:12:30
Hi Arif,
On Wed, 24 Aug 2016, Johannes Schindelin wrote:
On Tue, 23 Aug 2016, Arif Khokar wrote:
quoted
On 08/20/2016 03:57 PM, Jakub Narębski wrote:
quoted
But perhaps the problem is current lack of tooling in the opposite
direction, namely getting patches from mailing list and applying
them to GitHub repo, or Bitbucket, or GitLab. Though with working
Git, it is something easier than sending patches via email; it is
enough that email client can save email to a file (or better, whole
sub-thread to file or files).
Given that public-inbox provides an NNTP interface, couldn't the
ARTICLE <message-id> NNTP command be used to easily retrieve the
messages in a given patch series (at least compared to POP or IMAP).
Perhaps git-send-email could be modified to include the message-id
value of each patch in the series that it sends to the mailing list
and include it in the cover letter.
I am no expert in the NNTP protocol (I abandoned News long ago), but if
you go from HTML, you can automate the process without requiring changes
in format-patch.
quoted
Then a script could be written (i.e., git-download-patch) which could
parse the cover letter message (specified using its message-id), and
download all the patches in series, which can then be applied using
git-am. This would in fact take the email client out of the equation
in terms of saving patches.
Oh well. That would have been too easy a task, right?
As it happens, I needed this functionality myself (when reworking my
git-path-in-subdir patch to include Mike Rappazzo's previous patch series
that tried to fix the same bug).
I copy-edited the script to work with public-inbox.org, it accepts a
Message-ID or URL or GMane URL and will try to apply the patch (or patch
series) on top of the current revision:
https://github.com/git-for-windows/build-extra/blob/2268850552c7/apply-from-public-inbox.sh
Ciao,
Johannes
Oh well. That would have been too easy a task, right?
As it happens, I needed this functionality myself (when reworking my
git-path-in-subdir patch to include Mike Rappazzo's previous patch series
that tried to fix the same bug).
I copy-edited the script to work with public-inbox.org, it accepts a
Message-ID or URL or GMane URL and will try to apply the patch (or patch
series) on top of the current revision:
https://github.com/git-for-windows/build-extra/blob/2268850552c7/apply-from-public-inbox.sh
Thanks for the link. One thing that comes to mind that is that it may
be better to just download the patches and then manually apply them
afterwords rather than doing it in the script itself. Or at least add
an option to the script to not automatically invoke git am.
Getting back to the point I made when this thread was still active, I
still think it would be better to be able to list the message-id values
in the header or body of the cover letter message of a patch series
(preferably the former) in order to facilitate downloading the patches
via NNTP from gmane or public-inbox.org. That would make it easier
compared to the different, ad-hoc, methods that exist for each email client.
Alternatively, or perhaps in addition to the list of message-ids, a list
of URLs to public-inbox.org or gmane messages could also be provided for
those who prefer to download patches via HTTP.
Oh well. That would have been too easy a task, right?
As it happens, I needed this functionality myself (when reworking my
git-path-in-subdir patch to include Mike Rappazzo's previous patch
series that tried to fix the same bug).
I copy-edited the script to work with public-inbox.org, it accepts a
Message-ID or URL or GMane URL and will try to apply the patch (or
patch series) on top of the current revision:
https://github.com/git-for-windows/build-extra/blob/2268850552c7/apply-from-public-inbox.sh
Thanks for the link. One thing that comes to mind that is that it may
be better to just download the patches and then manually apply them
afterwords rather than doing it in the script itself. Or at least add
an option to the script to not automatically invoke git am.
I actually had expected *you* to put in a little bit of an effort, too. In
fact, I was very disappointed that you did not even look into porting that
script to use public-inbox instead of GMane.
Getting back to the point I made when this thread was still active, I
still think it would be better to be able to list the message-id values
in the header or body of the cover letter message of a patch series
(preferably the former) in order to facilitate downloading the patches
via NNTP from gmane or public-inbox.org. That would make it easier
compared to the different, ad-hoc, methods that exist for each email
client.
You can always do that yourself: you can modify your cover letter to
include that information.
Note that doing this automatically in format-patch may not be appropriate,
as 1) the Message-ID could be modified depending on the mail client used
to send the mails, and 2) it is not unheard of that a developer
finds a bug in the middle of sending a patch series, fixes that bug, and
regenerates the remainder of the patch series, completely rewriting those
Message-IDs.
Alternatively, or perhaps in addition to the list of message-ids, a list
of URLs to public-inbox.org or gmane messages could also be provided for
those who prefer to download patches via HTTP.
At this point, I am a little disinterested in a discussion without code. I
brought some code to the table, after all.
Ciao,
Johannes
From: Arif Khokar <hidden> Date: 2017-02-14 03:56:47
On 02/13/2017 09:37 AM, Johannes Schindelin wrote:
Hi Arif,
On Mon, 13 Feb 2017, Arif Khokar wrote:
quoted
Thanks for the link. One thing that comes to mind that is that it may
be better to just download the patches and then manually apply them
afterwords rather than doing it in the script itself. Or at least add
an option to the script to not automatically invoke git am.
I actually had expected *you* to put in a little bit of an effort, too. In
fact, I was very disappointed that you did not even look into porting that
script to use public-inbox instead of GMane.
I wasn't aware of that expectation. My idea was to use NNTP as a way to
facilitate the development of a new git utility that would serve as the
inverse of git-send-email (sort of like the relationship between git
format-patch and git am), rather than using a
IIRC, I had posted some proof-of-concept Perl code to do so back in
August in
[off-list ref]
Looking at public-inbox now at the archives of this group, it appears
that several of the messages I sent weren't archived for some reason
(and I didn't see any more responses to what I posted at the time). The
messages are accessible via NNTP when connecting to gmane though.
Also, looking at the source of the message I referenced, it appears that
my MUA decided to base64 encode the message for some reason (which may
have resulted in it getting filtered by those who I sent the message to).
I will look into this more now (given yours and Junio's responses).
quoted
Getting back to the point I made when this thread was still active, I
still think it would be better to be able to list the message-id values
in the header or body of the cover letter message of a patch series
(preferably the former) in order to facilitate downloading the patches
via NNTP from gmane or public-inbox.org. That would make it easier
compared to the different, ad-hoc, methods that exist for each email
client.
You can always do that yourself: you can modify your cover letter to
include that information.
Certainly, but it would be nice to be able to have it done automatically
by git format-patch (which I'll look into).
Note that doing this automatically in format-patch may not be appropriate,
as 1) the Message-ID could be modified depending on the mail client used
to send the mails
I think the best approach would be not to make including the message-id
values the default behavior. Specifying a new command-line option to
enable that behavior should address those concerns I think.
and 2) it is not unheard of that a developer
finds a bug in the middle of sending a patch series, fixes that bug, and
regenerates the remainder of the patch series, completely rewriting those
Message-IDs.
Perhaps, but should something like that not warrant a re-roll of sorts.
That is, one should reply to the partial patch series stating that there
is a bug that renders this particular patch (series) un-usable and the
re-roll could be posted as a reply to the original cover letter?
quoted
Alternatively, or perhaps in addition to the list of message-ids, a list
of URLs to public-inbox.org or gmane messages could also be provided for
those who prefer to download patches via HTTP.
At this point, I am a little disinterested in a discussion without code. I
brought some code to the table, after all.
If you have the time, please take a look at the message-id I referenced.
If you need, I can re-post the proof-of-concept code.
From: Arif Khokar <hidden> Date: 2017-02-14 04:00:02
On 02/13/2017 10:56 PM, Arif Khokar wrote:
I wasn't aware of that expectation. My idea was to use NNTP as a way to
facilitate the development of a new git utility that would serve as the
inverse of git-send-email (sort of like the relationship between git
format-patch and git am), rather than using a
...custom script that's tightly coupled to gmane and public-inbox.org
From: Eric Wong <hidden> Date: 2017-02-14 07:13:32
Arif Khokar [off-list ref] wrote:
On 02/13/2017 09:37 AM, Johannes Schindelin wrote:
quoted
I actually had expected *you* to put in a little bit of an effort, too. In
fact, I was very disappointed that you did not even look into porting that
script to use public-inbox instead of GMane.
I wasn't aware of that expectation. My idea was to use NNTP as a way to
facilitate the development of a new git utility that would serve as the
inverse of git-send-email (sort of like the relationship between git
format-patch and git am), rather than using a
Speaking for myself, I usually don't expect much, especially
from newcomers. So I am disappointed to see Dscho's disappointment
aimed at you, Arif. Especially since you're not a regular and
we have no idea how much free time, attention span, or familiarity
with Bourne shell you have.
IIRC, I had posted some proof-of-concept Perl code to do so back in August
in [off-list ref]
Looking at public-inbox now at the archives of this group, it appears that
several of the messages I sent weren't archived for some reason (and I
didn't see any more responses to what I posted at the time). The messages
are accessible via NNTP when connecting to gmane though.
Also, looking at the source of the message I referenced, it appears that my
MUA decided to base64 encode the message for some reason (which may have
resulted in it getting filtered by those who I sent the message to).
It probably wasn't base64, but maybe it was one of these:
http://vger.kernel.org/majordomo-taboos.txt
Or it was the SPF softfail which you can see in the headers on both
gmane and public-inbox.
It might even be the '_' (underscore) in your other address.
But even Junio gets dropped by vger sometimes:
https://public-inbox.org/git/20170127035753.GA2604@dcvr/
But if I had to guess, vger gets hit by truckloads of spam and
the the backscatter volume could become unimaginable, so perhaps
it has good reason to discard silently.
Anyways, the eventual goal of public-inbox is to flip the
mailing list model backwards into "archives first" mode,
so a message needs to make it into public archives before
it goes out to subscribers. That might prevent or avoid
such problems... *shrug*