From: Smylers Date: 05:28 on 31 Oct 2006 Subject: OfflineImap OfflineImap crashes partway through syncing my mail with: Thread 'New msg sync from spam' terminated with exception: Traceback (most recent call last): [Snipped many lines of Python stack trace] error: APPEND command error: BAD ['Invalid internal date.'] Last 11 debug messages logged for New msg sync from spam prior to exception: imap: savemessage: called imap: savemessage: using date "31-Nov-2006 02:50:01 +0000" [Snipped the entire spam message] So you've found something in my spam folder which has an invalid header? Oooh, that can't possibly be right, because obviously spammers _always_ make sure to follow all the relevant RFCs. And you've decided that this invalid header prevents you from being able copy the message between this computer and the imap server, rather than just, say, copying the invalid message as it is and letting me have a message with a stupid date in it? And this is despite the fact that the message is in the spam folder on this computer because I moved it there from my inbox, and it got to the inbox my, erm, you! So you can happily sync invalid messages from the server to this computer, just not cope with sending them in t'other direction? And further you've decided that your inability to copy this one, spam, message means you should abort any attempts to sync any of the rest of my mail, even that in other folders, until I've used something else to move the broken message out of the way? Hate! Smylers
From: Phil Pennock Date: 13:49 on 31 Oct 2006 Subject: Re: OfflineImap On 2006-10-31 at 05:28 +0000, Smylers wrote: > OfflineImap crashes partway through syncing my mail with: > > Thread 'New msg sync from spam' terminated with exception: > Traceback (most recent call last): > [Snipped many lines of Python stack trace] > error: APPEND command error: BAD ['Invalid internal date.'] That BAD is a message from the IMAP server which you're syncing to. OfflineIMAP's problem here apparently is that it's not gracefully coping with an error from the server. Buggy, yes. Rejecting bad dates, no. IMAP returns OK, NO or BAD in response to commands. I mostly see BAD from Cyrus IMAP, which can be rather picky. The official stance from the Cyrus team is roughly along the lines of: it's better to not expose flakey mail-clients to potentially bad stuff and undefined behaviour, so just reject it before it gets here. > And this is despite the fact that the message is in the spam folder on > this computer because I moved it there from my inbox, and it got to the > inbox my, erm, you! So you can happily sync invalid messages from the > server to this computer, just not cope with sending them in t'other > direction? Your local server is more forgiving; the rejecting server apparently has different checks for mail delivery from the MTA and for IMAP APPEND. Possibly the problem is that the complaint is for the internal date & time, which is per RFC set to time message received unless set via an IMAP client. In which case, the problem is explicitly in OfflineIMAP setting the internal time, not the message-header time. > Hate! Target the hate and it's more intense and is more likely to help you kick the hateful software for something more soothing. If such soothing software exists. Which I doubt. If the problem is OfflineIMAP providing a bad time string in the append, then the easiest hack is to probably to change the python code to simply not provide it -- it's optional. RFC 3501 §6.3.11 (ooh look, linked to that other recent thread). The tricker but "better" fix would to ensure that the internal times are preserved between servers by syncing the right data. -Phil
From: Smylers Date: 17:37 on 31 Oct 2006 Subject: Re: OfflineImap Phil Pennock writes: > On 2006-10-31 at 05:28 +0000, Smylers wrote: > > > OfflineImap crashes partway through syncing my mail with: > > > > Thread 'New msg sync from spam' terminated with exception: > > Traceback (most recent call last): > > [Snipped many lines of Python stack trace] > > error: APPEND command error: BAD ['Invalid internal date.'] > > That BAD is a message from the IMAP server which you're syncing to. Ah, thanks for that. > OfflineIMAP's problem here apparently is that it's not gracefully > coping with an error from the server. Buggy, yes. Rejecting bad > dates, no. > > IMAP returns OK, NO or BAD in response to commands. I mostly see BAD > from Cyrus IMAP, which can be rather picky. It's Dovecot in this case. > it's better to not expose flakey mail-clients to potentially bad stuff > and undefined behaviour, so just reject it before it gets here. That's fine, except ... the only reason my local mail folders have that mail at all is because they received it from that imap server, via OfflineImap. Dovecot is happy to let me download a mail message and then later objects to me uploading that very same message that it's given me! All I've done is move it from my inbox to my spam folder (and told OfflineImap to sync the change). If Dovecot won't accept such messages then it shouldn't let me download them in the first place! > Possibly the problem is that the complaint is for the internal date & > time, which is per RFC set to time message received unless set via an > IMAP client. In which case, the problem is explicitly in OfflineIMAP > setting the internal time, not the message-header time. The errant messages (for there have been more of them during today) have a Date: field set to November 31st (Mutt displays it as December 1st). What should OfflineImap send to Dovecot in this case? Smylers
From: Phil Pennock
Date: 02:25 on 01 Nov 2006
Subject: Re: OfflineImap
On 2006-10-31 at 17:37 +0000, Smylers wrote:
> The errant messages (for there have been more of them during today) have
> a Date: field set to November 31st (Mutt displays it as December 1st).
> What should OfflineImap send to Dovecot in this case?
Nothing to do with the Date: header. If Dovecot is rejecting the mail
based on the content, it's b0rked. The relevant time is meta-data about
the mail, the internal date, which should be preserved. OfflineIMAP
should be preserving this meta-data.
Two blurbs of cut&paste follow; the first is evidence with which to LART
software authors, whilst the second is a simple IMAP transaction showing
a FETCH retrieving the internal date. (The OpenBSD 4.0 announcement
email).
FETCH can retrieve multiple pieces of data; there are three
macro-equivalent names defined in RFC3501, "ALL", "FAST" & "FULL"; all
three include "INTERNALDATE". If that's not a big bloody hint that this
is data that clients should care about, well ... gaah.
RFC 3501 (IMAP) has:
-----------------------------< cut here >-------------------------------
2.3.3. Internal Date Message Attribute
The internal date and time of the message on the server. This
is not the date and time in the [RFC-2822] header, but rather a
date and time which reflects when the message was received. In
the case of messages delivered via [SMTP], this SHOULD be the
date and time of final delivery of the message as defined by
[SMTP]. In the case of messages delivered by the IMAP4rev1 COPY
command, this SHOULD be the internal date and time of the source
message. In the case of messages delivered by the IMAP4rev1
APPEND command, this SHOULD be the date and time as specified in
the APPEND command description. All other cases are
implementation defined.
-----------------------------< cut here >-------------------------------
-----------------------------< cut here >-------------------------------
% mailcheck-imap -NR
SSL connection [AES256-SHA]
Subject Name: /C=NL/ST=Noord Holland/O=GlobNIX Systems/CN=imap.spodhuis.org/emailAddress=postmaster@xxxxxxxx.xxx
Issuer Name: /C=NL/ST=Noord Holland/L=Noord-Scharwoude/O=GlobNIX Systems/CN=GlobNIX Certificate Authority/emailAddress=certificates@xxxxxxx.xxx
GSSAPI server ID: imap/imap.spodhuis.org@xxxxxxxx.xxx
a6>> SELECT "Shared Folders/globnix/openbsd-lists"
PerlIO layers [stdio]:
>>> a6 SELECT "Shared Folders/globnix/openbsd-lists"
<<< * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
<<< * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]
<<< * 1 EXISTS
<<< * 0 RECENT
<<< * OK [UIDVALIDITY 1162307062]
<<< * OK [UIDNEXT 3]
<<< * OK [NOMODSEQ] Sorry, modsequences have not been enabled on this mailbox
<<< a6 OK [READ-WRITE] Completed
a7>> FETCH 1 INTERNALDATE
PerlIO layers [stdio]:
>>> a7 FETCH 1 INTERNALDATE
<<< * 1 FETCH (INTERNALDATE " 1-Nov-2006 01:05:09 +0000")
<<< a7 OK Completed (0.000 sec)
a8>> FETCH 1 FULL
PerlIO layers [stdio]:
>>> a8 FETCH 1 FULL
<<< * 1 FETCH (FLAGS (\Seen) INTERNALDATE " 1-Nov-2006 01:05:09 +0000" RFC822.SIZE 30046 ENVELOPE ("Tue, 31 Oct 2006 17:15:58 -0700" "OpenBSD 4.0 released Nov 1, 2006" (("Theo de Raadt" NIL "deraadt" "cvs.openbsd.org")) ((NIL NIL "owner-announce" "openbsd.org")) (("Theo de Raadt" NIL "deraadt" "cvs.openbsd.org")) ((NIL NIL "announce" "openbsd.org")) NIL NIL NIL "<200611010015.kA10FwQx017003@xxx.xxxxxxx.xxx>") BODY ("TEXT" "PLAIN" ("CHARSET" "us-ascii") NIL NIL "7BIT" 28688 577))
<<< a8 OK Completed (0.000 sec)
-----------------------------< cut here >-------------------------------
Generated at 10:27 on 16 Apr 2008 by mariachi