Watch, Follow, &
Connect with Us

Welcome, Guest
Guest Settings
Help

Thread: Getting ALL new messages from IMAP



Permlink Replies: 4 - Last Post: May 22, 2017 2:26 PM Last Post By: Asger Joergensen Threads: [ Previous | Next ]
Asger Joergensen

Posts: 367
Registered: 11/18/08
Getting ALL new messages from IMAP
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 17, 2017 3:20 AM
Hi

Just to make sure I'm understanding the RFC 3501 right

[Quote]
The COPY command copies the specified message(s) to the end of the
specified destination mailbox. The flags and internal date of the
message(s) SHOULD be preserved, and the Recent flag SHOULD be set,
in the copy.
[Quote end]

I notice the: "to the end of the specified destination mailbox". Which to
me say, that they will be assigned UIDs, higher then the last one in the
mailbox

I also notice the:
"The flags and internal date of the message(s) SHOULD be preserved".
To me this say, that UIDSearchMailBox with skSince, will probably not get
these copied messages and using the skRecent or skNew flag will only
get these messages, if the user that copied those messages, didn't verify
his copying by opening the mailbox.

I have noticed that UIDSearchMailBox, used with skAll or skSince and a very
old date, returns a UID list similar to the one I get using IdPOP3->UIDL.

So my plan is:
If UIDValidity is the same as last
UIDList = UIDSearchMailBox with skSince with an old date;
Copy UIDList to CopyList
loop through LastUIDList and delete all in UIDList that match
Download all those messages left in UIDList (new since last time)
Assign CopyList To LastUIDList
Save DateTime for this download.
else
Ask user if they want to download all or since last download and
then I save the new UID list.

Am I missing something ?

Thanks in advance
Best regards
Asger
Remy Lebeau (Te...


Posts: 8,305
Registered: 12/23/01
Re: Getting ALL new messages from IMAP
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 17, 2017 11:42 AM   in response to: Asger Joergensen in response to: Asger Joergensen
Asger Joergensen wrote:
Just to make sure I'm understanding the RFC 3501 right

First, keep in mind that SHOULD is not the same as MUST, so be prepared to handle cases where the mentioned behavior is NOT actually implemented by various servers.

[Quote]
The COPY command copies the specified message(s) to the end of the
specified destination mailbox. The flags and internal date of the
message(s) SHOULD be preserved, and the Recent flag SHOULD be set,
in the copy.
[Quote end]

I notice the: "to the end of the specified destination mailbox". Which to
me say, that they will be assigned UIDs, higher then the last one in the
mailbox

That is guaranteed by the definition of UID in section 2.3.1.1:

A 32-bit value assigned to each message, which when used with the unique identifier validity value (see below) forms a 64-bit value that MUST NOT refer to any other message in the mailbox or any subsequent mailbox with the same name forever. **Unique identifiers are assigned in a strictly ascending fashion in the mailbox; as each message is added to the mailbox it is assigned a higher UID than the message(s) which were added previously**. Unlike message sequence numbers, unique identifiers are not necessarily contiguous.

However, it also says:

The unique identifier of a message MUST NOT change during the session, and SHOULD NOT change between sessions. **Any change of unique identifiers between sessions MUST be detectable using the UIDVALIDITY mechanism discussed below**. Persistent unique identifiers are required for a client to resynchronize its state from a previous session with the server (e.g., disconnected or offline access clients); this is discussed further in [IMAP-DISC].

So you do have to be prepared to handle the possibility that UIDs can change between different IMAP sessions.

Also note:

Associated with every mailbox are two values which aid in unique identifier handling: the next unique identifier value and the unique identifier validity value.

The next unique identifier value is the predicted value that will be assigned to a new message in the mailbox. Unless the unique identifier validity also changes (see below), the next unique identifier value MUST have the following two characteristics. First, the next unique identifier value MUST NOT change unless new messages are added to the mailbox; and second, the next unique identifier value MUST change whenever new messages are added to the mailbox, even if those new messages are subsequently expunged.

Note: The next unique identifier value is intended to provide a means for a client to determine whether any messages have been delivered to the mailbox since the previous time it checked this value. It is not intended to provide any guarantee that any message will have this unique identifier. A client can only assume, at the time that it obtains the next unique identifier value, that messages arriving after that time will have a UID greater than or equal to that value.

Also note the following from section 6.3.1 for the SELECT command:

Before returning an OK to the client, the server MUST send the following untagged data to the client.

...

OK [UIDNEXT <n>]
The next unique identifier value. Refer to section 2.3.1.1 for more information. **If this is missing, the client can not make any assumptions about the next unique identifier value.**

OK [UIDVALIDITY <n>]
The unique identifier validity value. Refer to section 2.3.1.1 for more information. **If this is missing, the server does not support unique identifiers.**

...


So, given this information, if the server supports UIDs, you can keep track of the mailbox's last known UIDVALIDITY and UIDNEXT values, and upon re-selecting the same mailbox at a later time, detect if the existing UIDs are still consistent with the last time (if the UIDVALIDITY has not changed), and if so then you can do a SEARCH for new UIDs that are the same or higher than the last known UIDNEXT, using the skUID search key with a range of UIDs, eg:

IdIMAP4->SelectMailBox(...);
if ((IdIMAP4->MailBox->UIDValidity != "") &&
    (IdIMAP4->MailBox->UIDNext != "") &&
    (IdIMAP4->MailBox->UIDValidity == LastKnownUIDValidity) &&
    (IdIMAP4->MailBox->UIDNext != LastKnownUIDNext))
{
    TIdIMAP4SearchRec sr;
    sr.SearchKey = skUID;
    sr.Text = LastKnownUIDNext + ":*";
 
    IdIMAP4->UIDSearchMailBox(&sr, 0);
    ...
}
else
{
    // re-sync mailbox as needed...
    LastKnownUIDValidity = IdIMAP4->MailBox->UIDValidity;
    LastKnownUIDNext = IdIMAP4->MailBox->UIDNext;
}


The mailbox's UIDValidity and UIDNext properties are updated by the StatusMailBox(), SelectMailBox(), and ExamineMailBox() methods. And speaking of that last method:

The EXAMINE command is identical to SELECT and returns the same output; however, the selected mailbox is identified as read-only. No changes to the permanent state of the mailbox, including per-user state, are permitted; in particular, EXAMINE MUST NOT cause messages to lose the \Recent flag.

I also notice the:

"The flags and internal date of the message(s) SHOULD be preserved".

To me this say, that UIDSearchMailBox with skSince, will probably not get
these copied messages

We already established that earlier.

and using the skRecent or skNew flag will only get these messages, if the
user that copied those messages, didn't verify his copying by opening the
mailbox.

Simply opening the mailbox does not change the flags on the messages, the messages themselves have to be opened and/or updated directly.

I have noticed that UIDSearchMailBox, used with skAll or skSince and a very
old date, returns a UID list similar to the one I get using IdPOP3->UIDL.

So my plan is:
If UIDValidity is the same as last
UIDList = UIDSearchMailBox with skSince with an old date;

Why would you use an old date if you can use skAll instead? You could also use skUID with a range of "0:" or "1:" (not sure if UIDs start at 0 or 1). However, the problem with this approach is that as the mailbox grows over time, your search will take longer and longer to run, unless old messages are deleted periodically. Once you know you have caught up with the current mailbox content, you can detect new UIDs (see above) and just process those.

--
Remy Lebeau (TeamB)
Asger Joergensen

Posts: 367
Registered: 11/18/08
Re: Getting ALL new messages from IMAP [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 17, 2017 2:59 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:


So you do have to be prepared to handle the possibility that UIDs can change
between different IMAP sessions.

Yes and I will be.:-)

So, given this information, if the server supports UIDs,

Are there proper IMAP servers that doesn't support UID's ????
By proper I mean servers that are used by email providers.

IdIMAP4->SelectMailBox(...);
if ((IdIMAP4->MailBox->UIDValidity != "") &&
    (IdIMAP4->MailBox->UIDNext != "") &&
    (IdIMAP4->MailBox->UIDValidity == LastKnownUIDValidity) &&
    (IdIMAP4->MailBox->UIDNext != LastKnownUIDNext))
{
    TIdIMAP4SearchRec sr;
    sr.SearchKey = skUID;
    sr.Text = LastKnownUIDNext + ":*";
 
    IdIMAP4->UIDSearchMailBox(&sr, 0);
    ...
}
else
{
    // re-sync mailbox as needed...
    LastKnownUIDValidity = IdIMAP4->MailBox->UIDValidity;
    LastKnownUIDNext = IdIMAP4->MailBox->UIDNext;
}

Yes, this most certainly could be a good alternative to my approach the
IdIMAP4->MailBox->UIDNext != LastKnownUIDNext part is brilliant, thanks!

I am testing on a mailbox with 649 messages and it takes about 1 sec.
whether I get the last three or all UID's and the list is only 4KB
My code for finding the new UID's in the list runs for less then 1 millisec.
So I guess the easiest way to recover from UID changes will determine
which way to go. Must do some thinking.:-)

Why do You check for empty string "" ??
and if it is there should that initiate a re-sync of the mailbox ??

I now check if UIDValidity and UIDNext are supported, when the account is created,
if they are empty, I reject the setup and tell the user to try POP3 instead.

Can these be empty for other reasons ??

I have noticed that UIDSearchMailBox, used with skAll or skSince and a very
old date, returns a UID list similar to the one I get using IdPOP3->UIDL.

So my plan is:
If UIDValidity is the same as last
UIDList = UIDSearchMailBox with skSince with an old date;

Why would you use an old date if you can use skAll instead?

Because the user can set a start date (oldest message) and the user probably
don't want to go more then a couple of years back even though they might want
to keep all their mails in the mailbox as history. In case of UID changes, one
year back is the max that is needed.

Thanks you so much for helping
Best regards
Asger
Remy Lebeau (Te...


Posts: 8,305
Registered: 12/23/01
Re: Getting ALL new messages from IMAP [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 22, 2017 1:55 PM   in response to: Asger Joergensen in response to: Asger Joergensen
Asger Joergensen wrote:

Are there proper IMAP servers that doesn't support UID's ????
By proper I mean servers that are used by email providers.

I don't know. But the IMAP spec allows for it, so best to write code to handle it.

Why do You check for empty string "" ??

Because the UIDValidity and UIDNext fields are strings, and if the server does not provide values for them then they will be blank.

I now check if UIDValidity and UIDNext are supported, when the account is created.

You need to check them every time you start a new IMAP session, since they can change value between sessions.

if they are empty, I reject the setup and tell the user to try POP3 instead.

I wouldn't do that, since IMAP is still preferable over POP3 in general, if for no other reason then the added SEARCH functionality. Use IMAP if it is available, otherwise fallback to POP3 if you have no other option.

Can these be empty for other reasons ??

Only if the server does not support them.

--
Remy Lebeau (TeamB)
Asger Joergensen

Posts: 367
Registered: 11/18/08
Re: Getting ALL new messages from IMAP [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 22, 2017 2:26 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

Asger Joergensen wrote:

I now check if UIDValidity and UIDNext are supported, when the account is created.

You need to check them every time you start a new IMAP session, since they can
change value between sessions.

Yes I do that to.

if they are empty, I reject the setup and tell the user to try POP3 instead.

I wouldn't do that, since IMAP is still preferable over POP3 in general, if
for no other reason then the added SEARCH functionality. Use IMAP if it is
available, otherwise fallback to POP3 if you have no other option.

But if I cant search on UIDs, I cant be sure I get all new messages, which means
I will have to fall back, to something similar as what I do in POP3, so why not
just use POP3, where I already have it set up ?

Thanks for helping
Best regards
Asger
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02