Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: TIdIMAP4, how to download only new message headers



Permlink Replies: 10 - Last Post: Apr 3, 2017 2:49 PM Last Post By: Asger Joergensen
Asger Joergensen

Posts: 370
Registered: 11/18/08
TIdIMAP4, how to download only new message headers
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 24, 2017 6:02 PM
Hi

I'm trying to learn how to use TIdIMAP4, so I have some questions,
in relation to only downloading the latest emails.

I was thinking to use the ID, but are they reliable when others client
use the same mailbox ?

The ID's are numerical, on the server that I use for testing, is that
the normal practice ?

I was thinking to convert the ID to int and then only download ID's
larger then the last one that i already downloaded.

I have seen mfRecent and mfSeen, but if there is more then one client
accessing the same mailbox, then the flags can be set by another client,
right ?

Is there an easy(fast) way of getting the timestamp for a message, so that
I could use a combination of date and ID.

Thanks in advance
Best regards
Asger
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TIdIMAP4, how to download only new message headers
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 27, 2017 12:00 PM   in response to: Asger Joergensen in response to: Asger Joergensen
Asger wrote:

I'm trying to learn how to use TIdIMAP4, so I have some questions,
in relation to only downloading the latest emails.

TIdIMAP4 has (UID)SearchMailbox() methods. You can do a search for flags
like skNew (recent unseen messages), or received skSince a specified date/time,
etc.

I was thinking to use the ID, but are they reliable when others client
use the same mailbox ?

UIDs (versus the Message-ID header inside the emails) are assigned by the
server, not by the clients. UIDs are persistent within a given IMAP session
and guaranteed to be unique within a given mailbox (but not across mailboxes).
UIDs can change between sessions, but that is detectable. See:

Unique ID in IMAP protocol
https://www.limilabs.com/blog/unique-id-in-imap-protocol

TIdMailBox has a UIDValidity property, which is populated by TIdIMAP4's StatusMailBox()
and SelectMailBox() methods.

The ID's are numerical, on the server that I use for testing, is that
the normal practice ?

UIDs are numerical in nature, yes, but Indy passes them around as strings,
not as integers.

I was thinking to convert the ID to int and then only download ID's
larger then the last one that i already downloaded.

That is not correct or reliable.

I have seen mfRecent and mfSeen, but if there is more then one client
accessing the same mailbox, then the flags can be set by another
client, right ?

Probably, yes.

Is there an easy(fast) way of getting the timestamp for a message, so
that I could use a combination of date and ID.

That is a little tricky. IMAP has a notion of an "internal" date/time that
is separate from the date/time specified in the email headers. The "internal"
date/time is the message's actual date/time within the mailbox. Unfortunately,
while IMAP does have a FETCH command that can retreive this "internal" date/time,
Indy's TIdIMAP4 does not expose access to that particular value, so you will
likely have to manually issue a FETCH command and parse its response yourself.

--
Remy Lebeau (TeamB)
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TIdIMAP4, how to download only new message headers
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 27, 2017 12:01 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy wrote:

I have seen mfRecent and mfSeen, but if there is more then one
client accessing the same mailbox, then the flags can be set by
another client, right ?
Probably, yes.

However, the mailbox belongs to a specific user, so the real question is,
why do you have multiple clients accessing the same mailbox at all?

--
Remy Lebeau (TeamB)
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: TIdIMAP4, how to download only new message headers
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 27, 2017 1:52 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

Probably, yes.

However, the mailbox belongs to a specific user, so the real question is,
why do you have multiple clients accessing the same mailbox at all?

The client I write is not a mail client, it is a program that fetch all
attachments (CAD drawings) from a mailbox, the actual mails are handled by
real mail clients.

And I also use my real mail client in both my office and in my workshop,
so the same mailbox can easily be accessed by two different client.

Best regards
Asger
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: TIdIMAP4, how to download only new message headers
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 27, 2017 1:55 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Lot of things to investigate.
Thank you very much.

Best regards
Asger
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: TIdIMAP4, how to download only new message headers
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 27, 2017 2:43 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

I was thinking to convert the ID to int and then only download ID's
larger then the last one that i already downloaded.

That is not correct or reliable.

If I understand the explanation in the link you posted about:
<quote>
Unique ID in IMAP protocol
.....
A 32-bit value assigned to each message
.....
UIDs are assigned in ascending fashion – higher the value newer the message.
<quote end>

Then it should be ok to fetch all new ID's from the server like this:

   int lastID = 103; // example of highest id downloaded last session
 
   int C = IdIMAP4->MailBox->TotalMsgs;
   for( int i = C; i > 0; --i )
   {
      String Uid;
      IdIMAP4->GetUID( i, Uid );
      if(  Uid.ToInt() > lastID  ) )
         DownloadList->Add( Uid );
      else
         break;
   }


I'm not sure though if it is ok to stop checking, when I meat the first
ID lower then the one I saved from last session or I have to check all
ID's in the mailbox.

Thanks for helping
Best regards
Asger
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TIdIMAP4, how to download only new message headers
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 27, 2017 4:10 PM   in response to: Asger Joergensen in response to: Asger Joergensen
Asger wrote:

If I understand the explanation in the link you posted about:
<snip>
Then it should be ok to fetch all new ID's from the server like this:

Maybe, but you would have to first make sure the mailbox's UIDValidity value
hasn't changed. If it does, the existing UIDs could have changed as well.

I would not rely on UIDs for tis purpose at all. They are fine for downloading
message details, but I would use (UID)SearchMailBox() instead for determining
which emails to download. For example, if you know the date/time of the
last downloaded message, you could then do something like this:

TDateTime lastDateTime = ...; // example of highest date/time downloaded 
last session
 
TIdIMAP4SearchRec item;
item.Date = lastDateTime;
item.SearchKey = skSince; // or skSentSince
 
if (IdIMAP4->UIDSearchMailBox(&item, 0))
{
    int cnt = IdIMAP4->MailBox->SearchResult.Length;
    for (int i = 0; i < cnt; ++i)
        DownloadList->Add( IdIMAP4->MailBox->SearchResult[i] );
}


Not only is it faster, but it uses much less bandwidth, since it is only
1 SEARCH command, instead of individual FETCH UID commands sent N times.

--
Remy Lebeau (TeamB)
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: TIdIMAP4, how to download only new message headers
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 28, 2017 3:11 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

TDateTime lastDateTime = ...; // example of highest date/time downloaded 
last session
 
TIdIMAP4SearchRec item;
item.Date = lastDateTime;
item.SearchKey = skSince; // or skSentSince
 
if (IdIMAP4->UIDSearchMailBox(&item, 0))
{
    int cnt = IdIMAP4->MailBox->SearchResult.Length;
    for (int i = 0; i < cnt; ++i)
        DownloadList->Add( IdIMAP4->MailBox->SearchResult[i] );
}


Not only is it faster, but it uses much less bandwidth, since it is only
1 SEARCH command, instead of individual FETCH UID commands sent N times.

Looks great, I will do that
Thank you very much
Best regards
Asger
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: TIdIMAP4, how to download only new message headers
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 3, 2017 5:50 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

TDateTime lastDateTime = ...; // example of highest date/time downloaded 
last session
 
TIdIMAP4SearchRec item;
item.Date = lastDateTime;
item.SearchKey = skSince; // or skSentSince
 
if (IdIMAP4->UIDSearchMailBox(&item, 0))
{
    int cnt = IdIMAP4->MailBox->SearchResult.Length;
    for (int i = 0; i < cnt; ++i)
        DownloadList->Add( IdIMAP4->MailBox->SearchResult[i] );
}


Not only is it faster, but it uses much less bandwidth, since it is only
1 SEARCH command, instead of individual FETCH UID commands sent N times.

Is it possible to do something similar with TIdPOP3 ?

or do I have to iterate backwards and stop when date become <= my last date

   TDateTime lastDateTime = ...; // example of highest date/time downloaded
 
   int MessageCount = IdPOP3->CheckMessages();
 
   for( int i = MessageCount; i > 0 ; --i )
   {
      TIdMessage* IdMsg = new TIdMessage(0);
 
      IdPOP3->Retrieve( i, IdMsg );
 
      if( IdMsg->Date <= lastDateTime )
      {
         delete IdMsg;
         break;
      }else
         Messages.Add( IdMsg );
 
   }


Thanks in advance
Best regards
Asger
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TIdIMAP4, how to download only new message headers
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 3, 2017 2:44 PM   in response to: Asger Joergensen in response to: Asger Joergensen
Asger wrote:

Is it possible to do something similar with TIdPOP3 ?

No. The POP3 protocol is very basic, it has no advanced features, like searching.
It simply reports whatever is currently in the mailbox, and you are expected
to delete whatever you don't need to keep. IMAP was specifically designed
for mailbox management, so it has a fairly rich API.

or do I have to iterate backwards and stop when date
become <= my last date

Basically, yes. Otherwise, don't use POP3.

--
Remy Lebeau (TeamB)
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: TIdIMAP4, how to download only new message headers
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 3, 2017 2:49 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

or do I have to iterate backwards and stop when date
become <= my last date

Basically, yes. Otherwise, don't use POP3.

Thanks Remy.

I don't want to use POP3, but I believe that I do need to
support it, just in case a customer only use that.

Best regards
Asger
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02