Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: Does timezones effect TIdMessage->Date ?



Permlink Replies: 13 - Last Post: Jun 3, 2017 6:14 AM Last Post By: Asger Joergensen
Asger Joergensen

Posts: 370
Registered: 11/18/08
Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 14, 2017 2:27 PM
Hi

I use:

   TIdIMAP4SearchRec item;
   item.Date = FLastDateTime;
   item.SearchKey = skSince;
 
   IdIMAP4->UIDSearchMailBox( &item, 0 )


to get the new mails in the IMAP mailbox and I use

   if( msg->Date > FLastDateTime )
      FLastDateTime = msg->Date;


To make sure I store the date of the last mail to be used the next time.

Now to my question:
Is the above a secure way of getting EVERY message that, over time, is
in the mailbox ?

I'm thinking can this be fooled by time-zones ?

Or can it be fooled by moving a mail from one mailbox to another using
another mail client ?

In POP3 I maintain a list of mail ID's, to make sure all ID's are
downloaded, but I can't seem to find a function in IMAP for downloading
all ID's, I can only find GetUID to download ID's one by one.

What is the secure way to get all messages on IMAP ?

Thanks in advance
Best regards
Asger
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 15, 2017 1:33 PM   in response to: Asger Joergensen in response to: Asger Joergensen
Asger wrote:

I use:

TIdIMAP4SearchRec item;
item.Date = FLastDateTime;
item.SearchKey = skSince;
IdIMAP4->UIDSearchMailBox( &item, 0 )


to get the new mails in the IMAP mailbox

Note that searching for emails by Date will search the internal date of
each email, which is independant of the date specified in the email headers.
The IMAP spec does not explain what the format of the internal date is,
whether it is in UTC or local to the server's timezone. It is for the server's
internal use only. The spec only says that:

The internal date and time of the message on the server. This is not the
date and time in the [RFC-2822] header, but rather a date and time which
reflects when the message was received. In the case of messages delivered
via [SMTP], this SHOULD be the date and time of final delivery of the message
as defined by [SMTP]. In the case of messages delivered by the IMAP4rev1
COPY command, this SHOULD be the internal date and time of the source message.
In the case of messages delivered by the IMAP4rev1 APPEND command, this
SHOULD be the date and time as specified in the APPEND command description.
All other cases are implementation defined.

However, when a FETCH command is used to retreive (portions of) an email,
INTERNALDATE is one of the metadata items that can be retrieved. But, none
of TIdIMAP4's native methods retreive that particular item, so you would
have to request it manually.

and I use

if( msg->Date > FLastDateTime )
FLastDateTime = msg->Date;


To make sure I store the date of the last mail to be used the next
time.

Note that the TIdMessage::Date property is always expressed in local time,
not in UTC.

Is the above a secure way of getting EVERY message that, over time,
is in the mailbox ?
I'm thinking can this be fooled by time-zones ?

I don't know, since I don't know what format the date value of a SEARCH command
is actually expecting. Since you are downloading emails anyway, you should
probably request their INTERNALDATE values and use the highest value for
the next search.

Or can it be fooled by moving a mail from one mailbox to another using
another mail client ?

Moving an email from one mailbox to another is supposed to preserve its INTERNALDATE
value, per the quote above from the IMAP spec.

In POP3 I maintain a list of mail ID's, to make sure all ID's are
downloaded

IDs in POP3 are not persistent between sessions. The IDs are nothing more
than indexes into the mailbox. Every new POP3 session starts with ID 1,
so once you end a session and start a new one, there is no guarantee that
the previous IDs refer to the same emails in the new session. You MUST download
the actual email headers and compare them to the emails you have already
downloaded. POP3 is not designed for mailbox management. That is why IMAP
exists, and it solves the identification problem by using UIDs instead of
indexes.

but I can't seem to find a function in IMAP for downloading all ID's,
I can only find GetUID to download ID's one by one.

TIdIMAP4 does not currently have such a method.

--
Remy Lebeau (TeamB)
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 15, 2017 2:10 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

In POP3 I maintain a list of mail ID's, to make sure all ID's are
downloaded

IDs in POP3 are not persistent between sessions. The IDs are nothing more
than indexes into the mailbox. Every new POP3 session starts with ID 1,
so once you end a session and start a new one, there is no guarantee that
the previous IDs refer to the same emails in the new session. You MUST download
the actual email headers and compare them to the emails you have already
downloaded. POP3 is not designed for mailbox management. That is why IMAP
exists, and it solves the identification problem by using UIDs instead of
indexes.

I don't think that is the truth, because when I use IdPOP3->UIDL
I get a list of ids that are much more then just a number, like these:
1 MD50000012441:MSG:6068:29885722:921102544
2 MD50000012442:MSG:3146:29885724:646725136
or these:
1 UID997-1145239868
2 UID1281-1145239868
3 UID1289-1145239868
4 UID1303-1145239868

and this site https://tools.ietf.org/html/rfc1939 say:
[quote]
The unique-id of a message is an arbitrary server-determined string,
consisting of one to 70 characters in the range 0x21 to 0x7E, which
uniquely identifies a message within a maildrop and which persists
across sessions.
This persistence is required even if a session ends without entering
the UPDATE state. The server should never reuse an unique-id in a
given maildrop, for as long as the entity using the unique-id exists.
[end quote]

And I've seen on many forums that they are used to manage the downloads
of mails from a pop3 mailbox.

Thanks for helping.
Best regards
Asger
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 15, 2017 3:30 PM   in response to: Asger Joergensen in response to: Asger Joergensen
Asger wrote:

<snip>

It also says:

While it is generally preferable for server implementations to store arbitrarily
assigned unique-ids in the maildrop, this specification is intended to permit
unique-ids to be calculated as a hash of the message. **Clients should be
able to handle a situation where two identical copies of a message in a maildrop
have the same unique-id.**

So you can't rely on UIDL alone to unique identify every email in the mailbox,
especially if you are going to be deleting the messages. You can use UIDL
to know if a given email has been downloaded (even if multiple copies of
the same email exist), but you still have to process each individual message-number
separately, and link any duplicates together as needed.

--
Remy Lebeau (TeamB)
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 15, 2017 3:47 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

Asger wrote:

<snip>

It also says:

While it is generally preferable for server implementations to store arbitrarily
assigned unique-ids in the maildrop, this specification is intended to permit
unique-ids to be calculated as a hash of the message. **Clients should be
able to handle a situation where two identical copies of a message in a maildrop
have the same unique-id.**

Aaaah, yea ok, but that is not a problem for me, I don't mind duplicates as long
as they are few I only use the attachments and they will be saved to unique names
...(1) ...(2) etc, and the user will just delete the extra. I am much more concerned
about getting ALL mails in the mailbox.

So you can't rely on UIDL alone to unique identify every email in the mailbox,
especially if you are going to be deleting the messages. You can use UIDL
to know if a given email has been downloaded (even if multiple copies of
the same email exist), but you still have to process each individual message-number
separately, and link any duplicates together as needed.

As I read the above:
I could avoid downloading duplicates, if I check the UIDL list for duplicates before
I start downloading, right ?

Thank you very much for helping
Best regards
Asger
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 15, 2017 4:07 PM   in response to: Asger Joergensen in response to: Asger Joergensen
Asger wrote:

As I read the above:
I could avoid downloading duplicates, if I check the UIDL
list for duplicates before I start downloading, right ?

You would have to connect to the server, loop through each message-number
that is present, request each message's UIDL and check if you have already
downloaded it, and if not then download it and mark it with the UIDL (such
as in the filename), and then optionally delete the messages you don't want
to keep in the mailbox anymore.

--
Remy Lebeau (TeamB)
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 15, 2017 4:30 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

Asger wrote:

As I read the above:
I could avoid downloading duplicates, if I check the UIDL
list for duplicates before I start downloading, right ?

You would have to connect to the server, loop through each message-number
that is present, request each message's UIDL and check if you have already
downloaded it, and if not then download it and mark it with the UIDL (such
as in the filename), and then optionally delete the messages you don't want
to keep in the mailbox anymore.

But in POP I can use IdPOP3->UIDL for getting a list of all message UID's
on the server, so it is actually quite easy to maintain ensure download of
all new mails.

UIDList = IdPOP3->UIDL;
Copy UIDList to CopyList

loop through LastUIDList and delete all in UIDList that match

Download all those messages left in UIDList

Assign CopyList To LastUIDList

ready for next time. :-)

Seem to work just fine
Thanks for helping
Best regards
Asger
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 15, 2017 5:14 PM   in response to: Asger Joergensen in response to: Asger Joergensen
Asger wrote:

But in POP I can use IdPOP3->UIDL for getting a list of all message
UID's on the server, so it is actually quite easy to maintain ensure
download of all new mails.

That doesn't change what I said in my last reply. Whether you download each
UIDL individually, or download the entire list in a single batch, you still
have to process all of the UIDLs currently in the mailbox to discover new
ones (and detect deleted ones, if needed).

UIDList = IdPOP3->UIDL;
Copy UIDList to CopyList
loop through LastUIDList and delete all in UIDList that match

Download all those messages left in UIDList

Assign CopyList To LastUIDList

And if there is an error downloading an email, that kind of logic will cause
you to re-download emails you have already downloaded. I would do it this
way instead:

UIDList = IdPOP3->UIDL;
remove duplicates from UIDList that exist in LastUIDList
clear LastUIDList
for each message left in UIDList, download and add to LastUIDList

--
Remy Lebeau (TeamB)
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 15, 2017 3:31 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy
Thank you very much for taking the time to explain.

Remy Lebeau (TeamB) wrote:

However, when a FETCH command is used to retreive (portions of) an email,
INTERNALDATE is one of the metadata items that can be retrieved. But, none
of TIdIMAP4's native methods retreive that particular item, so you would
have to request it manually.

Can you point me to an example of how this is done in code

I don't know, since I don't know what format the date value of a SEARCH command
is actually expecting. Since you are downloading emails anyway, you should
probably request their INTERNALDATE values and use the highest value for
the next search.

How do I request their INTERNALDATE ?

Or can it be fooled by moving a mail from one mailbox to another using
another mail client ?

Moving an email from one mailbox to another is supposed to preserve its
INTERNALDATE value, per the quote above from the IMAP spec.

Then i am screwed, because that means (as I understand the spec) that a mail
can be copied to a mailbox and it can have an internal date, that is way older
then the last time I checked the mailbox and I will therefor never download it.

I wonder how they do it in real mail client.

P.s. my program is a read only mail client, but I must be certain that I get
all mails that arrive to the mailbox, no matter how they arrive.

Thanks again
Best regards
Asger
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 15, 2017 4:03 PM   in response to: Asger Joergensen in response to: Asger Joergensen
Asger wrote:

Can you point me to an example of how this is done in code

How do I request their INTERNALDATE ?

I don't have an example of retreiving an email and its INTERNALDATE at the
same time (look at TIdIMAP4's source code for its Retreive methods and adapt
as needed). However, TIdIMAP4 does have (UID)RetrieveValue() methods for
retreiving a single item from a given email, eg:

String sInternalDate;
if (IdIMAP4->UIDRetrieveValue(UID, "INTERNALDATE", sInternalDate))
{
    // use sInternalDate as needed...
}


Moving an email from one mailbox to another is supposed to preserve
its INTERNALDATE value, per the quote above from the IMAP spec.
Then i am screwed, because that means (as I understand the spec) that
a mail can be copied to a mailbox and it can have an internal date,
that is way older then the last time I checked the mailbox and I will
therefor never download it.

Yup. To account for that, you would need to include the skRecent and/or
skNew flags in your search. When an email is copied from one mailbox to
another, the IMAP "\Recent" flag is supposed to be added to the copy. skRecent
searched for any emails with the "\Recent" flag, whereas skNew searches for
only emails that have the "\Recent" flag but not the "\Seen" flag.

I wonder how they do it in real mail client.

Sniff the IMAP traffic with a packet sniffer, like Wireshark, and look at
the SEARCH and FETCH commands for yourself.

P.s. my program is a read only mail client, but I must be certain that
I get all mails that arrive to the mailbox, no matter how they arrive.

IMAP has an IDLE command that delivers events in real-time as emails arrive,
delete, get moved around, etc, but that requires keeping a persistent connection
to the server to receive such events (and besides, TIdIMAP4 does not implement
IDLE yet anyway - see https://github.com/IndySockets/Indy/issues/147).

As soon as you disconnect from the server, users are free to manipulate the
mailbox without your knowledge, so when you reconnect, you will have to get
back in sync by scanning the mailboxes for new messages, deleted messages,
copied/moved messages, etc. Good luck.

--
Remy Lebeau (TeamB)
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 15, 2017 5:06 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

String sInternalDate;
if (IdIMAP4->UIDRetrieveValue(UID, "INTERNALDATE", sInternalDate))
{
    // use sInternalDate as needed...
}

That looks fine, thanks.

Yup. To account for that, you would need to include the skRecent and/or
skNew flags in your search.

That look like just what I need, but what happens if another mail client read
the mailbox, will those messages not lose their skRecent and skNew flags ?

Or is the skRecent and skNew flags really a timestamp when they arrived in
the mailbox ?

As soon as you disconnect from the server, users are free to manipulate the
mailbox without your knowledge, so when you reconnect, you will have to get
back in sync by scanning the mailboxes for new messages, deleted messages,
copied/moved messages, etc. Good luck.

Fortunately I only need to concern my self, with the newly added messages,
and only those with the right kind of attachments. :-)

Thanks for all your help
Best regards
Asger
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 16, 2017 3:51 PM   in response to: Asger Joergensen in response to: Asger Joergensen
Asger Joergensen wrote:
That look like just what I need, but what happens if another mail client read
the mailbox, will those messages not lose their skRecent and skNew flags ?

skRecent and skNew are Indy constants, not IMAP flags.

As I said, skRecent searches for emails that have the IMAP "\Recent" flag, whereas skNew searches for emails that have the IMAP "\Recent" flag but not also the IMAP "\Seen" flag.

If another client reads the mailbox before you, then yes, the emails could lose the "\Recent" flag, or gain the "\Seen" flag. So then you are back to searching timestamps again.

Or is the skRecent and skNew flags really a timestamp when they arrived in
the mailbox ?

Flags are not timestamps.

--
Remy Lebeau (TeamB)
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 17, 2017 5:20 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

....

Flags are not timestamps.

Thanks for clarifying
Best regards
Asger
Asger Joergensen

Posts: 370
Registered: 11/18/08
Re: Does timezones effect TIdMessage->Date ?
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 3, 2017 6:14 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy

Remy Lebeau (TeamB) wrote:

I'm thinking can this be fooled by time-zones ?

I don't know, since I don't know what format the date value of a SEARCH command
is actually expecting. Since you are downloading emails anyway, you should
probably request their INTERNALDATE values and use the highest value for
the next search.

I think I know now, after some testing and some more reading.

In the RFC it say, under all the flags that relate to search using date e.g.

SINCE <date>
Messages whose internal date (disregarding time and timezone)
is within or later than the specified date.

In other words, it is only the date part that is used and it also means, that
searching the mailbox every 10 min. for new mails, using since and a date, will
return the same mail ID's a lot of times.

It is still possible to be fooled by the timezone though!

I live in +0200 an my mailserver is located in +0000, when I send a mail
to my self at:
03-JUN-2017 01:00:00 +0200
it get this internal date on the mail server:
02-JUN-2017 23:00:11 +0000

Best regards
Asger
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02