Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: MessageId generation won't be accepted by SMTP server


This question is answered.


Permlink Replies: 7 - Last Post: Sep 14, 2016 3:53 PM Last Post By: John May
John May

Posts: 81
Registered: 6/25/10
MessageId generation won't be accepted by SMTP server  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 12, 2016 7:28 AM
I use Indy, version 5344, but if this is fixed in more recent version, I will gladly update to it.

Essentially the problem is - while using TIdMessageBuilderHtml to build a message and the code is essentially this:

boost::scoped_ptr<TIdMessage> IdMsg(new TIdMessage(this));
boost::scoped_ptr<TIdMessageBuilderHtml> IdMsgBldrHtml(new TIdMessageBuilderHtml);
 
IdMsg->From->Name = "Test";
IdMsg->From->Address = "test@test.com";
; more building To/Cc/Bcc list here
IdMsg->Subject = "test";
 
IdMsg->MsgId	     = "<12345@abcdef>";
IdMsg->InReplyTo   = "";
IdMsg->References = "";
 
IdMsgBldrHtml->PlainTextCharSet			= "utf-8";
IdMsgBldrHtml->FillMessage(IdMsg.get());


Some code has been stripped away but that is not the problem.

The problem is blank InReplyTo and References.

When Indy sees InReplyTo being blank it automatically fills it with the same thing which is in MsgId. As this is first-time message this is incorrect.

The result is that when I attempt to send such message on a few server the SMTP server reports "500" error (Syntax error, command unrecognised) after message is entirely submitted.

However, if I fill the InReplyTo with something which is not blank and different from MsgId, then the server accepts the message.

Since InReplyTo should be blank (as in RFC) for new messages and filled only when replying to a message, why is Indy filling it out automatically when I explicitly set it not to do so?

And more importantly, how can I avoid it?
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: MessageId generation won't be accepted by SMTP server  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 12, 2016 2:41 PM   in response to: John May in response to: John May
John wrote:

The problem is blank InReplyTo and References.

When Indy sees InReplyTo being blank it automatically fills it with
the same thing which is in MsgId. As this is first-time message this
is incorrect.

Indy was never intended to allow clients to generate their own message IDs,
it expects servers to generate the IDs during sending. And in that regard,
setting the TIdMessage.MsgId property for an outbound email is useless because
Indy will not send a Message-ID header. The TIdMessage.MsgId value is only
preserved when saving TIdMessage to a file.

Since InReplyTo should be blank (as in RFC) for new messages and
filled only when replying to a message, why is Indy filling it out
automatically when I explicitly set it not to do so?

So people can receive a message and then reply to it without having to fill
in the reply headers manually.

And more importantly, how can I avoid it?

You can set the TIdMessage.InReplyTo property to a space character. That
will bypass creation of the In-Reply-To header.

--
Remy Lebeau (TeamB)
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: MessageId generation won't be accepted by SMTP server  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 12, 2016 4:16 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy wrote:

Indy was never intended to allow clients to generate their own message
IDs, it expects servers to generate the IDs during sending. And in
that regard, setting the TIdMessage.MsgId property for an outbound
email is useless because Indy will not send a Message-ID header. The
TIdMessage.MsgId value is only preserved when saving TIdMessage to a
file.

If you must send your own Message-ID header value, you need to use the TIdMessage.ExtraHeaders
property for that purpose.

--
Remy Lebeau (TeamB)
John May

Posts: 81
Registered: 6/25/10
Re: MessageId generation won't be accepted by SMTP server  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 12, 2016 4:40 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:
Indy was never intended to allow clients to generate their own message IDs,
it expects servers to generate the IDs during sending. And in that regard,
setting the TIdMessage.MsgId property for an outbound email is useless because
Indy will not send a Message-ID header. The TIdMessage.MsgId value is only
preserved when saving TIdMessage to a file.

Some servers will complain if you send a message without Message-Id and report error. No servers complain if you send it - they can strip it away if they don't like it because all email clients do send one.
Anyway, I override that with:

IdMsg->ExtraHeaders->Values["Message-Id"] = IdMsg->MsgId;	// By design Indy does not stream MsgId and BCC, only saves it to file
SMTPClient->Send(IdMsg.get());							// Stream message


So my Message-Id gets sent properly. And it is not useless because servers complain.

And more importantly, how can I avoid it?
You can set the TIdMessage.InReplyTo property to a space character. That
will bypass creation of the In-Reply-To header.

Sorry, that doesn't work. Maybe that is something what it is supposed to do but that is not what it does - I get that same single space in "in-reply-to" header as a result after saving the message:

In-Reply-To: < >

(by the way, this forum has a bug also - does not escape less than and greater than characters - had to use HTML escape codes to display the above properly!)

Which is exactly what IdMessage.pas code does:

  {Generate In-Reply-To if at all possible to pacify SA.  Do this after FExtraHeaders
   added in case there is a message-ID present as an extra header.}
  if InReplyTo = '' then begin
    if FLastGeneratedHeaders.Values['Message-ID'] <> '' then begin  {do not localize}
      FLastGeneratedHeaders.Values['In-Reply-To'] := FLastGeneratedHeaders.Values['Message-ID'];  {do not localize}
    end else begin
     {CC: The following was originally present, but it so wrong that it has to go!}
     //Values['In-Reply-To'] := Subject;   {do not localize}
    end;
  end else begin
    // This line here adds single space as actual InReplyTo and does not avoid generating it.
    FLastGeneratedHeaders.Values['In-Reply-To'] := InReplyTo; {do not localize}
  end;


Can something like this can be checked in to really accomodate single-space character as blank (non-existent) in-reply-to?

  end else begin
  // check if single space - then don't generate header, if anything else, generate it:
  if InReplyTo <> ' ' then begin
    FLastGeneratedHeaders.Values['In-Reply-To'] := InReplyTo; {do not localize}
  end;
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: MessageId generation won't be accepted by SMTP server [Edit]
Helpful
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 12, 2016 5:37 PM   in response to: John May in response to: John May
John wrote:

Sorry, that doesn't work.

See further below.

I get that same single space in "in-reply-to" header as a result
after saving the message:

In-Reply-To: < >

(by the way, this forum has a bug also - does not escape less than and
greater than characters - had to use HTML escape codes to display the
above properly!)

You can use { code }{ code } blocks to wrap such text. You don't need to use HTML
entities.

For that to work this code would have to look maybe like this:

The problem was not in the code you quoted. I was expecting a non-blank
string passed to the TIdHeaderList Values[] property to trim leading/trailing
whitespace, but what I didn't take into account is that assigning a non-blank
value to the TIdMessage.InReplyTo property will wrap it in brackets regardless
of whitespace:

procedure TIdMessage.SetInReplyTo(const AValue: String);
begin
  FInReplyTo := EnsureMsgIDBrackets(AValue);
end;


And thus InReplyTo is not blank in GenerateHeader() and kicks in the logic
to generate the "In-Reply-To" header.

Sorry, old problem (I forgot about it):

https://forums.embarcadero.com/thread.jspa?messageID=792671

http://stackoverflow.com/questions/24679807/

I have changed the behavior so that the "In-Reply-To" header is no longer
auto-generated if the TIdMessage.InReplyTo property is blank.

http://indyproject.org/sockets/blogs/changelog/20160912.EN.aspx

--
Remy Lebeau (TeamB)
John May

Posts: 81
Registered: 6/25/10
Re: MessageId generation won't be accepted by SMTP server  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 14, 2016 6:45 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
You Sir, deserve a million thank you! - https://www.youtube.com/watch?v=fjkptSX4clE

However, I now have another problem. I looked at your blog and found the new helper methods:
http://indyproject.org/sockets/blogs/changelog/20150730.EN.aspx

But I have difficulties using them in C++ Builder 2010 (probably because I don't know how to use them).

Here is my old (now legacy) code for saving and loading IdMessage to stream:

void IndyMsgLoadFromStream(TIdMessage *IdMsg, TMemoryStream *MemStream)
{
IdMsg->Clear();
boost::scoped_ptr<TIdMessageClient>		 mc(new TIdMessageClient(NULL));
boost::scoped_ptr<TIdIOHandlerStreamMsg> io(new TIdIOHandlerStreamMsg(NULL, MemStream, NULL));
io->EscapeLines = true;															// this is the key step
io->FreeStreams = false;
mc->IOHandler	= io.get();
io->Open();
mc->ProcessMessage(IdMsg, false);
}
 
void IndyMsgSaveToStream(TIdMessage *IdMsg,	TMemoryStream *MemStream)
{
if (IdMsg->BccList->Count > 0) IdMsg->ExtraHeaders->Values["Bcc"]		 = EncodeAddress(IdMsg->BccList, 'B', "UTF-8");	// 'B' = base64, 'Q' = quoted-printable
							   IdMsg->ExtraHeaders->Values["Message-Id"] = IdMsg->MsgId;
boost::scoped_ptr<TIdMessageClient>		 mc(new TIdMessageClient(NULL));
boost::scoped_ptr<TIdIOHandlerStreamMsg> io(new TIdIOHandlerStreamMsg(NULL, NULL, MemStream));
io->UnescapeLines = true;														// this is the key step
io->FreeStreams	  = false;
mc->IOHandler	  = io.get();
mc->SendMsg(IdMsg, false);
}


New code is supposed to be:

#include "IdMessageHelper.hpp"
 
void IndyMsgLoadFromStream(TIdMessage *IdMsg, TMemoryStream *MemStream)
{
IdMsg->Clear();
Idmessagehelper::TIdMessageHelper_LoadFromStream(IdMsg, MemStream, false, false);
}
 
void IndyMsgSaveToStream(TIdMessage *IdMsg,	TMemoryStream *MemStream)
{
if (IdMsg->BccList->Count > 0) IdMsg->ExtraHeaders->Values["Bcc"]		 = EncodeAddress(IdMsg->BccList, 'B', "UTF-8");	// 'B' = base64, 'Q' = quoted-printable
Idmessagehelper::TIdMessageHelper_SaveToStream(IdMsg, MemStream, false, false);
}


Two things here:

1) I still need to use code to output Bcc (Message-ID gets outputted)
2) it issues warning: Accessing deprecated entity '_fastcall TIdMessageHelper_LoadFromStream(TIdMessage *,TStream *,const bool,const bool)'

How can I avoid the warning (or use it properly)? As you can see all I want is to save/load the message with all data in it (BCC, Message-ID and without double-dot escaping).
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: MessageId generation won't be accepted by SMTP server [Edit]
Correct
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 14, 2016 11:57 AM   in response to: John May in response to: John May
John wrote:

But I have difficulties using them in C++ Builder 2010 (probably
because I don't know how to use them).

You are using them correctly.

1) I still need to use code to output Bcc (Message-ID gets outputted)

Yes. TIdMessage still only outputs BCC when saving to a file, not to a stream.

2) it issues warning: Accessing deprecated entity
'_fastcall TIdMessageHelper_LoadFromStream(TIdMessage *,TStream *,const
bool,const bool)'

Remember that Indy is written in Delphi, not C++. On the Delphi side, the
flat functions are marked as deprecated in favor of the class helper. That
gets translated into corresponding #pragma statements when the C++ .hpp file
is generated.

How can I avoid the warning

You would have to either:

1. modify IdMessageHelper.pas to omit the deprecated statements, or at least
wrap them in {$IFDEF BCB} blocks.

2. in your C++ code, use #pragma statements to disable the warnings, eg:

void IndyMsgLoadFromStream(TIdMessage *IdMsg, TMemoryStream *MemStream)
{
    IdMsg->Clear();
    #pragma warn -obs
    TIdMessageHelper_LoadFromStream(IdMsg, MemStream, false, false);
    #pragma warn .obs
}
 
void IndyMsgSaveToStream(TIdMessage *IdMsg, TMemoryStream *MemStream)
{
    if (IdMsg->BccList->Count > 0) IdMsg->ExtraHeaders->Values["Bcc"] = EncodeAddress(IdMsg->BccList, 
'B', "UTF-8");    // 'B' = base64, 'Q' = quoted-printable
    #pragma warn -obs
    TIdMessageHelper_SaveToStream(IdMsg, MemStream, false, false);
    #pragma warn .obs
}


Or:

void IndyMsgLoadFromStream(TIdMessage *IdMsg, TMemoryStream *MemStream)
{
    IdMsg->Clear();
    #pragma warn -8053
    TIdMessageHelper_LoadFromStream(IdMsg, MemStream, false, false);
    #pragma warn .8053
}
 
void IndyMsgSaveToStream(TIdMessage *IdMsg, TMemoryStream *MemStream)
{
    if (IdMsg->BccList->Count > 0) IdMsg->ExtraHeaders->Values["Bcc"] = EncodeAddress(IdMsg->BccList, 
'B', "UTF-8");    // 'B' = base64, 'Q' = quoted-printable
    #pragma warn -8053
    TIdMessageHelper_SaveToStream(IdMsg, MemStream, false, false);
    #pragma warn .8053
}


or use it properly

You already are. Just disable/bypass the warning.

--
Remy Lebeau (TeamB)
John May

Posts: 81
Registered: 6/25/10
Re: MessageId generation won't be accepted by SMTP server [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 14, 2016 3:51 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
small correction for anyone else reading this (wrong warning was suppressed):

#pragma warn -dpr
Idmessagehelper::TIdMessageHelper_LoadFromStream(IdMsg, MemStream, false, false);
#pragma warn .dpr


or...

#pragma warn -8111
Idmessagehelper::TIdMessageHelper_LoadFromStream(IdMsg, MemStream, false, false);
#pragma warn .8111


Remy Lebeau (TeamB) wrote:
void IndyMsgLoadFromStream(TIdMessage *IdMsg, TMemoryStream *MemStream)
{
    IdMsg->Clear();
    #pragma warn -8053
    TIdMessageHelper_LoadFromStream(IdMsg, MemStream, false, false);
    #pragma warn .8053
}
 
void IndyMsgSaveToStream(TIdMessage *IdMsg, TMemoryStream *MemStream)
{
    if (IdMsg->BccList->Count > 0) IdMsg->ExtraHeaders->Values["Bcc"] = EncodeAddress(IdMsg->BccList, 
'B', "UTF-8");    // 'B' = base64, 'Q' = quoted-printable
    #pragma warn -8053
    TIdMessageHelper_SaveToStream(IdMsg, MemStream, false, false);
    #pragma warn .8053
}
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02