Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: RS2007/Indy10.6.2 (5359) - TIdFTP - Connected(), OnStatus & OnDisconnected


This question is answered.


Permlink Replies: 9 - Last Post: May 13, 2016 1:06 PM Last Post By: Remy Lebeau (Te...
Ted Lyngmo

Posts: 117
Registered: 10/3/06
RS2007/Indy10.6.2 (5359) - TIdFTP - Connected(), OnStatus & OnDisconnected  
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 11, 2016 9:56 PM
Hi!

I recently upgraded Indy from 10.6.1.5186 to 10.6.2.5359 and wonder a bit about TIdFTP Connected(), OnStatus and OnDisconnected.

When the server disconnects because the connection has been idle too long, I don't seem to get any indication of that until I call Connected() (which I therefor do regularly), which will trigger an OnStatus event with AStatus == hsDisconnected.
In 10.6.1, the Connected() method then returned false.
In 10.6.2, it returns true.

In 10.6.1, I dealt with the missing (I'm guessing that it should be there?) OnDisconnected event by calling the OnDisconnected event handler from the OnStatus handler.
In 10.6.2, the above approach will cause my program to indicate that the connection has been lost but it'll continue to call Connected() regularly since it's still returning true. It'll continue to return true until I actually try to do some work (like Put) which will fail with a Timeout - and then Connected() will return false. If I try Connect() while Connected()==true it raises an exception, "Already connected", which it's not.

I've tried to get around this by calling Disconnect() in the OnStatus handler, but that doesn't seem to have any effect. I also tried Socket->Close(), but Connected() still returns true afterwards.

How the server disconnects seems to make a difference. If I restart my FTP server (vsftpd 3.0.2-14.fc23) before the idle timeout, then Connected() will actually return false.

I created a simple test program with a TIdFTP, a memo, three buttons (Test, Connect and Disconnect) and three event handlers (OnConnected, OnDisconnected and OnStatus) to verify it.

//---------------------------------------------------------------------------
void __fastcall TForm1::btnTestClick(TObject *Sender)
{
	memo->Lines->Add("Connected() ...");
	if( ftp->Connected() ) {
		memo->Lines->Add("Connected() true");
	} else {
		memo->Lines->Add("Connected() false");
	}
}
//---------------------------------------------------------------------------
void __fastcall TForm1::btnConnectClick(TObject *Sender)
{
	memo->Lines->Add("Connect() ...");
	try {
		ftp->Connect();
		memo->Lines->Add("Connect() OK");
	} catch( const Exception &ex ) {
		memo->Lines->Add( AnsiString("Connect() ")+ex.Message );
	}
}
//---------------------------------------------------------------------------
void __fastcall TForm1::btnDisconnectClick(TObject *Sender)
{
	try {
		memo->Lines->Add("Disconnect() ...");
		ftp->Disconnect();
		memo->Lines->Add("Disconnect() OK");
	} catch( const Exception &ex ) {
		memo->Lines->Add( AnsiString("Disconnect() ")+ex.Message );
	}
}
//---------------------------------------------------------------------------
void __fastcall TForm1::ftpConnected(TObject *Sender)
{
	memo->Lines->Add( "-OnConnected" );
}
//---------------------------------------------------------------------------
void __fastcall TForm1::ftpDisconnected(TObject *Sender)
{
	memo->Lines->Add( "-OnDisconnected" );
}
//---------------------------------------------------------------------------
void __fastcall TForm1::ftpStatus(TObject *ASender, const TIdStatus AStatus,
	  const AnsiString AStatusText)
{
	memo->Lines->Add( AnsiString("-OnStatus: ") + AStatusText );
}
//---------------------------------------------------------------------------


And some sample output:

Connected() ...
Connected() false
Connect() ...
-OnStatus: Resolving hostname xxxx.xx.
-OnStatus: Connecting to xx.xx.xx.xx.
-OnStatus: Connected.
-OnConnected
-OnStatus: Connection established
Connect() OK
Connected() ...
Connected() true
Connected() ...       // Clicked after server idle timeout
-OnStatus: Disconnected.
Connected() true      // odd?
Connect() ...
Connect() Already connected.
Disconnect() ...
Disconnect() OK
Connected() ...
Connected() true      // also odd


The only time I can manage to get the OnDisconnected event to fire is if I click the Disconnect button while being connected. After that, Connected() will return false like it should.

Connected() ...
Connected() true
Disconnect() ...
-OnStatus: Disconnecting.
-OnDisconnected
-OnStatus: Disconnected.
Disconnect() OK
Connected() ...
Connected() false


I find this behaviour strange, but I may have misunderstood how it's supposed to work. Any ideas?

Best regards,
Ted Lyngmo
Ted Lyngmo

Posts: 117
Registered: 10/3/06
Re: RS2007/Indy10.6.2 (5359) - TIdFTP - Connected(), OnStatus & OnDisconnected  
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 12, 2016 9:21 AM   in response to: Ted Lyngmo in response to: Ted Lyngmo
More tests: After the idle timeout it seems impossible to get TIdFTP to realize that it's not Connected() using any of the natural closing/disconnecting methods. Not even Connect(), Login() or the deprecated Quit() will put it in a useful state (either closed or logged in). It gives the exception "Already connected" and Connected() continues to return true - although it's not connected.

This OnStatus handler seems to do the trick though:

void __fastcall TForm1::ftpStatus(TObject *ASender, const TIdStatus AStatus,
	  const AnsiString AStatusText)
{
	memo->Lines->Add( AnsiString("-OnStatus: ") + AStatusText );
	if( AStatus == hsDisconnected ) {
		try {
			memo->Lines->Add(" closing ...");
			ftp->Put("ftptest.exe");  // force it to realize
			memo->Lines->Add(" closing OK");
		} catch( const Exception &ex ) {
			memo->Lines->Add( AnsiString(" closing ")+ex.Message );
		}
	}
}


It produces this:

Connected() ...
-OnStatus: Disconnected.
 closing ...
-OnStatus: Starting FTP transfer
-OnStatus: Transfer complete
 closing Timeout.
Connected() false


And now the TIdFTP state machine is a known state again. I could solve it like above but that'd require me to create a state machine on top of TIdFTP to track the events that follows the Put() and to hide them from the user. I hope there's a better way.

Br,
Ted
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: RS2007/Indy10.6.2 (5359) - TIdFTP - Connected(), OnStatus &OnDisconnected
Helpful
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 12, 2016 11:50 AM   in response to: Ted Lyngmo in response to: Ted Lyngmo
Ted wrote:

I recently upgraded Indy from 10.6.1.5186 to 10.6.2.5359 and wonder a
bit about TIdFTP Connected(), OnStatus and OnDisconnected.

The behavior of those events has not changed.

When the server disconnects because the connection has been idle too
long, I don't seem to get any indication of that until I call
Connected() (which I therefor do regularly), which will trigger an
OnStatus event with AStatus == hsDisconnected.

That is normal behavior. For the most part, Indy is not event-driven. Socket
status changes are not detected until socket I/O operations are performed.
Connected() performs a read operation, and will close the socket if a disconnect
is detected, thus trigging the OnStatus event. If you don't do that, the
next time you try to perform a normal I/O operation, like sending a new FTP
command, an exception will be raised since the connection was previously
closed by the server. This is normal behavior and is by design.

In 10.6.1, the Connected() method then returned false.

In 10.6.2, it returns true.

Connected() returns true, even if the socket is closed, if there is unread
data in the IOHandler's InputBuffer, and always has.

In 10.6.1, I dealt with the missing (I'm guessing that it should be
there?) OnDisconnected event by calling the OnDisconnected event
handler from the OnStatus handler.

The OnDisconnected event is fired by the Disconnect() method. Connected()
does not call Disconnect() when closing the socket.

In 10.6.2, the above approach will cause my program to indicate that
the connection has been lost but it'll continue to call Connected()
regularly since it's still returning true.

When you detect a disconnect, call Disconnect(), and also Clear() the InputBuffer
if an IOHandler is still assigned. Then Connected() will not return true
anymore.

I've tried to get around this by calling Disconnect() in the OnStatus
handler, but that doesn't seem to have any effect.

Disconnect() ensures the socket is closed, but it doesn't clear the InputBuffer.
You have to do that manually, either through normal reading operations,
or by clearing the InputBuffer directly. The InputBuffer is designed to
continue satisfying read operations, even after the socket is closed, as
long as there is pending data available to read without going back to the
socket for more data. Sometimes data transmissions are terminated by a disconnect
and socket errors get reported by the OS before all of the data has actually
been read from the socket. Indy tries to avoid that by reading as much data
from the socket as possible beforehand and then allow you to keep performing
reads after disconnect until the data has been exhausted.

--
Remy Lebeau (TeamB)
Ted Lyngmo

Posts: 117
Registered: 10/3/06
Re: RS2007/Indy10.6.2 (5359) - TIdFTP - Connected(), OnStatus &OnDisconnected  
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 12, 2016 12:51 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi and thanks Remy!

Remy Lebeau (TeamB) wrote:
Ted wrote:
I recently upgraded Indy from 10.6.1.5186 to 10.6.2.5359 and wonder a
bit about TIdFTP Connected(), OnStatus and OnDisconnected.

The behavior of those events has not changed.
[...]
In 10.6.1, the Connected() method then returned false.
In 10.6.2, it returns true.

Connected() returns true, even if the socket is closed, if there is unread
data in the IOHandler's InputBuffer, and always has.

Okay, it may be my FTP server that has changed its behavior, leaving data in the InputBuffer nowadays. Come to think of it, the user who reported this problem uses my FTP server too.... and I think it's also been upgraded recently.

Disconnect() ensures the socket is closed, but it doesn't clear the InputBuffer.
You have to do that manually, either through normal reading operations,
or by clearing the InputBuffer directly. The InputBuffer is designed to
continue satisfying read operations, even after the socket is closed, as
long as there is pending data available to read without going back to the
socket for more data. Sometimes data transmissions are terminated by a disconnect
and socket errors get reported by the OS before all of the data has actually
been read from the socket. Indy tries to avoid that by reading as much data
from the socket as possible beforehand and then allow you to keep performing
reads after disconnect until the data has been exhausted.

Ahh... So, I could do something like this?

void __fastcall OnStatus(System::TObject* ASender, const TIdStatus AStatus, const AnsiString AStatusText) {
  TIdFTP *ftp = static_cast<TIdFTP*>(ASender);
  if( AStatus == hsDisconnected ) {
    if( ftp->IOHandler ) ftp->IOHandler->InputBuffer->Clear();
    ftp->Disconnect();
  }
}


Or should I Disconnect() first and Clear() after?

Br,
Ted
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: RS2007/Indy10.6.2 (5359) - TIdFTP - Connected(), OnStatus&OnDisconnected  
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 12, 2016 2:02 PM   in response to: Ted Lyngmo in response to: Ted Lyngmo
Ted wrote:

Okay, it may be my FTP server that has changed its behavior, leaving
data in the InputBuffer nowadays.

I was referring to the client-side InputBuffer inside of TIdFTP. And if
your TIdFTP is sitting idle, there shouldn't be any data in the InputBuffer
since there are no commands in flight. That being said, if you want to keep
the connection alive while idle for long periods of time, you should send
a new command periodically, such as Noop(), if you are not already doing so.

Ahh... So, I could do something like this?
<snip>
Or should I Disconnect() first and Clear() after?

You can do it either way, the order is not important.

--
Remy Lebeau (TeamB)
Ted Lyngmo

Posts: 117
Registered: 10/3/06
Re: RS2007/Indy10.6.2 (5359) - TIdFTP - Connected(), OnStatus&OnDisconnected  
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 12, 2016 10:50 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:
Ted wrote:

Okay, it may be my FTP server that has changed its behavior, leaving
data in the InputBuffer nowadays.

I was referring to the client-side InputBuffer inside of TIdFTP.

Yes, me too. If nothing has changed in the way TIdFTP deals with disconnects I guess the data its has to deal with has changed - and that somehow the data the FTP server sends leaves TIdFTP with some residue in its InputBuffer. Just speculating...

And if your TIdFTP is sitting idle, there shouldn't be any data in the InputBuffer
since there are no commands in flight.

And that's how it's worked previously. After the connection had been idle for 5 minutes the server disconnected and TIdFTP Connected() first triggered OnStatus AStatus==hsDisconnected and then returned false.

Since I put all my Indy10 code in a DLL some time back (to be able to use it with BCB4) it's easy to try out the old 10.6.1 based DLL. It now acts just like the 10.6.2 based one does - so I guess the change must be on the server side?

That being said, if you want to keep the connection alive while idle for long
periods of time, you should send a new command periodically, such as
Noop(), if you are not already doing so.

Ok, no, I don't want to keep it alive. I just want to know the state it's in so that I can take the correct action when the time comes to start transferring files again after a long idle period.

Ahh... So, I could do something like this?
<snip>
Or should I Disconnect() first and Clear() after?

You can do it either way, the order is not important.

Ok.

Br,
Ted
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: RS2007/Indy10.6.2 (5359) - TIdFTP - Connected(),OnStatus&OnDisconnected
Helpful
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 13, 2016 1:06 PM   in response to: Ted Lyngmo in response to: Ted Lyngmo
Ted wrote:

Yes, me too. If nothing has changed in the way TIdFTP deals with
disconnects I guess the data its has to deal with has changed - and
that somehow the data the FTP server sends leaves TIdFTP with some
residue in its InputBuffer. Just speculating...

Or, use a packet sniffer, or attach an IdLog... component to each socket
connection, and look at the actual FTP commands/responses to verify if something
has changed between different versions.

--
Remy Lebeau (TeamB)
Ted Lyngmo

Posts: 117
Registered: 10/3/06
Re: RS2007/Indy10.6.2 (5359) - TIdFTP - Connected(), OnStatus &OnDisconnected  
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 12, 2016 1:24 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:

When you detect a disconnect, call Disconnect(), and also Clear() the InputBuffer
if an IOHandler is still assigned. Then Connected() will not return true
anymore.

Clearing the InputBuffer sure solved the Connected() issue! Great!
Calling Disconnect() does not trigger OnDisconnected though. I've tried doing it both before clearing the buffer and after.

Perhaps it skips that event if the socket is already closed when calling Disconnect()?

void __fastcall OnStatus(System::TObject* ASender, const TIdStatus AStatus, const AnsiString AStatusText) {
  TIdFTP *ftp = static_cast<TIdFTP*>(ASender);
  if( AStatus == hsDisconnected ) {
    if( ftp->IOHandler ) ftp->IOHandler->InputBuffer->Clear();
    ftp->Disconnect(); // No effect
  }
}


Br,
Ted
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: RS2007/Indy10.6.2 (5359) - TIdFTP - Connected(), OnStatus&OnDisconnected
Correct
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 12, 2016 2:03 PM   in response to: Ted Lyngmo in response to: Ted Lyngmo
Ted wrote:

Calling Disconnect() does not trigger OnDisconnected though.

Yes, it does, bu only if the socket is still open.

Perhaps it skips that event if the socket is already closed
when calling Disconnect()?

Yes, it does.

--
Remy Lebeau (TeamB)
Ted Lyngmo

Posts: 117
Registered: 10/3/06
Re: RS2007/Indy10.6.2 (5359) - TIdFTP - Connected(), OnStatus&OnDisconnected  
Click to report abuse...   Click to reply to this thread Reply
  Posted: May 12, 2016 2:34 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:
Ted wrote:

Calling Disconnect() does not trigger OnDisconnected though.

Yes, it does, bu only if the socket is still open.

I see, but what will calling Disconnect() after I've detected a disconnect accomplish in that case?

Perhaps it skips that event if the socket is already closed
when calling Disconnect()?

Yes, it does.

Ok, so now I clear the InputBuffer and call the OnDisconnected event handler manually as I did before and it seems to work just fine again.

Many thanks!

Br,
Ted
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02