Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: Indy TIdHTTP worker events


This question is answered.


Permlink Replies: 8 - Last Post: Sep 7, 2015 6:56 AM Last Post By: rene v.d. berge
rene v.d. berge

Posts: 9
Registered: 2/27/01
Indy TIdHTTP worker events  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jul 28, 2015 5:14 AM
I'm using the Indy tIdHTTP component on a form and have defined the OnWorkBegin, OnWork, OnWorkEnd events
I've used this code in C++Builder XE2 and Indy 10.5.8.0. and it worked just fine but in XE8, Indy 10.6.2.5263. all the 'worker' events are fired when IdHTTP1->Get(URL, pFStream)
is completed. All Events are "piled up" and fired sequentially after Get() is completed.

{
....
FileName = fl.GetFolderPath(&FOLDERID_InternetCache) + "myfile.exe";
pFStream = new TFileStream(FileName, fmCreate);
IdHTTP1->Get(URL, pFStream);
delete pFStream;
...
}
 
void __fastcall TFormSettings::IdHTTP1WorkBegin(TObject *ASender, TWorkMode AWorkMode,  __int64 AWorkCountMax)
{
	ProgressBar1->Max = AWorkCountMax;
	ProgressBar1->Position = 0;
	ProgressBar1->Visible = true;
	LabelDownload->Visible = true;
	Update();
}
 
void __fastcall TFormSettings::IdHTTP1Work(TObject *ASender, TWorkMode AWorkMode, __int64 AWorkCount)
{
	ProgressBar1->Position = AWorkCount;
	Update();
}
 
void __fastcall TFormSettings::IdHTTP1WorkEnd(TObject *ASender, TWorkMode AWorkMode)
{
	ProgressBar1->Visible = false;
	LabelDownload->Visible = false;
}


What can be the cause of this behaviour?
I'm running just 1 thread.

Thanks in advance,
René

Edited by: rene v.d. berge on Jul 28, 2015 7:22 AM

Edited by: rene v.d. berge on Jul 28, 2015 7:23 AM
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Indy TIdHTTP worker events [Edit]
Helpful
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jul 28, 2015 11:41 AM   in response to: rene v.d. berge in response to: rene v.d. berge
rene wrote:

I've used this code in C++Builder XE2 and Indy 10.5.8.0. and it
worked just fine but in XE8, Indy 10.6.2.5263. all the 'worker'
events are fired when IdHTTP1->Get(URL, pFStream) is completed.

That is physically impossible, in any version of Indy, as the OnWork... events
are triggered inside of Get() itself while it is reading/writing blocks of
data. It is not possible for the events to trigger after Get() has exited.
What is actually happening is that your UI updates are getting backed up
instead, not that the OnWork... events are getting backed up.

All Events are "piled up" and fired sequentially after Get() is completed.

The events are triggered in the same thread context that is calling TIdHTTP.Get().
If you are calling Get() in the main UI thread, it is going to block the
main message loop from processing new messages, unless you have a TIdAntiFreeze
component on the Form. What you are seeing happen is your UI updates are
getting stuck in the message queue and not being processed until after Get()
has exited and returned control to the main message loop. You are calling
the Form's Update() method, but all that is going to do is process pending
paint messages, not all pending messages. A themed progress bar, for instance,
runs a timer internally to update its drawing state smoothly, and Update()
will not process/dispatch those timer messages, but TIdAntiFreeze would.

pFStream = new TFileStream(FileName, fmCreate);
IdHTTP1->Get(URL, pFStream);
delete pFStream;

FYI, if Get() fails, an exception is raised, which you are not catching,
so you would leak the TFileStream object and its associated file handle.
You need to add an exception handler:

pFStream = new TFileStream(FileName, fmCreate);
try
{
    try
    {
        IdHTTP1->Get(URL, pFStream);
    }
    __finally
    {
        delete pFStream;
    }
}
catch(const Exception&)
{
    DeleteFile(FileName);
    //...
}


void __fastcall TFormSettings::IdHTTP1WorkBegin(TObject *ASender,
TWorkMode AWorkMode, __int64 AWorkCountMax)
{
ProgressBar1->Max = AWorkCountMax;

Keep in mind that:

1. AWorkCountMax will be 0 if TIdHTTP does not know the size of the file
being sent, either because the data is being sent using the "chunked" transfer
encoding, or because the server ends the file by closing the socket.

2. You should check the AWorkMode parameter and only do your logic if the
mode is wmRead and not wmWrite.

void __fastcall TFormSettings::IdHTTP1Work(TObject *ASender, TWorkMode
AWorkMode, __int64 AWorkCount)
{
ProgressBar1->Position = AWorkCount;

If the Max is set to 0, setting the Position is useless. Also, same thing
about checking the AWorkMode parameter.

What can be the cause of this behaviour?
I'm running just 1 thread.

Do you mean you are running the TIdHTTP in its own worker thread? Or in
the main UI thread? It makes a big difference.

If you are running TIdHTTP in the main thread, you need to either use TIdAntiFreeze
or move TIdHTTP to a worker thread to unblock the main thread.

If you are running TIdHTTP in a worker thread, the OnWork... handlers you
have shown are not thread-safe, you need to sync with the main thread to
perform UI updates safely.

--
Remy Lebeau (TeamB)
rene v.d. berge

Posts: 9
Registered: 2/27/01
Re: Indy TIdHTTP worker events [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jul 29, 2015 1:47 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Thanks Remy for your prompt response, much appreciated!

Do you mean you are running the TIdHTTP in its own worker thread? Or in
the main UI thread? It makes a big difference.

If you are running TIdHTTP in the main thread, you need to either use TIdAntiFreeze
or move TIdHTTP to a worker thread to unblock the main thread.

If you are running TIdHTTP in a worker thread, the OnWork... handlers you
have shown are not thread-safe, you need to sync with the main thread to
perform UI updates safely.

Yes I'm running just the main UI thread. After I've incorporated the TIdAntiFreeze component
Window messages are beeing serviced again by my application, but a 'HTTP worker' event still isn't fired while Get() is working...

René
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Indy TIdHTTP worker events [Edit]
Helpful
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jul 29, 2015 10:19 AM   in response to: rene v.d. berge in response to: rene v.d. berge
rene wrote:

Window messages are beeing serviced again by my application,
but a 'HTTP worker' event still isn't fired while Get() is working...

And as I said, that is physically impossible. The OnWork... events are
not message-based, they are direct calls made inside the I/O functions.
Read a block of data, call OnWork, read a block of data, call OnWork, and
so on. You can verify that for yourself using the debugger, or via a real-time
logging solution, such as OutputDebugString(), CodeSite, etc.

--
Remy Lebeau (TeamB)
rene v.d. berge

Posts: 9
Registered: 2/27/01
Re: Indy TIdHTTP worker events [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 3, 2015 7:52 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
I wasn't able to pick this up earlier and 'cause this wasn't a showstopper I
released my application. Today I had some time to test a bit more...
As far as I can tell the whole event fire/handling works on platforms/systems I've been able to test...
but it still fails on my develop PC.
This is what's happening:

Stub_select() in IdWinsock2.pas:
Result := select(nfds, readfds, writefds, exceptfds, timeout);
Returns '1' (socket available for guaranteed non-blocking read) AFTER the socket is closed (FIN received and acknowledged)
so ALL data is received including the HTTP response and then Select returns. But this happens only on my system.
For now I have now idea what can cause this. On other (equivalent systems this does not occur....)

Best regards René
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Indy TIdHTTP worker events [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 3, 2015 1:57 PM   in response to: rene v.d. berge in response to: rene v.d. berge
rene wrote:

Result := select(nfds, readfds, writefds, exceptfds, timeout);

Returns '1' (socket available for guaranteed non-blocking read) AFTER
the socket is closed (FIN received and acknowledged)

As it should be. FIN indicates that the other party is gracefully closing
the connection. That is a readable condition, so select() leaves the socket
in the readfds set and returns > 0 to indicate that readfds has a socket
in it. select() cannot differentiate between inbound data and a graceful
close, but a subsequent call to recv() can. recv() will return 0 to indicate
the socket has been closed gracefully. Indy handles that condition. If
data is being read from a socket after a FIN closure has been received, Indy
will raise an EIdConnClosedGracefully exception. The fact that select()
is being called at all means the connection's IOHandler.InputBuffer does
not have enough data to satisfy a requested read operation and Indy has to
go back to the socket to receive more data into the InputBuffer. Which implies
that TIdHTTP might not be detecting the end of the HTTP response and is trying
to read past it. Hard to say without an example that reproduces it.

--
Remy Lebeau (TeamB)
rene v.d. berge

Posts: 9
Registered: 2/27/01
Re: Indy TIdHTTP worker events [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 4, 2015 12:35 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Thanks for your prompt response Remy.

Returns '1' (socket available for guaranteed non-blocking read) AFTER the socket is closed (FIN received and acknowledged)
My bad, it's not the FIN closure that makes select() return but
as far as I can tell it's the last TCP-frame (HTTP response). Before that
about 5MB has already been transfered but select() didn't return despite
data availability.
I will try to find out more today

Cheers René
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Indy TIdHTTP worker events [Edit]
Correct
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 4, 2015 3:44 PM   in response to: rene v.d. berge in response to: rene v.d. berge
rene wrote:

My bad, it's not the FIN closure that makes select() return but
as far as I can tell it's the last TCP-frame (HTTP response).

That means the last frame has not been read from the socket yet.

Before that about 5MB has already been transfered but select()
didn't return despite data availability.

If it is not returning, that means there is no data waiting to be read from
the socket.

--
Remy Lebeau (TeamB)
rene v.d. berge

Posts: 9
Registered: 2/27/01
Re: Indy TIdHTTP worker events [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 7, 2015 6:56 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Problem solved,
Virus scanner 'hijacked' the socket :/
Thanks for your input Remy!

René
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02