Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: TCP Receive window size is zero



Permlink Replies: 6 - Last Post: Jul 28, 2015 2:53 PM Last Post By: Curtis Lending
Curtis Lending

Posts: 33
Registered: 11/24/14
TCP Receive window size is zero
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jul 23, 2015 12:07 PM
I have an application which has 6 TIdTCPClients not all of which are always connected. Each second 96000 application packets of length 576 are sent (55 MBytes) per channel. I have intermittent failures which I do not understand. Sometimes things run properly with all channels. Other times it fails with only one (or more) channels. Using Wireshark, I can tell the failures occur because the TCP receive window drops to zero. This happens even though the packets are being properly ACKed. I thought if the packet got ACKed, the receive window would not decrease. Task manager shows the CPU usage is around 30%.

I have the receive buffer set to 100,000,000. (I understand this is a different buffer but it should supply plenty of room for the TCP data to be stored if I do not read them on time.)
ChanTCPClient[*chIndex]->IOHandler->RecvBufferSize = 100000000;
I read 576 bytes at a time:
ChanTCPClient[chIndex]->IOHandler->ReadBytes(readBuffer, readLength, APPEND);

In looking at a Wireshark trace, I see the following.
1) The servers always sends 4 application packets in two TCP packets of size 1460 and 844.
2) The Indy protocol always ACKs every other packet even when the error occurs.
3) When I look at the ACKs, I see that the TCP receive windows starts at 16425. In a normal run, it drops lower and then increases again.

In examining a Wireshark trace with a failure:
1) The window size starts decreasing on the ACK in frame 432 and goes to zero in packet 539.
16425, 15489, 15273, 14553 .... 1161, 585, 9, 0. Notice that on the packets on the end the window decreases by 576 each time.
2) Every other packet is still be ACKed.

Here are the packets at the end of the Wireshark trace.
Svr	Client
507		Data 1514/1460 bytes - 1514 is the packet size.  1460 is number of application bytes.
	508	Ack to 507, Window size 229
509 	Data 898/844 bytes
510		Data 1514/1460 bytes
	511	Ack to 510, Window size 1161
512		Data 898/844 bytes
513		Data 1514 bytes
	514	Ack to 513, Window size 585
515		Data 898/844 bytes
516		Data 1514/1469 bytes
	517	Ack to 516, Window size 9
518		Data 90/36 Window Full
	519	Zero Window, Ack to 518, Window size 0
520		Keep Alive
	521	Zero Window, Window size 0
522		Keep Alive
	523	Zero Window, Window size 0
	524 from switch
525		Keep Alive
	526	Zero Window, Window size 0
527		RST, ACK	


Thank you,

Curtis
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TCP Receive window size is zero
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jul 23, 2015 12:38 PM   in response to: Curtis Lending in response to: Curtis Lending
Curtis wrote:

Using Wireshark, I can tell the failures occur because the TCP
receive window drops to zero.

That means you are not reading inbound data fast enough (if at all), so the
socket's internal receive buffer is filling up to the point where it cannot
receive any more data, causing the sender to block transmissions on its side
waiting for you to read data from the receive buffer to free up space for
new data.

What does your code actually look like? Are you managing the TIdTCPClient
connections in a single thread or in separate threads? What exactly does
a "failure" consist of (error code? app crash?), and is it a failure on the
sending side or the receiving side?

--
Remy Lebeau (TeamB)
Curtis Lending

Posts: 33
Registered: 11/24/14
Re: TCP Receive window size is zero
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jul 23, 2015 2:11 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
First of all, the forum takes minutes to respond when I click reply or edit. This is very annoying.

That means you are not reading inbound data fast enough (if at all), so the
socket's internal receive buffer is filling up to the point where it cannot
receive any more data, causing the sender to block transmissions on its side
waiting for you to read data from the receive buffer to free up space for
new data.
That's why I made IOHandler->RecvBufferSize so large. I thought Indy would buffer the data there so I can read it later.
https://forums.embarcadero.com/message.jspa?messageID=703303#703303
It seems to work this way in another case where I know I am lagging behind for several seconds because of other signal processing. In this case, I should be reading the data. I have no way to know for sure because the failures are intermittent but I cannot imagine that I fail to do a read for the 2 seconds during which the receive window is zero.

What does your code actually look like? Are you managing the TIdTCPClient
connections in a single thread or in separate threads?
Single thread where I read all the channels in a loop via the following call:
ChanTCPClient[chIndex]->IOHandler->ReadBytes(readBuffer, readLength, APPEND); 

What exactly does a "failure" consist of (error code? app crash?), and is it a failure on the
sending side or the receiving side?
The server I am talking to (sending side) resets the connection because the window size stayed at zero for over two seconds while it sent keep alive messages. (See trace in first post.)

BTW, I gave the wrong speed in the first post. 375 application packets of size 576 bytes per second for a speed of 0.2 Mbytes/second.

Thanks,

Curtis
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TCP Receive window size is zero
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jul 23, 2015 4:35 PM   in response to: Curtis Lending in response to: Curtis Lending
Curtis wrote:

That's why I made IOHandler->RecvBufferSize so large.

That has nothing to do with how often you are calling ReadBytes(), or whether
you are able to call ReadBytes() on one socket while another socket is blocked
waiting for its ReadBytes() to finish. Remember, everything in Indy is designed
for blocking I/O. When you call ReadBytes(), it does not exit until it has
read the specified number of bytes, or a timeout/error occurs. During that
time, the calling thread is not able to do anything else.

I thought Indy would buffer the data there so I can read it later.

It does, but not automatically. You have to perform a reading operation
in order to invoke the buffering. When you call a reading method, such as
ReadBytes(), it checks for existing bytes in the IOHandler's InputBuffer
first, and if the InputBuffer does not have enough bytes to satisfy the requested
read then it waits for the remaining bytes to arrive on the socket, then
they are removed from the socket (incrementing the receive window) and placed
into the InputBuffer, and then extracted to your specified output buffer.
So, if you are not performing read operations frequently enough, Indy does
not have a chance to get the bytes out of the socket.

It seems to work this way in another case where I know I am lagging
behind for several seconds because of other signal processing. In this
case, I should be reading the data. I have no way to know for sure
because the failures are intermittent but I cannot imagine that I fail
to do a read for the 2 seconds during which the receive window is
zero.

You can, if you are blocking yourself from doing the read because you are
waiting on something else to finish first. That is why I asked you to show
your actual code, to find out whetehr your code for the various TIdTCPClient
objects is designed to run sequentially or in parallel.

Single thread where I read all the channels in a loop via the following
call

Sequentially then. That is why your code is failing. While you are waiting/reading
data for one channel, you are not reading data for the other channels, so
they fill up and time out. You need to change your logic. The best option
is to run each TIdTCPClient in its own worker thread. And thread can continuously
read from its own socket and buffer that data somewhere that the rest of
your code can get to it when needed. For example, based on code you showed
a month ago, it might look something like this (semi-pseudo code):

void __fastcall TM3RReadingThread::Execute()
{
    TIdBytes readBuffer;
    readBuffer.Length = M3R_EXPECTED_LENGTH;
 
    while (!Terminated)
    {
        try
        {
            ChanTCPClient[chIndex]->IOHandler->ReadBytes(readBuffer, M3R_EXPECTED_LENGTH, 
false);
 
            bufferLock->Enter();
            try {
                // append M3R_EXPECTED_LENGTH bytes from readBuffer to end 
of thread buffer...
            }
            __finally {
                bufferLock->Leave();
            }
        }
        catch (...)
        {
            // disconnect ...
        }
 
        // let ReadChannel() know that something happened...
        ChanTCPSignal[chIndex]->SetEvent();
    }
}
 
bool __fastcall TM3RReadingThread::GetPacket(M3R_DATA_PACKET *packet, U16 
*seqNum)
{
    TIdBytes readBuffer;
 
    bufferLock->Enter();
    try {
        // extract up to M3R_EXPECTED_LENGTH bytes from front of thread buffer...
 
        if (fewer than M3R_EXPECTED_LENGTH byte are in the thread buffer)
            ChanTCPSignal[chIndex]->ResetEvent();
    }
    __finally {
        bufferLock->Leave();
    }
 
    if (readBuffer.Length <= M3R_EXPECTED_LENGTH) {
        DBG_PRINT("Stop - read done");
        return false;
    }
 
    if (!MoveAndCheckPacket(readBuffer, packet))
        return false;
 
    *seqNum = packet->header.hdr.seq_num;
 
    return true;
}
 
bool __fastcall TM3RParamsForm::ReadChannel(long channel, M3R_DATA_PACKET 
*packet, U16 *seqNum)
{
    int chIndex = getIndexForChannel(channel);
 
    try
    {
        if (ChanTCPSignal[chIndex]->WaitFor(some timeout period) != wrSignaled)
            return false;
 
        return ChanTCPThread[chIndex]->GetPacket(packet, seqNum);
    }
    catch (...)
    {
        gError(STR_M3R_COMM_FAIL);
        DisconnectChannel(channel);
        return false;
    }
}


At least this way, you can still call ReadChannel() in a sequential loop,
but the actual data collection is running the background so each socket connection
stays responsive to the server.

The server I am talking to (sending side) resets the connection
because the window size stayed at zero for over two seconds while it
sent keep alive messages. (See trace in first post.)

That is a pretty short timeout. All the more reason to run each TIdTCPClient
connection in its own thread.

--
Remy Lebeau (TeamB)
Curtis Lending

Posts: 33
Registered: 11/24/14
Re: TCP Receive window size is zero
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jul 27, 2015 11:38 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Let me clarify my scenario.
1) The servers each send a pair of TCP packets every 10 msec. This always happens and using Wireshark we have verified that it continues even when the error occurs. If I read data, I will never wait more than 10 msecs.
2) Sometimes, I get the zero window error when I have only one channel which doubly means I cannot be blocked in a read on another channel.

In the one channel Wireshark trace I am looking at, the TCP read window size is 16,425 for about 1.5 seconds (approximately 280 TCP packets or 560 ReadBytes or 300,000 bytes), then it starts dropping. I do not understand how this could occur with RecvBufferSize = 100,000,000.

If I had a bug and never called ReadBytes(), I thought Indy continue to read from the socket and stored the 300,000 bytes in an internal buffer of size RecvBufferSize. Because data is being read from the socket, the TCP read window size would stay at 16,425 until the buffer filled. If I later called ReadByes, I would be able read the data. Is this correct or am missing something?

I understand that my thread blocks when I call ReadBytes. I thought Indy was interrupt driven and, in the background, would continue process the sockets regardless of what any of my thread are or are not doing.

Thank you,

Curtis
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TCP Receive window size is zero
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jul 27, 2015 1:18 PM   in response to: Curtis Lending in response to: Curtis Lending
Curtis wrote:

In the one channel Wireshark trace I am looking at, the TCP read
window size is 16,425 for about 1.5 seconds (approximately 280 TCP
packets or 560 ReadBytes or 300,000 bytes), then it starts dropping.
I do not understand how this could occur with RecvBufferSize =
100,000,000.

The TIdIOHandler::RecvBufferSize property only affects the size of the in-memory
buffer that Indy uses to read bytes into before putting them into the TIdIOHandler::InputBuffer.
RecvBufferSize has no effect whatsoever on the soize of the socket's internal
buffer that receives bytes from the network. If you want to change the size
of the socket's internal buffer, you need to use the TIdSocketHandle::SetSocketOpt()
method, eg:

TheTCPConnection->Socket->Binding->SetSockOpt(Id_SOL_SOCKET, Id_SO_RCVBUF, 
TheDesiredBufferSize);


If I had a bug and never called ReadBytes(), I thought Indy continue
to read from the socket and stored the 300,000 bytes in an internal
buffer of size RecvBufferSize.

No, it does not. As I explained in my previous reply, Indy only reads from
the socket when a reading operation, such as ReadBytes(), needs more bytes
than are already cached in the TIdIOHandler::InputBuffer. If you do not
tell Indy do read something from the IOHandler, it won't read anything from
the socket at all, it will just sit idle, and thus any bytes received on
the network will stay in the socket's internal buffer, reducing the socket's
recv window. So, if you are expecting data to arrive at regular intervals,
you need to keep reading from the IOHandler at regular intervals to keep
the socket's internal buffer free to receive new bytes.

BTW, this also includes the Connected() method, which internally reads from
the socket with a 0ms timeout, and stores any data it happens to read into
the InputBuffer.

Because data is being read from the socket, the TCP read window size
would stay at 16,425 until the buffer filled.

It would only be read from the socket when you tell Indy to read data.

If I later called ReadByes, I would be able read the data.

It would read whatever is already in the InputBuffer, and if more bytes are
needed THEN it reads from the socket, caching whatever is left over in the
InputBuffer for later reads.

Is this correct or am missing something?

No, it is not correct. You are missing something.

I understand that my thread blocks when I call ReadBytes.

Only if the InputBuffer does not already have enough bytes cached, otherwise
it returns immediately.

I thought Indy was interrupt driven

No, it is not. Wherever did you get that idea from?

It is not event-driven, either. Though some Indy components do provide events
when new data is available, but they are simply triggered by an internal
thread that blocks until data is received.

and, in the background, would continue process the sockets
regardless of what any of my thread are or are not doing.

Indy I/O operations block the calling thread until the I/O is finished.
Whatever thread is reading inbound data is blocked until the reqeusted data
is available. You can certainly have other threads running in parallel.
But if you have multiple sockets being served by the same thread, they are
going to block each other accordingly.

--
Remy Lebeau (TeamB)
Curtis Lending

Posts: 33
Registered: 11/24/14
Re: TCP Receive window size is zero
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jul 28, 2015 2:53 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Do you know of a good example of using Indy with multiple threads?

Thanks,

Curtis
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02