Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: IdHTTPProxyServer. Cascading.


This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 5 - Last Post: Jan 13, 2017 2:14 PM Last Post By: Remy Lebeau (Te...
Alex Gubarev

Posts: 8
Registered: 11/12/16
IdHTTPProxyServer. Cascading.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Nov 9, 2016 12:25 AM
Hi, all!
I'm trying to write a kinda sniffer for browser requests. All I need are all headers and "GET" requests. No direct Internet connection available. Via external proxy only.
So, I have IdHTTPProxyServer on my form, listening port 3128. Bindings: 0.0.0.0:3128.
External proxy is: 172.16.0.1:8080 - no auth required.
void __fastcall TForm1::ProxyHTTPBeforeCommand(TIdHTTPProxyServerContext *AContext)
{
TIdIOHandlerStack* tempIO=new TIdIOHandlerStack(AContext->Connection->IOHandler);
TIdConnectThroughHttpProxy* tempProxy=new TIdConnectThroughHttpProxy(AContext->Connection->IOHandler);
 
tempProxy->Host="172.16.0.1";
tempProxy->Port=8080;
tempProxy->Username="user";
tempProxy->Password="pass";
tempProxy->Enabled=true;
AContext->OutboundClient->Socket->TransparentProxy=tempProxy;
AContext->OutboundClient->Socket->TransparentProxy->Enabled=true;
// or
// tempIO->TransparentProxy=tempProxy;
// tempIO->TransparentProxy->Enabled=true;
// AContext->OutboundClient->IOHandler=tempIO;
}

IE tries to open URL and throws "cannot display the webpage". Of course, with external proxy settings in browser all works fine.
What's the correct way to make forwarding to external proxy?
Thanks!
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: IdHTTPProxyServer. Cascading.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Nov 9, 2016 11:06 AM   in response to: Alex Gubarev in response to: Alex Gubarev
Alex wrote:

TIdIOHandlerStack* tempIO=new
TIdIOHandlerStack(AContext->Connection->IOHandler);

You need to assign that temp IOHandler to the OutboundClient->IOHandler property,
otherwise it is just wasting resources.

TIdConnectThroughHttpProxy* tempProxy=new
TIdConnectThroughHttpProxy(AContext->Connection->IOHandler);

Not important to your issue, but I would suggest setting the OutboundClient
as the Owner instead.

AContext->OutboundClient->Socket->TransparentProxy=tempProxy;
AContext->OutboundClient->Socket->TransparentProxy->Enabled=true;

The OutboundClient's IOHandler property, and thus its Socket property, is
NULL when the OnHTTPBeforeCommand event handler is called. Assigning your
temp IOHandler to the OutboundClient->IOHandler will also populate the OutboundClient->Socket
property.

What's the correct way to make forwarding to external proxy?

You are already doing it, assuming the external proxy is actually an HTTP
proxy and not some other kind of proxy, like SOCKS.

I suggest you debug your code, and use an actual sniffer like Wireshark,
to make sure your app is connecting to the external proxy correctly and sending
the browser's HTTP request correctly.

--
Remy Lebeau (TeamB)
Alex Gubarev

Posts: 8
Registered: 11/12/16
Re: IdHTTPProxyServer. Cascading.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 10, 2017 12:28 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy, thanks for your help! I've found out that my problem is well known: IdHTTPProxyServer sends to forward proxy CONNECT before GET.
Some external proxy rejects CONNECT attempts due security reasons. So, they're just throws 403 and close connection.

Well, maybe somebody can give me advice how to intercept browsers requests via my "fake" proxy through external proxies? I need HTTP and HTTPS GET/POST requests, headers and cookies. Something like that: browser <-> my proxy <-> external proxy.
Yes, it's kinda man-in-the-middle stuff. :)
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: IdHTTPProxyServer. Cascading.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 13, 2017 2:13 PM   in response to: Alex Gubarev in response to: Alex Gubarev
Alex wrote:

I've found out that my problem is well known: IdHTTPProxyServer sends
to forward proxy CONNECT before GET.

The client sends CONNECT to TIdHTTPProxyServer when the client wants to
establish a tunnel to the target server before sending any data through TIdHTTPProxyServer.
This is commonly used for HTTPS requests, so that SSL/TLS data is forwarded
as-is, allowing the client and target server to verify each other. TIdHTTPProxyServer
does not send CONNECT unless you explicitly set it up in code to do that
(such as by assigning a TIdConnectThroughHttpProxy to the OutboundClient,
like I described earlier).

Some external proxy rejects CONNECT attempts due security reasons.

Then they will also reject HTTPS requests, since CONNECT is typically required
for proxying SSL/TLS connections.

Well, maybe somebody can give me advice how to intercept browsers
requests via my "fake" proxy through external proxies?

The only way I can think of to make this work with TIdHTTPProxyServer is
to use its OnHTTPBeforeCommand event:

1. manually read the client's complete request body (at the time of the event,
only the headers have already been read and stored in the TIdHTTPProxyServerContext.Headers
property).

2. then use TIdHTTP to send a new request to the target server, using TIdHTTP's
built-in HTTP proxy support, and read the server's response in full.

3. then send that response data back to the client that is connected to TIdHTTPProxyServer.

4. and then finally raise an exception when finished, so that TIdHTTPProxyServer's
default OutboundClient logic is bypassed (it will still allocate a TIdTCPClient,
but does not connect it to the server until after OnHTTPBeforeCommand exits).

Unfortunately, this will only work if the client sends GET/POST requests
directly to TIdHTTPProxyServer itself, providing a full URL for use in the
next connection leg. If the client sends CONNECT instead (which is likely
for HTTPS requests), this approach will not work.

TIdHTTPProxyServer is simply not designed for what you are attempting.

I need HTTP and HTTPS GET/POST requests, headers and cookies.

Good luck. Sniffing HTTP is easy, but sniffing HTTPS is very difficult.
Unless you have access to the target server's private encryption key, you
need two separate HTTPS sessions, one between the client and yourself, and
another between yourself and the next server, and then translate data betwen
the two sessions. TIdHTTPProxyServer (and TIdMappedPortTCP) is not designed
for that. You might have some success using TIdHTTPServer and TIdHTTP instead,
implementing all of the proxy logic manually, but it will be a bit of work
to pull off.

Something like that: browser <-> my proxy <-> external proxy.

Yes, it's kinda man-in-the-middle stuff. :)

Which means you can't sniff HTTPS requests, since the whole point of SSL/TLS
is to prevent MITM attacks. You would have to implement a full-on HTTP server
instead of an HTTP proxy, where browsers connect to your server thinking
it is the target server (instead of being a proxy to the target server).
And then your server can silently communicate with the target server as
needed when the browser sends requests to your server.

This is how Fiddler implements HTTPS sniffing, for instance. Fiddler pretends
to be an HTTP server instead of an HTTP proxy. It works, but doing this
has security concerns, not to mention it requires explicit permission from
the browser user, since the browser knows it is not connected to the real
server and will warn the user accordingly. You simply cannot sniff HTTPS
transparently (otherwise there would be no point in HTTPS existing at all).

--
Remy Lebeau (TeamB)
Alex Gubarev

Posts: 8
Registered: 11/12/16
Re: IdHTTPProxyServer. Cascading.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 12, 2017 9:55 PM   in response to: Alex Gubarev in response to: Alex Gubarev
Well, I have an idea how to intercept requests in my case.
With TIdMappedPortTCP in OnBeforeConnect:
void __fastcall TForm1::TCPBeforeConnect(TIdContext *AContext)
{
AnsiString buf="";
do
{
if(buf!="") buf=buf+AContext->Connection->Socket->ReadLn()+"\r\n";
}while(buf!="");
... save buf somewhere
AContext->Connection->Socket->WriteLn(buf);
}

But it doesn't work. TIdMappedPortTCP doesn't send anything forward. What the correct way to do this?
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: IdHTTPProxyServer. Cascading.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 13, 2017 2:14 PM   in response to: Alex Gubarev in response to: Alex Gubarev
Alex wrote:

Well, I have an idea how to intercept requests in my case.
<snip>
But it doesn't work.

Why do you expect it to work? You are reading the client's HTTP request
headers and sending them back to the same client.

TIdMappedPortTCP doesn't send anything forward.

Because you are not allowing it to. You are reading the inbound data yourself
and sending it on to the wrong target. There is nothing left for TIdMappedPortTCP
to read and forward.

If all you want to do is sniff the data that is sent through TIdMappedPortTCP,
use the OnExecute (client-to-server) and OnOutboundData (server-to-client)
events instead. Type-cast the provided TIdContext to TIdMappedPortContext,
and then the raw data will be available in its NetData property. Just note
that TIdMappedPortTCP is a proxy for arbitrary data, so you will have to
account for NetData having no structure to it, and the data may span across
multiple events. And in the case of SSL/TLS connections, the data will be
encrypted.

--
Remy Lebeau (TeamB)
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02