Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: Problem with Downloading files when using proxy



Permlink Replies: 10 - Last Post: Oct 13, 2014 5:45 AM Last Post By: Andy Murphy
Andy Murphy

Posts: 10
Registered: 2/13/03
Problem with Downloading files when using proxy
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 6, 2014 9:28 AM
Hi,

I have written a service application in delphi 2007 to get updates from our Amazon S3 file folders.

Its pretty simple, it has a CHRON style timer so basically when its between certain hours (usually 1am until 6am) it downloads a file from Amazon S3, stops a service (Firebird server), unzips the file and replaces the old one held locally. It then restarts the firebird service.

All simple stuff but I have had a few clients report the same problem. The Service crashes with I/O error 1784. I think its not downloading the file and hence cant save it. It only happens when proxy servers are in place. And like I say, most clients work fine.

The service is logged on as Local System account.

Here is where it gets tricky. I want to use IE as opposed to the third party download components as i have a lot of clients (300+) and if I use my download comps i have to set proxy settings and store them securely for every client that uses a proxy. So if I use IE to do it then usually IE has the proxy setup and of course I can test this by manually downloading my file from Amazon S3 on the clients server using IE.

I used some code from Torry's delphi pages to do this:

uses
URLMon, ShellApi;

function DownloadFile(SourceFile, DestFile: string): Boolean;
begin
try
Result := UrlDownloadToFile(nil, PChar(SourceFile), PChar(DestFile), 0, nil) = 0;
except
Result := False;
end;
end;

However this had a massive problem. IE downloads this to a cache usually (windows version dependant) c:\windows\system 32\config\systemprofile\appdatazlocal\microsoft\temporary internet files/content.ie5 etc.

now the problem was that we give them about 4gb of data a week and the function above kept copies of all the cached files so in 10 weeks we were using 40Gb of temporary files! not good.

So i changed the code.

I had some more code from torrys delpi pages that downloads a text file into a string value without hitting the hard drive as a file (very useful for downloading a small text file of update dates or versions for example).

function DownloadURL_NOCache(const aUrl: string; var s: String): Boolean;
var
hSession: HINTERNET;
hService: HINTERNET;
lpBuffer: array[0..1024 + 1] of Char;
dwBytesRead: DWORD;
begin
Result := False;
s := '';
// hSession := InternetOpen( 'MyApp', INTERNET_OPEN_TYPE_DIRECT, nil, nil, 0);
hSession := InternetOpen('MyApp', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
try
if Assigned(hSession) then
begin
hService := InternetOpenUrl(hSession, PChar(aUrl), nil, 0, INTERNET_FLAG_RELOAD, 0);
if Assigned(hService) then
try
while True do
begin
dwBytesRead := 1024;
InternetReadFile(hService, @lpBuffer, 1024, dwBytesRead);
if dwBytesRead = 0 then break;
lpBuffer[dwBytesRead] := #0;
s := s + lpBuffer;
end;
Result := True;
finally
InternetCloseHandle(hService);
end;
end;
finally
InternetCloseHandle(hSession);
end;
end;


but this doesnt download a file but interestingly has a flag of INTERNET_FLAG_RELOAD

so i ended up using this code i adpated, it works fine but like i say when there is a proxy involved even though i can download the file with IE (with the proxy settings) it returns I/O error 1784 because there is no file to save.

function GetInetFile(const fileURL, FileName: String): boolean;
const BufferSize = 1024;
var
hSession, hURL: HInternet;
Buffer: array[1..BufferSize] of Byte;
BufferLen: DWORD;
f: File;
sAppName: string;
begin
Result:=False;
sAppName := 'SyncService.exe';
hSession := InternetOpen(PChar(sAppName),
INTERNET_OPEN_TYPE_PRECONFIG,
nil, nil, 0);
try
hURL := InternetOpenURL(hSession,
PChar(fileURL),
nil,0,INTERNET_FLAG_RELOAD,0);
try
AssignFile(f, FileName);
Rewrite(f,1);
repeat
InternetReadFile(hURL, @Buffer,
SizeOf(Buffer), BufferLen);
BlockWrite(f, Buffer, BufferLen)
until BufferLen = 0;
CloseFile(f);
Result:=True;
finally
InternetCloseHandle(hURL)
end
finally
InternetCloseHandle(hSession)
end
end;

Can anyone see what is wrong with this code, why it might not use the proxy? or does the service have to be logged on by someone with special prvilages?

Any help much appreciated.

PS if anyone knows how to download using IE with proxy straight to the loacation with no cache at all used, even better as this still caches a copy under system profile before discarding.

many thanks

Andy

Lajos Juhasz

Posts: 801
Registered: 3/14/14
Re: Problem with Downloading files when using proxy
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 6, 2014 11:00 AM   in response to: Andy Murphy in response to: Andy Murphy
Andy Murphy wrote:

it returns I/O error 1784 because there is no file to save.

Unfortunately I cannot test your code. However I've found on
stackoverflow a similair C++ problem and there the GetLastError was:

"err returns #1784 which translates to supplied user buffer is not
valid for the requested operation. ERROR_INVALID_USER_BUFFER"

My first attemt to guards that if there was read 0 do not try to write
to file:

function GetInetFile(const fileURL, FileName: String): boolean;
const BufferSize = 1024;
var hSession, hurl:  HInternet;
    Buffer: array[1..BufferSize] of Byte;
    BufferLen: DWORD;
    f: File;
    sAppName: string;
 
begin
  Result:=False;
  sAppName := 'SyncService.exe';
   hSession := InternetOpen(PChar(sAppName),
                            INTERNET_OPEN_TYPE_PRECONFIG,
                            nil, nil, 0);
  try
    hURL := InternetOpenURL(hSession, PChar(fileURL),
nil,0,INTERNET_FLAG_RELOAD,0);
    try
      AssignFile(f, FileName);
      Rewrite(f,1);
      repeat
        InternetReadFile(hURL, @Buffer, SizeOf(Buffer), BufferLen);
        if BufferLen>0 then
           BlockWrite(f, Buffer, BufferLen)
      until BufferLen = 0;
      CloseFile(f);
      Result:=True;
    finally
      InternetCloseHandle(hURL)
    end
  finally
   InternetCloseHandle(hSession)
  end
end;


My next step would be to forget about the old Turbo Pascal style file
variable that should be ancient pass. These days the streams should be
used to write data to disc:

function GetInetFile(const fileURL, FileName: String): boolean;
const BufferSize = 1024;
var hSession, hurl:  HInternet;
    Buffer: array[1..BufferSize] of Byte;
    BufferLen: DWORD;
    f: TFileStream;
    sAppName: string;
 
begin
  Result:=False;
  sAppName := 'SyncService.exe';
   hSession := InternetOpen(PChar(sAppName),
                            INTERNET_OPEN_TYPE_PRECONFIG,
                            nil, nil, 0);
  try
    hURL := InternetOpenURL(hSession, PChar(fileURL),
nil,0,INTERNET_FLAG_RELOAD,0);
    try
      f:=TFileStream.Create(fileName, fmCreate+fmShareExclusive);
      try
        repeat
          InternetReadFile(hURL, @Buffer, SizeOf(Buffer), BufferLen);
          if BufferLen>0 then
            f.Write(Buffer[1], BufferLen);
        until BufferLen = 0;
        Result:=True;
      finally
        f.Free;
      end;
 
    finally
      InternetCloseHandle(hURL)
    end
  finally
   InternetCloseHandle(hSession)
  end
end;


Warning these procedures are untested!
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Problem with Downloading files when using proxy
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 6, 2014 11:16 AM   in response to: Andy Murphy in response to: Andy Murphy
Andy wrote:

All simple stuff but I have had a few clients report the same problem.
The Service crashes with I/O error 1784.

That is ERROR_INVALID_USER_BUFFER, "The supplied user buffer is not valid
for the requested operation."

That is a old-style Pascal file I/O error message. Why are you using old-style
Pascal I/O?

Here is where it gets tricky. I want to use IE as opposed to the third
party download components

Why?

i have a lot of clients (300+) and if I use my download comps i have to
set proxy settings and store them securely for every client that uses a
proxy. So if I use IE to do it then usually IE has the proxy setup

Yes, but it is a global setting, shared by every WinInet instance, including
the IE web browser. If you want to control the proxy setup just for your
dowwnload clients, you have to set the proxy values in code on a per-socket
basis anyway. So using WinInet vs another library just to gain proxy support
really doesn't offer a very big advantage. You could just store the proxy
settings in one place, or even query the IE settings, and have each client
apply the settings to each download when needed.

I used some code from Torry's delphi pages to do this:
<snip>
However this had a massive problem. IE downloads this to a cache

Yup. The only way to bypass that is to download the file manually so you
control how the data is read and saved. If you want to stay with the WinInet
API, you can use InternetReadFile(), just be sure to specify the INTERNET_FLAG_NO_CACHE_WRITE
flag when creating the request.

but this doesnt download a file

Yes, it does.

However, the code as shown will not work in Unicode versions of Delphi due
to the use of Char, which is an alias for WideChar now. The code assigns
that Char and String are Ansi, which means this is an old example. It is
also not doing ay error handling on the InternetReadFile() call.

but interestingly has a flag of INTERNET_FLAG_RELOAD

All that flag does is tells WinInet to force the request to be sent to the
target HTTP server, skipping the local cache and any intermediate caching
proxies along the way, so that the latest version of the data is retreived.

so i ended up using this code i adpated, it works fine but like i say
when there is a proxy involved even though i can download the file
with IE (with the proxy settings) it returns I/O error 1784 because
there is no file to save.

That is not the reason for the error. And there is a file saved in the cache,
because you are not using the INTERNET_FLAG_NO_CACHE_WRITE flag.

Try something more like this instead:

function GetInetFile(const fileURL, FileName: String): boolean;
const
BufferSize = 1024;
var
hSession, hURL: HInternet;
Buffer: array[0..BufferSize-1] of Byte;
BufferLen: DWORD;
f: TFileStream;
sAppName: string;
begin
Result := False;
hSession := InternetOpen('SyncService.exe', INTERNET_OPEN_TYPE_PRECONFIG,
nil, nil, 0);
if hSession = 0 then Exit;
try
hURL := InternetOpenURL(hSession, PChar(fileURL), nil, 0, INTERNET_FLAG_RELOAD
or INTERNET_FLAG_NO_CACHE_WRITE, 0);
if hURL = 0 then Exit;
try
f := TFileStream.Create(FileName, fmCreate);
try
try
repeat
if not InternetReadFile(hURL, @Buffer[0], SizeOf(Buffer), BufferLen)
then RaiseLastOSError;
if BufferLen = 0 then Break;
f.WriteBuffer(Buffer[0], BufferLen);
until False;
finally
f.Free;
end;
except
DeleteFile(FileName);
Exit;
end;
Result := True;
finally
InternetCloseHandle(hURL);
end
finally
InternetCloseHandle(hSession);
end
end;
{code}

--
Remy Lebeau (TeamB)
Andy Murphy

Posts: 10
Registered: 2/13/03
Re: Problem with Downloading files when using proxy
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 7, 2014 3:24 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy,

Many thanks, will that code handle the proxy if required too?

my knowledge of this stuff is pretty limited as I normally develop database apps with Delphi using Dev Express grids menus etc etc So this is a first go at updating my data using a service and its my first service I have written.

Ill give your function a try and see how we get on and report back, but many thanks again.
Andy Murphy

Posts: 10
Registered: 2/13/03
Re: Problem with Downloading files when using proxy
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 7, 2014 3:50 AM   in response to: Andy Murphy in response to: Andy Murphy
Ps I had to change hSession = 0 to hSession = nil and hURL = 0 to hURL = nil as it said that operand was not applicable to that type, is that correct?
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Problem with Downloading files when using proxy
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 7, 2014 12:58 PM   in response to: Andy Murphy in response to: Andy Murphy
Andy wrote:

Many thanks, will that code handle the proxy if required too?

It uses INTERNET_OPEN_TYPE_PRECONFIG, so it will use whatever proxy is configured
in IE. If you want to assign the proxy in code instead, look at InternetSetOption()
and its INTERNET_OPTION_PER_CONNECTION_OPTION and INTERNET_OPTION_PROXY...
flags. See this for an example:

How to programmatically query and set proxy settings under Internet Explorer
http://support.microsoft.com/kb/226473

--
Remy Lebeau (TeamB)
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Problem with Downloading files when using proxy
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 7, 2014 1:27 PM   in response to: Andy Murphy in response to: Andy Murphy
Andy wrote:

Many thanks, will that code handle the proxy if required too?

It uses INTERNET_OPEN_TYPE_PRECONFIG, so it will use whatever proxy is configured
in IE. If you want to assign the proxy in code instead, look at InternetSetOption()
and its INTERNET_OPTION_PER_CONNECTION_OPTION and INTERNET_OPTION_PROXY...
flags. See this for an example:

How to programmatically query and set proxy settings under Internet Explorer
http://support.microsoft.com/kb/226473

--
Remy Lebeau (TeamB)
Andy Murphy

Posts: 10
Registered: 2/13/03
Re: Problem with Downloading files when using proxy
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 8, 2014 3:50 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
hi Remy,

Many thanks your code worked well and it no longer caches the file which is ideal.

We found the problem with the proxy.

The service logs on as Local System account, we found that just because a user sets up the proxy in IE that doesn't mean the Local Service account has the proxy settings. Hence the file could not be downloaded. Using your function I now handle that in a much better way where the service effectively does nothing but writes a warning to the event viewer to say the Internet was unavailable and not the service no longer crashes and stops working.

We found three solutions to this I'm still not sure which is best practice but I'll post them here for other devs to see.

Fix 1. Use regedit to export the current users reg entry for a user for whom IE is set up with the proxy. The key is normally HKEY_current_user/software/Microsoft/current version/Internet settings

Then edit the key in notepad, replace HKEY_CURRENT_USER with HKEY_USER/S-1-5-18 through out the file then import it back into the registry. This adds all the proxy settings to the LOCAL SYSTEM account.

Fix 2 open CMD as admin type NETSH WINHTTP IMPORT PROXY SOURCE = ie
On 64 bit machines you have to change directory to SystemRoot%\SysWOW64 folder first. This works on 2008/12 server I have no idea how to do it on 2003.

Fix 3 setup an account with full local admin access (in our case because our service stops and starts other services) and make sure the proxy settings for that user are correct and then log the service on as that account. While this might seem simple and the safest option a lot of our clients have password expiry policies which means they would have to remember to change the service log on password frequently, which they won't hence the service won't run.

Anyway what is odd is the vast majority of clients we have with proxy servers run fine, but we have found a handful that need this fix so I'm guessing it's a group policy thing.

Anyone with any thoughts on a neater solution, let me know.

Many thanks for the improved code again!
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Problem with Downloading files when using proxy
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 8, 2014 5:54 PM   in response to: Andy Murphy in response to: Andy Murphy
Andy wrote:

The service logs on as Local System account, we found that just
because a user sets up the proxy in IE that doesn't mean the Local
Service account has the proxy settings. Hence the file could not be
downloaded.

Yes, IE settings are stored per-user.

We found three solutions to this I'm still not sure which is best
practice but I'll post them here for other devs to see.

Is there a problem using the solution I mentioned to you?

If you want to assign the proxy in code instead, look at InternetSetOption()
and its INTERNET_OPTION_PER_CONNECTION_OPTION and INTERNET_OPTION_PROXY...
flags. See this for an example:

How to programmatically query and set proxy settings under Internet Explorer
http://support.microsoft.com/kb/226473

--
Remy Lebeau (TeamB)
Andy Murphy

Posts: 10
Registered: 2/13/03
Re: Problem with Downloading files when using proxy
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 9, 2014 4:15 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Hi Remy,

i havent tried it simply because i dont know how to create a bit of pascal from the code on that site, dont get me wrong if we could have our service pickup on the proxy regardless of the user its logged in as then i would be very happy i just didnt really understand how to add the code in that link you sent into my download file function.
Andy Murphy

Posts: 10
Registered: 2/13/03
Re: Problem with Downloading files when using proxy
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 13, 2014 5:45 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Anyone got any ideas how to build this proxy finding/setting code found in Remy's answer into the function that we use?

Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02