Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: TIdHTTP with SSL certificate verification


This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 10 - Last Post: Nov 2, 2015 9:45 AM Last Post By: Gennadiy Kiryuk...
Gennadiy Kiryuk...

Posts: 19
Registered: 10/25/02
TIdHTTP with SSL certificate verification  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 29, 2015 12:36 PM
I'm trying to use TIdHTTP with TIdSSLIOHandlerSocketOpenSSL download a page from a website.
I can load the page with a browser (https://...) and the browser (Chrome) does not complain about the certificate (issues by GoDaddy).
However, when running a TIdHTTP->Get("https://..."), OnVerifyPeer is called with AOk set to false and AError equal 20.
OnVerifyPeer returns AOk value without any other checking.

From different web searches an a lot of head scratching I came up with this code to setup my TIdHTTP object.

	HTTPD = new TIdHTTP(NULL);
	HTTPD->AllowCookies = false;
	HTTPD->OnWorkBegin = WorkBegin;
	HTTPD->OnWorkEnd = WorkEnd;
	HTTPD->OnWork = Work;
	SSL = new TIdSSLIOHandlerSocketOpenSSL(NULL);
	HTTPD->IOHandler = SSL;
	HTTPD->HandleRedirects = false;
	SSL->OnVerifyPeer = VerifyPeerEvent;
	SSL->OnStatus = IdStatusEvent;
	SSL->SSLOptions->Method = sslvSSLv23;
	SSL->SSLOptions->Mode = sslmUnassigned;
	SSL->SSLOptions->VerifyMode = TIdSSLVerifyModeSet()<<sslvrfPeer;
 
	...
	HTTPD->Get("https://my-url-goes-here");


How can I troubleshoot the problem? What is AError 20?

Thank you.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TIdHTTP with SSL certificate verification  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 29, 2015 1:46 PM   in response to: Gennadiy Kiryuk... in response to: Gennadiy Kiryuk...
Gennadiy wrote:

From different web searches an a lot of head scratching I
came up with this code to setup my TIdHTTP object.

This is required only if you actually want to validate the server's identity.
In which case, before calling TIdHTTP.Get(), you have to load the server's
trusted certificate(s) (that are used to sign the server's certificate) into
TIdSSLIOHandlerSocketOpenSSL via its SSLOptions.RootCertFile or SSLOptions.VerifyDirsVerifyDirs
property. Certificate validation is not required to make a connection, only
to validate the server's identity if you want to avoid MITM attacks. Otherwise,
ignore the certificate and remove the sslvrfPeer flag from the SSLOptions.VerifyMode
property.

What is AError 20?

It is X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY:

unable to get local issuer certificate

the issuer certificate of a locally looked up certificate could not be found.
This normally means the list of trusted certificates is not complete.

--
Remy Lebeau (TeamB)
Gennadiy Kiryuk...

Posts: 19
Registered: 10/25/02
Re: TIdHTTP with SSL certificate verification  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 29, 2015 2:32 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
At a minimum, I would like to check and make sure that the certificate is not self-signed and corresponds to the requested url.
Our certificate is issues for www.url.com. However, if I browse the website via https://updates.url.com, the browser warns me about the mismatched certificate ("This Connection is Untrusted"). I'm pretty sure Chrome and Firefox do not have our certificate and yet they are able to verify the certificate somehow. Can they detect MITM attacks?

Is it safe to distribute our web certificate along with our software? What if the certificate that was included with our software expires?
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TIdHTTP with SSL certificate verification  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 29, 2015 3:11 PM   in response to: Gennadiy Kiryuk... in response to: Gennadiy Kiryuk...
Gennadiy wrote:

At a minimum, I would like to check and make sure that the certificate
is not self-signed and corresponds to the requested url.

That means comparing and validating the chain of certificates that the server's
certificate was signed with, which means you need prior access to those certificates
and have them loaded into OpenSSL's certificate store. There is no flag
on the server's certificate that says "hey, I'm self-signed".

Our certificate is issues for www.url.com. However, if I browse the
website via https://updates.url.com, the browser warns me about the
mismatched certificate ("This Connection is Untrusted"). I'm pretty
sure Chrome and Firefox do not have our certificate and yet they are
able to verify the certificate somehow.

They may not have your particular certificate, but they likely have the root
authority certificates that your certificate was signed with. Most browsers
do.

Besides, if nothing else, it is relatively easy to validate if a certificate
belongs to the hostname being connected to, so it is easy to check that "www.url.com"
and "updates.url.com" are not the same hostname. You can obtain the server's
certificate after connecting and then read its Subject Alternative Name or
Subject CommonName fields. OpenSSL has X509_check_host() and X509_check_ip()
functions for those manual validations:

https://www.openssl.org/docs/manmaster/crypto/X509_check_host.html

OpenSSL does not perform certificate hostname validation by default, but
you can set it up (but not with Indy, unless you alter its source code).
Look at X509_VERIFY_PARAM_set1_host():

https://www.openssl.org/docs/manmaster/crypto/X509_VERIFY_PARAM_set1_host.html
https://wiki.openssl.org/index.php/Hostname_validation

Or SSL_set_tlsext_host_name().

What if the certificate that was included with our software expires?

Then you have to distribute a new certificate.

--
Remy Lebeau (TeamB)
Gennadiy Kiryuk...

Posts: 19
Registered: 10/25/02
Re: TIdHTTP with SSL certificate verification  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 30, 2015 10:37 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:

Besides, if nothing else, it is relatively easy to validate if a certificate
belongs to the hostname being connected to, so it is easy to check that "www.url.com"
and "updates.url.com" are not the same hostname. You can obtain the server's
certificate after connecting and then read its Subject Alternative Name or
Subject CommonName fields. OpenSSL has X509_check_host() and X509_check_ip()
functions for those manual validations:

Or SSL_set_tlsext_host_name().

Should I be able to verify the certificate from inside VerifyPeerEvent(TIdX509* Certificate, bool AOk, int ADepth, int AError) function?
Instead of just returning AOk I would do the additional verification myself, right?

What if the certificate that was included with our software expires?

Then you have to distribute a new certificate.

That means that for the software to be able to download files from our servers, it is guaranteed that I will need to periodically issue software updates to replace the expired certificates, right?
Just to be clear, we do not put restrictions on who can download the files. The user has an option to download a file from out website through "right-click and SaveAs...". We just need to make sure that when the software downloads the files automatically it gets them from the correct source.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TIdHTTP with SSL certificate verification  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 30, 2015 12:49 PM   in response to: Gennadiy Kiryuk... in response to: Gennadiy Kiryuk...
Gennadiy wrote:

Should I be able to verify the certificate from inside
VerifyPeerEvent(TIdX509* Certificate, bool AOk, int ADepth,
int AError) function?

Yes. OpenSSL performs some initial validations, and then calls the verify
callback to allow the app to perform additional validations and/or override
OpenSSL's initial validation result. The return value of the OnVerifyPeer
event is the final result that OpenSSL uses to accept/reject the certificate.

That means that for the software to be able to download files
from our servers, it is guaranteed that I will need to periodically
issue software updates to replace the expired certificates, right?

If you embed the certificate inside your executable itself, yes (and there
are ways to load certificates into OpenSSL from memory instead of file, though
Indy does not support those ways yet). If you load the certificate from
an external file, then you can simply issue updates to that file.

Just to be clear, we do not put restrictions on who can download
the files. The user has an option to download a file from out website
through "right-click and SaveAs...". We just need to make sure that
when the software downloads the files automatically it gets them
from the correct source.

Then you don't need any client-side certificates. Just validate the certificate
that is given to you in the OnVerifyPeer event.

--
Remy Lebeau (TeamB)
Gennadiy Kiryuk...

Posts: 19
Registered: 10/25/02
Re: TIdHTTP with SSL certificate verification  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 30, 2015 1:05 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:
Then you don't need any client-side certificates. Just validate the certificate
that is given to you in the OnVerifyPeer event.

Do I still need to have the RA certificate(s)?

I do not have any certificates or public keys in my software (my part of code).
When I run the code, VerifyPeerEvent function is called with AOk set to false indicating that it has already failed with error 20. Is that because I did not load/provide the RA certificate(s)?

If it helps to illustrate my problem, if I set AOk to true (return value) in debugger, the next call has error 27 with AOk = false. After setting it to true, the next call is with Error 21, AOK=false again. Setting AOK to true finally makes the code proceed with downloading.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TIdHTTP with SSL certificate verification  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 30, 2015 1:24 PM   in response to: Gennadiy Kiryuk... in response to: Gennadiy Kiryuk...
Gennadiy wrote:

Do I still need to have the RA certificate(s)?

Only if you want to verify that the server's certificate chain was signed
with your expected RA certificates and is not a forgery.

When I run the code, VerifyPeerEvent function is called with
AOk set to false indicating that it has already failed with
error 20. Is that because I did not load/provide the RA
certificate(s)?

Yes. You did not load any certificates into TIdIOHandlerSocketOpenSSL, so
it could not verify the server's certificate chain for you.

If it helps to illustrate my problem, if I set AOk to true (return
value) in debugger, the next call has error 27 with AOk = false.

Error 27 is X509_V_ERR_CERT_UNTRUSTED.

After setting it to true, the next call is with Error 21, AOK=false again.

Error 21 is X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE.

--
Remy Lebeau (TeamB)
Gennadiy Kiryuk...

Posts: 19
Registered: 10/25/02
Re: TIdHTTP with SSL certificate verification  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 30, 2015 2:04 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:

Only if you want to verify that the server's certificate chain was signed
with your expected RA certificates and is not a forgery.

"Forgery" in this case would also include a self-signed certificate, right?

If I just want to verify the host match, can I still do that from inside OnVerifyPeer event? I tried to inspect the Certificate parameter in debugger, but most of the fields were NULL (I think Subject was one of them). Does that parameter contain the certificate from our server?

Is it possible to get the RA certificates from the operating system and somehow load them into OpenSSL? How easy would that be? I looked into madExcept and they use winhttp.dall. I used their code to upload files to our server via HTTPS POS requests. winhttp.dll was able to verify certificate (if I remember correctly it was refusing a self-signed certificate). I guess it was getting RA certificates from the operating system, right? It would be nice if I could use those certificates too... or maybe use winhttp.dll instead.

I'd rather have the OS handle RA certificates and not worry about updating the software for the sake of preventing included certificates from expiring. If a user does not update their software for a year (and some do not), their automatic download would stop working because their update "expired". That would be a major inconvenience.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TIdHTTP with SSL certificate verification  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 30, 2015 3:19 PM   in response to: Gennadiy Kiryuk... in response to: Gennadiy Kiryuk...
Gennadiy wrote:

"Forgery" in this case would also include a self-signed certificate,
right?

Yes.

If I just want to verify the host match, can I still do that from
inside OnVerifyPeer event?

Yes, and I already explained how to do that.

I tried to inspect the Certificate parameter in debugger, but most of
the fields were NULL (I think Subject was one of them).

TIdX509 loads the individual certificate values on an as-needed basis when
your code reads the respective property values. The debugger would not trigger
that logic, unless you inspect the properties rather than the data fields
directly.

Does that parameter contain the certificate from our server?

Yes.

Is it possible to get the RA certificates from the operating system
and somehow load them into OpenSSL?

Yes, but not easily, and not with Indy. You would have to use Microsoft
APIs to export the relevant certificates from Microsoft's certificate store,
then use OpenSSL APIs to import them directly into OpenSSL's certificate
store.

I looked into madExcept and they use winhttp.dall. I used their code to
upload files to our server via HTTPS POS requests. winhttp.dll was able
to verify certificate (if I remember correctly it was refusing a
self-signed certificate).

winhttp.dll is a Microsoft system DLL, using Microsoft's certificate store.

I guess it was getting RA certificates from the operating system, right?

Yes.

It would be nice if I could use those certificates too...

You would have to ask in the OpenSSL community about that.

or maybe use winhttp.dll instead.

You can (just not with Indy). It is part of the WinHTTP API:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa384273.aspx

I'd rather have the OS handle RA certificates and not worry about
updating the software for the sake of preventing included certificates
from expiring.

Then you have to use OS services for your HTTP operations.

--
Remy Lebeau (TeamB)
Gennadiy Kiryuk...

Posts: 19
Registered: 10/25/02
Re: TIdHTTP with SSL certificate verification  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Nov 2, 2015 9:43 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy,

Thank you for your help. It is a deep topic and answers to a simple questions often cannot fit in one post or even a forum thread.
Using Indy/OpenSSL for server certificate verification requires more knowledge than I possess now. However, I was able to use winhttp.dll.

Here is my code I quickly threw together. It is merely a skeleton of what one would do.

#include <System.Classes.hpp>
 
#include <winhttp.h>
//---------------------------------------------------------------------------
#pragma package(smart_init)
 
#pragma comment(lib, "winhttp.lib")
 
int WinHTTP_Download(TStream * Stream, UnicodeString url)
{
	DWORD dwSize = 0;
	DWORD dwDownloaded = 0;
	LPSTR pszOutBuffer;
	BOOL  bResults = FALSE;
	UnicodeString server;
	UnicodeString path;
	HINTERNET  hSession = NULL,
			   hConnect = NULL,
			   hRequest = NULL;
	DWORD err;
	int i;
 
	i = url.Pos("/");
	server = url.SubString(1, i-1);
	path = url.SubString(i, url.Length());
 
	// Use WinHttpOpen to obtain a session handle.
	hSession = WinHttpOpen( L"WinHTTP Test/1.0",
                            WINHTTP_ACCESS_TYPE_DEFAULT_PROXY,
							WINHTTP_NO_PROXY_NAME,
                            WINHTTP_NO_PROXY_BYPASS, 0);
 
    // Specify an HTTP server.
	if (hSession)
	{
		hConnect = WinHttpConnect( hSession, server.c_str(),
								   INTERNET_DEFAULT_HTTPS_PORT, 0);
		// Create an HTTP request handle.
		if (hConnect)
		{
			hRequest = WinHttpOpenRequest( hConnect, L"GET", path.c_str(), //NULL,
										   NULL, WINHTTP_NO_REFERER,
										   WINHTTP_DEFAULT_ACCEPT_TYPES,
										   WINHTTP_FLAG_SECURE);
 
			// Send a request.
			if (hRequest)
			{
				bResults = WinHttpSendRequest( hRequest,
											   WINHTTP_NO_ADDITIONAL_HEADERS,
											   0, WINHTTP_NO_REQUEST_DATA, 0,
											   0, 0);
 
 
				// End the request.
				if (bResults)
				{
					bResults = WinHttpReceiveResponse( hRequest, NULL);
 
					// Keep checking for data until there is nothing left.
					if (bResults)
					{
						do
						{
							// Check for available data.
							dwSize = 0;
							if (!WinHttpQueryDataAvailable( hRequest, &dwSize))
							{
								//printf( "Error %u in WinHttpQueryDataAvailable.\n",
								//		GetLastError());
								bResults = false;
								break;
							}
 
							// No more available data.
							if (!dwSize)
								break;
 
							// Allocate space for the buffer.
							pszOutBuffer = new char[dwSize+1];
							if (!pszOutBuffer)
							{
								//printf("Out of memory\n");
								bResults = false;
								break;
							}
 
							// Read the Data.
							ZeroMemory(pszOutBuffer, dwSize+1);
 
							if (!WinHttpReadData( hRequest, (LPVOID)pszOutBuffer,
												  dwSize, &dwDownloaded))
							{
								bResults = false;
								//printf( "Error %u in WinHttpReadData.\n", GetLastError());
							}
							else
							{
								Stream->Write(pszOutBuffer, dwSize);
								//printf("%s", pszOutBuffer);
							}
 
							// Free the memory allocated to the buffer.
							delete [] pszOutBuffer;
 
							// This condition should never be reached since WinHttpQueryDataAvailable
							// reported that there are bits to read.
							if (!dwDownloaded)
								break;
 
						} while (dwSize > 0);
					}
					else
					{
						// Report any errors.
						//printf( "Error %d has occurred.\n", GetLastError() );
					}
				}
			}
		}
	}
 
	if (!bResults)
	{
		err = GetLastError();
	}
 
    // Close any open handles.
    if (hRequest) WinHttpCloseHandle(hRequest);
    if (hConnect) WinHttpCloseHandle(hConnect);
	if (hSession) WinHttpCloseHandle(hSession);
	return bResults;
}


This code was taken from MSDN and with a bit of Googling (added #pragma comment(lib, "winhttp.lib"); again it was you, Remy) tweaked to turn into a simple function.
Here is how I used it:

void __fastcall TForm1::Button1Click(TObject *Sender)
{
	TMemoryStream * mstream = new TMemoryStream();
	UnicodeString url = "www.example.com/path/to/my/document/";
 
	if (WinHTTP_Download(mstream, url))
	{
        mstream->Position = 0;
		Memo1->Lines->LoadFromStream(mstream);
	}
	else
	{
		Memo1->Lines->Add(L"Error while reading page");
	}
 
	delete mstream;
}
 


When I replaced "www." with "subdomain." (a valid subdomain on our website) the function returned false and err is ERROR_WINHTTP_SECURE_FAILURE.
So, it appears to check the certificate for you to some extent. I have not checked it on a self-signed certificate yet. But at least it is a good start as the function works downloads a file.

Edited by: Gennadiy Kiryukhin on Nov 2, 2015 9:44 AM
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02