Integrating Amazon’s WAF with opEvents monitoring automation

Integrating Amazon’s WAF with opEvents monitoring automation

At Opmantek, we use our own software heavily for monitoring our production and development systems, solving our own IT Operations challenges that we know our customers share, it also helps us to develop the products faster in real-world environments through early testing.

We have been using Amazon’s Web Application Firewall (WAF) to help protect our web-facing infrastructure. One of the issues with the out-of-the-box solution is how does one monitor the firewall’s logs as part of your overall IT operations and how do you perform analysis of those logs, with context, to the workloads they relate to.

Firstly to help check newly implemented rules are working as intended and secondly, to provide quick diagnoses in the event of attack.

We first tested a 3rd party product to help visualise the logs and hopefully provide out-of-the-box insights into the data, but we found the TCO of this solution was much higher than using the extensibility of the Opmantek products.  The results from the out-of-the-box solution would also have been isolated from the overall network health visibility.

Our WAF is set up with the rules sets provided by the AWS marketplace as well as internally developed custom rules sets with reputation / ip blacklists that are constantly evolving.

Our Architecture follows:

  • AWS Web Application Firewall (Layer 7 Firewall)
  • AWS Kinesis Delivery Stream (durable real-time data streaming service)
  • Opmantek Kinesis Log Service (Kinesis stream receiving transformation service)
  • opEvents ( Centralised logging and Event Management)

Our WAF is set up to send all logs through to our Kinesis Delivery Stream. 

AWS kinesis delivery stream is set up to deliver batched requests over HTTPS to a specified endpoint within your own environment. We developed a small HTTP service in GoLang to securely ingest the batched logs from AWS, and we provide this AWS ingestion service to customers on request.

Our GoLang service also remaps JSON keys before writing the file out to disk. 

To see what AWS publishes in its logs you can find them – here.

opEvents jsons_log service listens for filesystem changes, reads them, and runs the event through the opEvents Engine.

We have added a new property country which is the ISO country code of the request.

Element: is mapped to the Requestor’s  IpAddress.

Node: is the name of our WAF in AWS.

Description: is the WAF action, WAF rule which was triggered, ipAddress and country code. This gives opEvents unique enough data to create rolled up event counts for WAF actions. Through opEvents’ dashboard you can see a quick count of clients who have made the most POST requests, or a bot trying SQL injection against your site.

opEvents event action - 650

We are using opEvents to store metadata about the WAF log, headers, requesting IP, country and which WAF rules were terminated. Using the IP address we can quickly make an assumption about the requesting origin and know if we have bots scraping us from data centres or users acting unlawfully. With this quick drill down into the event data we can make quick operational changes to implement rules to stop certain traffic or add entire subnets to our IP blacklist.

opEvents event feed - 650

How we are using this information…

Debugging WAF rules

Implementing WAF rules can be challenging especially when you have to go back and look at access history. opEvents is storing 30 days of WAF logs which we can quickly filter to find the blocked request and debug the rule and make an exception or change how our application works for better security.

Detecting bots

Some crawlers generate quite a large amount of web traffic as they quickly scan our domains, being able to have an aggregate view of requests for an ip Address and rules being triggered it’s easy to find the block of address causing issues. We then drill down into the request metadata checking the headers, location, who owns the IP and past requests patterns. From this we can then quickly ban malicious bot IP ranges.

Website usage statistics 

With MongoDB backing opEvents it’s easy to write queries and run them through the mongo shell and aggregate usage data for more in depth reporting. Which Country Code uses this endpoint the most, Which user-agent makes the most requests.

Automated Response

Using Opmantek’s Operational Process Automation methodology when we correlate sets of WAF events we trigger automated actions into our AWS Environment to respond to the incidents and avoid issues. 

opEvents engine makes it flexible to ingest any type of structured data and we could quickly integrate into our production monitoring to give us greater insight into our public facing web systems.

If you would like to know more about about using opEvents and processing web firewall logs we offer live demos with our technical team here.

Getting Started with the Opmantek VM

Getting Started with the Opmantek VM

This page details the four simple steps you need to get started using the Opmantek Virtual Machine.

The virtual machine is a vendor neutral image (OVF) that can be converted and used with VMware (ESXi, Fusion, et al), Virtual Box, KVM and Hyper-V. It’s a quick and easy way to get up and running for Proof of Concepts, Trials and other scenarios where a full production install is not required. Download it, convert it and run it and you’re good to go.

Of course you may have requirements around configuring your Linux machines in a company standard way, and in that case our individual applications are also available to install on a machine built by you to your requirements. To download individual applications, you can here.

Download Icon

Step 1 – Download

Download the Opmantek Virtual Appliance from here.

Deploy Icon

Step 2 – Deploy

So you’ve downloaded the latest version of the Opmantek Virtual Appliance and now you want to set it up and take it for a test drive to see how NMIS and the Opmantek networking modules can help you administer your network. No problem. Easy. Simply:

  1. Import the OVA (Open Virtual Appliance Format) or OVF file into VMware, Virtualbox,…
  2. Start up the new VM, optionally set a static IP address.
  3. Access NMIS and the other Opmantek applications in your browser.

For more details on how to do this including specific hypervisor detail, follow the instructions for your hypervisor of choice below:

Configure Icon

Step 3 – Configure

Out of the box there is nothing to do – just start the virtual machine and connect to the application URL, done!

If you would like to configure the virtual machine to use HTTPS, set the hostname or set the IP options to your liking, see here – Getting Started, Configuring the Appliance

Run Icon

Step 4 – Run

Start the virtual machine and configure the application settings, see here – Getting Started, Configuring the Applications

Our individual applications are covered in great detail in their own wiki spaces (see below).

Getting Started, Configuring the Applications

Getting Started, Configuring the Applications

Introduction

Getting up and running is as simple as finding the virtual machines IP and calling the application page’s URL. After that you will want to enter your license (or get a free license) and we would suggest using the Wizard to kick start your journey.

If you need additional configuration over and above what the wizard offers, our applications are all extremely configurable. The in depth material is hosted on each applications individual space within our wiki. For links to those, see the bottom of this page.

If you need assistance, feel free to ask on  Opmantek’s Community Questions site or contact Opmantek Support.

If you have more in-depth or unique requirements Opmantek is ready and able to customise our solutions to meet your needs – just ask!

Accessing the Application Modules on the VM

Finding the IP and hostname of the VM

Log in to the console using username: root, password: NM1$88 (you’ll need console access if you don’t know the IP to SSH to).

The default credentials can be found here.

By default the VM boots with DHCP enabled. On the command line run the below to determine the IP.
ifconfig | grep inet | grep -v inet6 | grep -v "127\.0\.0\.1"
To determine the VMs hostname, run the below.
hostname

Calling a URL

The applications can be access by using the URL http://{ip from above}/omk

You will be presented with the following screen.

VM Guide 12 - 700

All the module names are clickable and link to the respective applications; but you can also navigate directly to NMIS, Open-AudIT or any of the other Opmantek applications.

Simply open your favorite web browser and navigate to:

  • NMIS8 Virtual Appliance:
    • http://<vm ip address>/nmis8 for NMIS,
    • http://<vm ip address>/open-audit for Open-AudIT Community,
    • or http://<vm ip address>/omk for the list of all Opmantek applications.
  • NMIS 9 Virtual Appliance:
    • http://<vm ip address>/nmis9 for NMIS,
    • or http://<vm ip address>/omk for the list of all Opmantek applications.

The default access credentials are username “nmis”, password “nm1888” (more info regarding credentials can be found on this page).

Once you’ve got either the NMIS dashboard or one of the Opmantek application dashboards open, you can also navigate between modules using the “NMIS Modules” and “Modules” menus, respectively.

Getting Started Wizard

On the default /omk page you will see a section in the top left titled “Simple Configuration Wizard”. We highly recommend you use this to easily configure items such as authentication and email (among others).

Config-Wizard - 700

Application Module Licensing

The first step to using a module is knowing how to open it, the next is to get a license for it. This process is simple and painless: all products on the appliance (except opFlow) now support free licenses (which are not time-limited but limited to 20 nodes). For these applications the initial dialog offers the generation of such a free license; for opFlow the licensing dialog offers guidance for obtaining a trial or a full license. It is a straightforward process to activate a license.

Simply click “Activate a Free License”:

Activate-Free-License - 700

And then fill out the details, note these details are used for your license key and recovery information, ensure they are accurate:

License-Information - 700
You can also use the “opLicensing” module (reachable at http://<vm ip address>/omk/opLicense or from the Modules menu) to manage your licenses comfortably (including the ability to download previously purchased licenses from the Opmantek website). More info about opLicensing can be found here.

Exploring and Configuring Application Modules

All modules on the Virtual Appliance come with their configuration set to very minimal but safe defaults; to explore the full functionality available you will very likely have to make config modifications to suit your environment. These configuration options are all documented on the Opmantek Community Wiki, in the per-application section (or “Application Space”); certain common, shared aspects are discussed in the “opCommon” space.

For all recent Opmantek applications you’ll find links to the respective sections of Wiki in the “Help” menu (usually under “Online Documentation”); there’s also an online version check on the welcome/landing page, and on each application’s “About” page which helps with keeping your modules up to date.

Our individual applications are covered in great detail in their own wiki spaces (see below).

Getting Started, Configuring the Appliance

Getting Started, Configuring the Appliance

Out of the box, the Opmantek VM requires no configuration changes to work, however you may wish to make changes so it complements your particular network environment. Items such as using HTTPS, setting the hostname or configuring the IP stack are all configurable.

Secure Access with https (optional)

The VM ships with support for secure https access enabled, and you can use https instead of http in any of the URLs mentioned.

However, the included certificate-key pair is the same for all VM downloads, and it is a self-signed certificate (for “opmantek.local”). This will cause your browser to display security warnings.

If you want to use https access operationally it is highly recommended that you replace this shared default key-certificate pair with your own (or use an automated certification authority like letsencrypt/ for example).

Debian – Customizing Network Settings (optional)

The Debian team has an excellent wiki page dedicated to customizing the network configuration; this page is available in several languages:

Debian Network Configuration

CentOS – Set a custom IP Address and DNS (optional)

At this point the VM has DHCP enabled, and therefore it will receive a dynamic IP address suitable for your environment. The easiest way to access the VM at that point is from the virtualization software terminal. So go ahead and boot the VM. After the VM is finished booting you will be welcomed by a login prompt. First, log in using the default credentials.

If it’s desirable to customize the network settings we offer the following options:

  • Option #1 – Manual configuration
    The VM appliance is a normal CentOS 7 system, hence all the mechanisms for network configuration in CentOS are available.
    Most specifically you’ll be interested in the  files:
    /etc/hosts

    /etc/sysconfig/network

    /etc/sysconfig/iptables

    /etc/sysconfig/network-scripts/<interface_name>

  • Option #2 – Use our example network configuration files
    Root’s home directory contains 2 example network configurations, one for static IP’s and one for DHCP.  Copy the one you require to /etc/sysconfig/network-scripts/ and edit it to suit:
    <code”>cp ifcfg-ens192.static /etc/sysconfig/network-scripts/ <interface_name></code”>

Be sure to verify the following settings:

  • IPADDR
  • NETMASK
  • BROADCAST
  • GATEWAY
  • NAME; the interface name as found with the ip address command
  • DEVICE; the interface name as found with the ip address command
  • DNS;

Next restart the network service for the setting to take affect.  In our testing the DHCP address was still present, and the static IP assigned as a secondary (this can be seen with the ‘ip address’ command).  The DHCP address can be removed by using the ‘ip address command del’ command or be rebooting the system.
TYPE=Ethernet
IPADDR=192.168.10.201
NETMASK=255.255.255.0
BROADCAST=192.168.10.255
GATEWAY=192.168.10.51
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE= yes
IPV4_FAILURE_FATAL=no
IPV6INIT= yes
IPV6_AUTOCONF= yes
IPV6_DEFROUTE= yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp0s3
UUID=9099afe7-6dd6-4aec-bd93-8950b67ab8f5
DEVICE=enp0s3
ONBOOT= yes
DNS1=192.168.10.80
DNS2=8.8.8.8