Securing MQTT Traffic with Cloud Internet Services

Recently myself and a colleague of mine in IBM Client Engineering (Manjer Shaikh) had the opportunity to explore the potential for securing MQTT traffic using the Cloud Internet Services (CIS) capability within the IBM Cloud. The focus was on MQTT traffic being passed from a Web Client application to an IBM Message Gateway service running in OpenShift hosted in a Virtual Private Cloud (VPC) running in the IBM Cloud. The following summarises the architecture we were looking to apply CIS to.

Basic Architecture

With CIS applied things would look like.

Architecture with CIS

Before I dive into the details of what we did let me provide an overview of what CIS will do.

CIS Structure

CIS works by assuming control of a registered domain (1) and then manages the flow of traffic to and from that domain. To route traffic to a target service / website an origin pool (2) is created. This defines the endpoints that are available to fulfil a request. Finally Global Load Balancers (3) provide the linkage between requested URI’s and the origin pool targets that can fulfil them.

So lets dig into the configuration of each of these areas.

CIS Configuration

IBM Cloud Internet Services is a simple set of edge network services for customers looking to secure their internet-facing applications from DDoS attacks, data theft and bot attacks, as well as for those customers needing to optimize their web applications or ensure global responsiveness and the ongoing availability of their internet-facing applications. The Cloud Internet Services includes Domain Name Service (DNS), global load balancer (GLB), distributed denial-of-service (DDoS) protection, web application firewall (WAF), Transport Layer Security (TLS), and caching to bring market-leading security and performance to your internet applications.

We followed the following steps to configure the CIS with a domain name. In this case we used the GoDaddy domain registrar, but it may be different from a case to another, depending on which registrar is being used.

1. Deployed the CIS instance by using the IBM Cloud console or API.

Recommendation: Mirror the name of the CIS instance with the domain.

2. Clicked Let’s get started.

3. To set up the domain navigated to Overview page on left hand side menu

• Click on Add Domain, if setting up the first domain in the CIS or
• On the upper right corner click on three dots to popup option (a) new domain and (b) delete, if a domain is already configured.

4. Click of new domain

5. Entered the domain in the Domain name field and clicked Connect and continue.

6. Clicked Next Step for new domain but for existing domains click Import records.

7. Made note of the New NS records for entry into the GoDaddy DNS Management system.

8. Logged in to GoDaddy domain Registrar.

9. Navigated to the DNS Management page for the candidate CIS domain. IBM Cloud must be updated through the API.

10. Clicked ADD to create a NS Host record for each Name Server entry provided from CIS

11. Setup a DNS record with the supplied CIS NS Records

• Type Nameserver
• Host domain (this is name of the domain, for example, example.com
• Point to ns004.name.cloud.ibm.com
• TTL ½ hour

12. DNS Management page now reflected two additional NS records for domain,

13. Navigated back to the IBM Cloud Internet Services instance,

14. After the NS Records have DNS propagated, it is reflected in the DNS tools.

15. With completed NS propagation clicked Check name servers.

16. CIS begins to check NS configuration for the domain and might be in a Pending State.

Caution: Any typos in this section may produce a “silent” failure. If the domain is in the pending state for more than 30 minutes, there might be an error.

Upon successful domain Name Server / NS record confirmation, the CIS instance reports Active. In the final step we had delegated the domain management of the domain to CIS by changing the Name Server records of the domain.

Once domain setup is completed then the overview page would load, with information on plan, DDoS protection, account information, etc.

Please note the status against the domain name in the overview

1. Active: It indicates that the domain is set up correctly and CIS is configured successfully.
2. Inactive: It indicates that the CIS could not be successfully configured for the domain.

We are now ready to start configuring the service and routing traffic to our services.

Ok so we now have a domain linked in so lets look at how we defined access to actual “services”. In CIS Origin pools define a target endpoint where “service” is going to be provided from. In our case this would be the OpenShift cluster running in the IBM Cloud.

Below you can see that we created two Origin pools, this was to allow me to test the health check feature. In this case I’ve applied health check to the MGW-Cluster-Ingress but not mqtt-test-pool.

Origin pools

When we created the origin pool the origin address was set to the public Ingress for the OpenShift Cluster. The following shows how we set up the mqtt-test-pool which didn't have a health check…

Origin pool details

and here we see the MGW-Cluster-Ingress pool with a health check configured.

Origin pool with health check

A Health check defines the target URL, protocol and port to call in the “Origin” pool.

Health check

The “Advanced Options” allow the health check to be tuned e.g. test interval etc. Finally for the health check ping to flow into OpenShift the ”Host” header needs to be set. I will cover the OpenShit set up we had later in this post.

Although not part of this the work we did, it is possible to set up ACL’s on the IBM Cloud VPC side to restrict the IP addresses which are allowed to access the public Ingress to just those provided by CIS. Doing this ensures the CIS layer can not be bypassed.

So now we had the front and the back ends defined we needed to “wire” the two together and this was done by using Global Load Balancers. Global Load Balancers (GBL’s) provide hostnames which are then mapped to origin pools to call the actual services.

For this our work we set up 3 GLB’s

1. Message Gateway Admin UI — This was only to test CIS set up and not needed for the actual use case testing

2. NodeRED environment where our web apps were hosted

3. MQTT traffic

Global Load Balancers

Lets look at how we defined each.

MQTT GLB

NodeRED GLB

Message Gateway UI GLB

OpenShift Configuration

With the CIS networking defined we needed to look at our OpenShift configuration. To support traffic flowing from CIS we needed to set up routes in OpenShift to match the URL’s that would be presented from CIS.

Updated routes

Here is an expanded via of the route we created to allow MQTT traffic to flow from CIS. The key change is the “Host” and you can see that we set this to refer to the domain that CIS would be managing.

Expanded Route

CIS Security

With all the connectivity set up we next turned our focus to the security configurations within CIS.

The first thing we did was to configure DDoS protection. DDoS is enabled by default in CIS when proxies are enabled, but we can further configure Layer 7 security by:

1. Configuring proxies
2. Configuring WAF ruleset sensitivity and response behaviour
3. Adding rate limiting
4. Adding firewall rules

With these we can customize Layer 7 mitigation of both volumetric and non-volumetric attacks.

Lets start with enabling the GLB proxies. You must enable proxies on the Domain Name Servers (DNS) and Global Load Balancers (GLB) to enable DDoS protection. Proxies control whether traffic should flow through the security and performance functions on CIS. If this is set ‘Off’ it bypasses these systems.

To enable GLB proxies.

1. Go to Reliability tab on the left side menu.
2. Go to Global load balancers tab on the top-center.
3. Move the slider switch of each load balancers to enable proxies for them.

Enabling GLB proxies

Next we looked at configuring the Web Application Firewall within CIS. The WAF would not have any control over the MQTT traffic as its not HTTP but would protect the web app and we wanted to check the WAF didn't adversely impact the MQTT traffic flow. The CIS WAF contains rulesets to mitigate non-volumetric attacks, including cross-site forgery, cross-site-scripting (XSS), file inclusion, and SQL injection .To configure WAF with CIS rulesets…

1. Go to Security on the left side menu.
2. Go to WAF tab on the top.
3. Move the slider switch under Web Application Firewall to enable WAF for the domain.
4. Select CIS Ruleset tab.
5. You may enable one or more applicable group by moving its slider to the right, which would configure the rule in the ‘default mode’.
6. Default mode leaves individual ruleset in CIS pre-set mode.
7. You may change the mode by clicking on a mode, which would allow you to choose from one of the five modes.

CIS WAF Rulesets

CIS allows WAF configuration with either or both CIS and OWASP rules. To configure WAF with OWASP rulesets

1. Select OWASP Ruleset tab.
2. You may enable one or more group by moving its slider to the right, which would configure all individual rules under the group name.
3. You can expand the group name to view and configure individual rulesets if you do not want to enable all individual rules of a group name.
4. Finally configure ‘sensitivity’ and ‘action’ to enable the OWASP ruleset.
5. Select from 4 sensitivity option and 3 actions to configure the OWASP WAF

CIS recommends to always keep the WAF configured and enabled and for our explorations we had configured WAF with complete OWSAP rulesets and CIS rulesets for WordPress.

OSWASP Ruleset

We then looked at the CIS facilitates blocking of traffic via -

1. IP Access Rules — Recommended for blocking multiple IP addresses, /16 or /24 IP ranges, or Autonomous System Numbers (ASNs).
2. Firewall Rules — Recommended for blocking a country, any valid IP range, or more complex attack patterns.

We couldn’t use IP rules as the IP ranges would be non determined as the use case we were exploring was one where any browser could access the Web App and hence connect via MQTT. So we focused on Firewall Rules.

Firewall rules have limits but are more flexible and allow matching upon a wider variety of fields and expressions than IP Access Rules. To configure firewalls rules

1. Navigate to Security then Firewall Rules.
2. Click Create Firewall Rule.
3. Enter a rule name and optional description.
4. Optionally, input a priority, if necessary. If a priority is zero then it is evaluated last.

Use the UI builder in the Incoming requests section to add a condition. To build an expression with multiple conditions, click either:

And — to evaluate conditions using and logic

Or — to evaluate conditions or groups of previously and’ed conditions using or logic

Pick an action from the Response list menu.

Firewall configuration

An additional layer CIS provides the ability to deploy “functions” at the edge of the network to control access. We created a simple “function” to check the headers. If we saw a valid set of headers the connection was allowed, if not it was rejected with a 403 status code.

Edge Functions

The “Edge Function” is enabled via a “Trigger”. We created a “tigger” to apply the ”edge function” to the MQTT traffic route.

Edge Function Trigger

Finally, CIS provides special controls, i.e., ‘Defense Mode’ to protect your assets when it is already under a DDoS attack. To enable Defense Mode

1. Navigate to overview.
2. Locate service modes.
3. Enable defence mode by moving corresponding slider to the right.
4. Click on confirm in the pop up to enable the defense mode

Once defense mode is enabled, then it will show on the top of the page ’defense mode is active’. Defense Mode is intended to help protect against existing or predicted attack and prevents all traffic from reaching your origin server. Defense Mode is also configurable for specific URLs via CIS Page Rules.

Testing

With everything in place we ran the following test to explore the behaviour of the configured environment.

a) Load generated direct to message gateway

Connect directly to Message Gateway using a NodeJS program which presents a REST API to invoke an MQTT flow. Generate load by using Locust on local machine.

Goal:

Attempt to generate level of load to trigger action in CIS also review how Message Gateway handles load.

b) Load generated using web app in protected domain

Using a combination of a “spawning” application and the “Tester” application initiate a large number of MQTT connections using locally running web browser. All calls initiated within the scope of the protected domain.

Goal:

Mirror the use case where people are connecting to the application via browsers and attempt to generate load to trigger detection of attack in CIS.

c) Load generated using web app outside protected domain

Same b) as previous test but executed outside of the protected domain.

Goal:

Test ability to block MQTT traffic which is not originated from the protected domain.

Load generated direct to message gateway

In this use case we created a small NodeJS Express app and hosted it in another OpenShift Cluster running on the IBM Cloud.

A route was created to this application and then we used Locust to generate the load. For each call made to the route a message “Hello World!” is returned but also a connection is made to Message Gateway and the following actions performed

1. Subscribe to a topic
2. Publish message to the topic
3. On receipt of message close connection

Test overview

We ran the load with the edge function disabled and then enabled. Using the Message Gateway WebUI we could track the effect on the Message Gateway.

Message Gateway WebUI

We could see significant load hitting the Message Gateway server but we could not generate the level or type of load which CIS determined to be a DDoS attack. Enabling the “Edge Function” to restrict traffic over the MQTT route successfully blocked traffic being generated by a run.

Load generated using web app in protected domain

In this use case we created a single page web application “looper” which we hosted in NodeRED. When loaded into the browser “looper” allowed the user to specify a number of “tabs” to create each one loading and executing the “tester” page. The “tester” page initiated the MQTT traffic to Message Gateway.

All pages in this use case were loaded from the “testinterface.co.uk” domain paths.

Application Load Test

Again we could see significant load hitting the Message Gateway server but we could not generate the level or type of load which CIS determined to be a DDoS attack.

Message Gateway Load

Load generated using web app outside protected domain

This use case mirrored the previous one but this time the page loads were targeted directly to the NodeRED environment and NOT via “testinterface.co.uk” domain paths.

Direct connection load test

Again we could see significant load hitting the Message Gateway server but we could not generate the level or type of load which CIS determined to be a DDoS attack.

Direct connection load

Conclusions

Although we were unable to generate a level of load that would be detected as a DDoS attack by CIS we have shown that it is able to support and protect MQTT traffic flowing over Secure Web Sockets

The following were the key learning points

1. Its hard to generate a DDoS attack against MQTT
2. Edge Functions provide an approach for restricting access to MQTT “routes”
3. “Out of the box” CIS provides a comprehensive web protection layer