Using Instana in NodeRED to Trace MQTT Traffic

Instana is a recent addition to the IBM family and after an initial exploration of its awesome capabilities I wanted to dig deeper. Based on this I decided that I would like to look at how I could use Instana to trace flows running over the MQTT protocol. The reason for this is that I have a couple of solutions I am working on which use MQTT and could benefit from the sophisticated monitoring that Instana could bring.

So with the goal I then needed to find a way to create a test set up and given its 1st class MQTT support and ease / speed of use I decided to use NodeRED. Finally I would run this in an OpenShift cluster where I have the Instana agent installed.

Now before I get into the detail a little bit of information on how Instana operates. Firstly out of the box it will automatically instrument a wide range of workloads. When this is done Instana will create a “span” to track the flow of a request when it is detected as being recieved by the workload. So in the case of NodeRED (which is a NodeJS app so automatically instrumented) when an HTTP request is received by a flow, Instana can track that flow via a “span”. Now if the flow then calls MQTT we need to use the Instana SDK to expand the scope of the span to cover the flow over the MQTT protocol. In additon where a flow is initiated by and internal event e.g. a periodically injected event, Instana does not see a received request and in this case the SDK is used the start the span manually.

Hopefully that makes senses and you are still with me :-)

So what this means is that in addition to the automatic instrumentation I needed to enable NodeRED to access the Instana SDK.

As my deployment target is OpenShift I needed to build a Docker container to host NodeRED. On top of this I needed to do 3 things:

1. Update package.json to import Instana
2. Update red.js to initialise Instana
3. Update settings.js to expose Instana as a global

Lets start with package.json my updated version looks as follows:

package.json update

The highlighted line shows the entry I added to import Instana. Next I updated red.js to initialise Instana as shown below:

red.js update

Here you can see that I’ve inserted a “require” to bring in Instana. Finally I updated settings.js to expose a global to allow access to the Instana SDK.

settings.js update

As you can see I’ve added Instana to the global context under the name instana.

With the changes done I used the following Docker file to build a new image.

Docker file

With this all in place I built the image, pushed it to my IBM Registry and used it to create two NodeRED deployments in OpenShift. One depoloyment to act as a “Client” and the other to act as “Server” so that I could test tracking a flow between the two. In addition I already have a Message Gateway deployment running in another OpenShift cluster which I decided to use as my MQTT broker. So my architure looked like:

Architecture

With the NodeRED environments up and running I created flows in the Client and Server instances to do what I needed. The client instance was:

Client NodeRED flow

Here you can see that I have 2 inject nodes which are set to inject messages into a topic for Andy and a topic for Tony. The Andy publish is set to trigger every 3 seconds and the Tony one every 4 seconds. In addition I set up a REST endpoint to allow me to trace based on an externally received event (remember Instana automatically creates a span here).

Now lets look at the key function nodes that handle the calls to the Instana SDK… The “Instana Create Span” for the injected publish calls looks like:

Create Span

In the flow line 1 gets access to the Instana SDK via the global. Next line 3 defines a set of “tags” to be applied to the span. These are used by Instana to allow filtering for reporting and analysis. I am using the “message_bus” type of span and in here I set a number of static elements but also have a dynamic one where I insert the topic being published to. This means that I can break down calls by topic. Then in line 16 I create an Entry span and set the tags on this span. Following creating the span I use the SDK to pull out the trace Id and span Id as can be seen in lines 17–19.

At this stage I have created the entry span which is automatically generated by Instana when it detects an inbound request to the flow. Next in line 21 I create an exit span to wrapper my outbound publish to MQTT. When I create this I set the trace and span id’s to be those generated by the entry span. Finally I store the span and trace id’s in the payload to be published. This is because I need the id’s on the server side in order to reconnect to the span.

Once the function node completes I publish the data to MQTT and move to the “Instana Complete Span” function node. This node looks like:

Complete Span

In here I again get access to the Instana SDK and then I complete the exit and entry spans.

In the case of the REST flow the only difference is that I don’t need to create an entry span and only need to retrieve the trace and span id’s from the create span.

Now lets look at the server flow.

Server NodeRED flow

In this flow the MQTT listener is listening on all topics and when it receives a message it is converted to JSON and is passed to the “Instana Start Span” function. In the case of the server flow this is slightly different as I need to hook in the passed trace and span id’s.

Server Create Span

This flow is very similar to the client side one but you can see that on line 15 I am slotting in the passed trace and span id’s. The lines 20–24 are used for me to check that the id’s are set and line 26–29 are used to set the HTTP headers so that any down-stream calls can also be picked up if Instana is monitoring.

Once the span has been set up I make a call to a dummy REST endpoint (which is not monitored by Instana) and then I flow to the “Instana Stop Span” which is the same as the client one.

So that was what I needed to do but what does it look like in Instana :-)

I set up a Application to map to my MQTT Test app and the dashboard for this looks like.

Application Dashboard

If I drill into the calls I can see the detail of the flow.

Flow Details

In addition as I provide tags about the topic that is being published to I can use Instana’s Analytics view to allow me to look at the flows in that level of detail.

Calls By Topic

So thats it, I have managed to track my calls as they flow over MQTT. Now I am sure this isn’t 100% water tight but its certainly a good starting point. I am planning to experiment with some more areas so there may be more posts to follow.