The beginnings of DevSecOps for NodeRED

In the begining…

Since completing our work on running NodeRED on OpenShift (see here), Mark Taylor and I have continued to look at what else we can do with NodeRED. One of the threads that Mark progressed was looking at how we can improve our deployment / management process to align more with a DevSecOps approach and between us we have been testing this out.

NodeRED configuration overview

Setting up NodeRED Flow Editor environment

The first step is to create the Flow Editor environment as this will initialise the git repository such that we can then augment it to allow deployments to be driven off of that repository. The Flow Editors role is to support access to the NodeRED flow editor interface so Flows can be created and stored. On top of this it will use the Projects feature which allows flows to be stored in a GitHub repository. Once stored in Git, these NodeRED flows can be managed as part of a CI/CD pipeline.

FROM nodered/node-red
COPY settings.js node_modules/node-red/settings.js
COPY package.json .
USER rootRUN apk add --no-cache inotify-toolsCOPY ssh-sidecar/ssh-manage.sh /tmp
RUN chmod a+rwx /tmp/ssh-manage.sh
COPY ssh-sidecar/ssh-key-watch.sh /tmp
RUN chmod a+rwx /tmp/ssh-key-watch.sh
RUN chgrp 0 node_modules/node-red/settings.js && \
mkdir /data/.ssh && \
chgrp -R 0 /data && \
chmod -R g+rwX /data
USER 1001
module.exports = {
uiPort: process.env.PORT || 1880,
mqttReconnectTime: 15000,
serialReconnectTime: 15000,
debugMaxLength: 1000,
flowFile: 'flows.json',
adminAuth: {
type: "credentials",
users: [{
username: "mark",
password: "$2b$08$sv6U21t.VGS71wrXLo.SrO6tpiFexGhWqmPUZUgd5CKz93bwTS9N6",
permissions: "*"
},
{
username: "admin",
password: "$2b$08$w3b03WJJMGCig8G6Q0GhmOgSluy3zi./D6ZR.yEXrt6AgQA1paq3W",
permissions: "*"
}]
},
functionGlobalContext: {
},
exportGlobalContextKeys: false,
logging: {
console: {
level: "info",
metrics: false,
audit: false
}
},
// Customising the editor
editorTheme: {
page: {
title: "Node-RED Flow Editor"
},
header: {
title: "Node-RED Flow Editor"
},
projects: {
enabled: true
}
}
}
{
"name": "node-red-docker",
"version": "1.2.0",
"description": "Low-code programming for event-driven applications",
"homepage": "http://nodered.org",
"license": "Apache-2.0",
"repository": {
"type": "git",
"url": "https://github.com/node-red/node-red-docker.git"
},
"main": "node_modules/node-red/red/red.js",
"scripts": {
"start": "/tmp/ssh-manage.sh",
"debug": "node --inspect=0.0.0.0:9229 $NODE_OPTIONS node_modules/node-red/red.js $FLOWS",
"debug_brk": "node --inspect=0.0.0.0:9229 --inspect-brk $NODE_OPTIONS node_modules/node-red/red.js $FLOWS"
},
"contributors": [{
"name": "Dave Conway-Jones"
},
{
"name": "Nick O'Leary"
},
{
"name": "James Thomas"
},
{
"name": "Raymond Mouthaan"
}],
"dependencies": {
"node-red": "1.2.0"
},
"engines": {
"node": ">=10"
}
}
#### Define NodeRED Credentials key
CREDENTIALS=tonykey
#### Create a project for the Flow Editor
oc new-project floweditor \
--display-name="Flow Editor" \
--description="NodeRED Flow Editor environment"
#### Create the secret that will allow the builder to access github
oc create secret generic nodered-floweditor-repo-at-github \
--from-file=ssh-privatekey=ocp-access \
--type=kubernetes.io/ssh-auth
oc secrets link builder nodered-floweditor-repo-at-github
#### Build the new app
oc new-app git@us-south.git.cloud.ibm.com:tony_hickman/nodered.git \
--source-secret nodered-floweditor-repo-at-github \
--env=NODE_RED_CREDENTIAL_SECRET=${CREDENTIALS}\
--name floweditor
#### Create an https route to the app
oc create route edge floweditor --service=floweditor
ssh-agent bash -c 'ssh-add ocp-access; /
oc new-app git@us-south.git.cloud.ibm.com:tony_hickman/nodered.git \
--source-secret nodered-floweditor-repo-at-github \
--env=NODE_RED_CREDENTIAL_SECRET=${CREDENTIALS}\
--name floweditor'
oc patch dc/floweditor --patch '{"spec":{"strategy":{"type":"Recreate"}}}'
oc set volume dc/floweditor --add --name=floweditor-volume-1 \
--type=pvc --claim-size=512M \
--claim-class=ibmc-vpc-block-general-purpose \
--overwrite
oc set volume dc/floweditor --add --name=floweditor-ssh \
--type=pvc --claim-size=12M \
--claim-class=ibmc-vpc-block-general-purpose \
--mount-path='/.ssh'
#!/bin/bash
TARGET=/data/projects
PROCESSED=/data/.ssh
# Start NodeRed
node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data" &
# Wait for NodeRED tos start
sleep 10
# Start the key create watche script in the back ground
/tmp/ssh-key-watch.sh &
# Set up watch on projects and create sub watches if project created
inotifywait -m -e create --format "%f" $TARGET \
| while read FILENAME
do
# Start watching the new project
echo "Start wait on "$FILENAME
PROJECT=$TARGET"/"$FILENAME
inotifywait -m -e create --format "%f" $PROJECT \
| while read FILENAME
do
case "$FILENAME" in
*git)
# Check for a git directory and watch it
echo "got git ->"$FILENAME
inotifywait -m -e modify --format "%f" $PROJECT"/"$FILENAME \
| while read FILENAME
do
case "$FILENAME" in
*config.*)
# Process config change
echo "got git config ->"$FILENAME
# Back up known_hosts file
cp /data/.ssh/known_hosts /data/.ssh/known_hosts.old
# Use keyscan to populate know_hosts
ssh-keyscan $(cat $PROJECT/.git/config | grep 'url =' | cut -f 2 -d '@' | cut -f 1 -d ':') >> /data/.ssh/known_hosts
# Remove duplicate entries
sort /data/.ssh/known_hosts | uniq > /data/.ssh/known_hosts.new
# Copy in new known_hosts file
cp /data/.ssh/known_hosts.new /data/.ssh/known_hosts
;;
esac
done
;;
esac
done
done
#!/bin/bash
TARGET=/data/projects/.sshkeys
PROCESSED=/data/.ssh
# Wait for files to be created in .sshkeys
inotifywait -m -e create --format "%f" $TARGET \
| while read FILENAME
do
# Process the new key
echo "copy "$FILENAME
# copy to .ssh
cp $TARGET/$FILENAME $PROCESSED
case "$FILENAME" in
*.pub)
# Use .pub version to get name and add to ssh config
echo "IdentityFile /data/.ssh/"$(echo $FILENAME | cut -f 1 -d '.') >> $PROCESSED/config
;;
esac
done
Access Create New Project
Create Project
Provide user details
Project details
Project files
Project created
Project created
Project settings
Add remote
Access settings
Add key
New key
Github repository settings
Add new key
New deploy key

Creating a git managed flow

So with the Flow Editor up and running lets set about creating a Flow. We need to get a flow managed under git in order to move on to the next step of deploying a runtime instance.

Add change
Commit changes
Manage remote branch
Access remote branch
Create new branch
Push local changes to git
Updated commit history
Committed files

NodeRED runtime deployment

There are a number of steps to take the ‘code’ committed in the GitHub repository during the last step and create a running application in OpenShift.

module.exports = {
// The file containing the flows. If not set, it defaults to
flowFile: 'flows.json',
// By default, credentials are encrypted in storage using a generated key. To
// specify your own secret, set the following property.
credentialSecret: process.env.NODE_RED_CREDENTIAL_SECRET,
// Securing Node-RED
// -----------------
// To password protect the Node-RED editor and admin API, the following
// property can be used. See http://nodered.org/docs/security.html for details.
adminAuth: {
type: "credentials",
users: [{
username: "admin",
password: "$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.",
permissions: "*"
}]
},
// The following property can be used to seed Global Context with predefined
// values. This allows extra node modules to be made available with the
// Function node.
// For example,
// functionGlobalContext: { os:require('os') }
// can be accessed in a function block as:
// global.get("os")
functionGlobalContext: {
os:require('os')
},
// Customising the editor
editorTheme: {
page:{
title: "NodeRED Runtime"
},
header: {
title: "NodeRED Runtime"
},
projects: {
// To enable the Projects feature, set this value to true
enabled: false
}
}
}
FROM nodered/node-red# Copy package.json to the WORKDIR so npm builds all
# of your added nodes modules for Node-RED
COPY package.json .
RUN npm install — unsafe-perm — no-update-notifier — no-fund — only=production
# Copy _your_ Node-RED project files into place
# NOTE: This will only work if you DO NOT later mount /data as an external volume.
# If you need to use an external volume for persistence then
# copy your settings and flows files to that volume instead.
#### We are going to create an ephemeral /data2 and change the ENTRYPOINT to use /data2USER root
RUN mkdir /data2 && \
chown -R node-red:root /data2 && \
chmod -R g+rwX /data2
USER node-red
COPY settings.js /data2/settings.js
COPY flow_cred.json /data2/flows_cred.json
COPY flow.json /data2/flows.json
ENTRYPOINT [“npm”, “start”, “ — cache”, “/data2/.npm”, “ — “, “ — userDir”, “/data2”]
Updated git repository
  1. Setup more SSH keys. To access the GitHub repositories we need to define another set of SSH keys. We define the private key to OpenShift and create a Deploy key in github using the public key. We could have copied the SSH keys from the file system of the flow editor but as the Deploy keys give write access it is better to define another set with just read access to the repository.
    Create the SSH key using
    ssh-keygen -C "nodered-processing/repo@github" -f nodered-github -N ''
    Register the repository SSH key with your private repository on GitHub, go to the Settings for the repository. On GitHub the repository SSH key is referred to by the term Deploy key. Search down the settings page and find the Deploy keys section and select it. Click on the Add deploy key button. In this section, give the key a name and paste in the contents of the public key file from the SSH key pair. This is the file with the .pub extension.
    The next step is to create a secret in OpenShift to hold the private key of the SSH key pair. When using the command line, to create the secret run
    $ oc create secret generic nodered-github \
    --from-file=ssh-privatekey=nodered-github \
    --type=kubernetes.io/ssh-auth
  2. Create the application:
    oc new-app git@us-south.git.cloud.ibm.com:tony_hickman/ace-nodered.git \
    --source-secret nodered-github \
    --env=NODE_RED_CREDENTIAL_SECRET="<some secret you remembered to make a note of>" \
    --name nodered-processing
    Check the progress of the build
    oc logs -f bc/nodered-processing
    If the builds completes successfully verify there is one pod running
    oc get pods
    You should see that a build pod has complete, a deploy pod has completed, and that there is now an application pod running:
    NAME READY STATUS RESTARTS AGE
    nodered-processing-1-build 0/1 Completed 0 72s
    nodered-processing-1-cbp4j 1/1 Running 0 9s
    nodered-processing-1-deploy 0/1 Completed 0 12s

Conclusion

After all this where have I got to… Well I have an approach which does allow me to bring support a DevSecOps style approach on top of NodeRED. There is room for improvement but its working. Going forward I want to look at creating a “side-car” to handle the SSH management as I think that will be cleaner. If I get this working I will create another post.

I‘ve worked for IBM all of my career and am an avid technologist who is keen to get his hands dirty. My role affords me this opportunity and I share what I can