Are all your devices scattered, with data floating everywhere? Here’s how to bring them together.

BUILD TIME: 2 Hours
DIFFICULTY RATING: BEGINNER

If the concept of “big data” has taught us anything, it’s that data is only as useful as what you can do with it. Aggregating statistics, behaviours, and other metrics about people or things is pointless unless you can make use of the resulting data in a meaningful way. Data companies are acutely aware that the masses of data they hold is only useful or valuable if they can provide insights, rather than merely terrabytes of random data with no practical purpose.

In some ways, we have a similar problem with Internet of Things (IoT) devices. We can data log, we can Tweet it, we can complete all sorts of fragmented actions very easily; but what’s more important is bringing the data together to create meaningful actions as a result. In this project, we’ll use a Raspberry Pi and Node-RED to achieve this.

THE BROAD OVERVIEW

Node-RED is a powerful development tool that can be used not only for bringing hardware devices together, but also for creating a single manageable dashboard for APIs and other online services.

Node-RED provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to an IoT device in a single-click. However, it isn’t specifically limited to IoT devices. It can monitor APIs and virtually any type of web-based feed for data, as well as take action. This makes it a great interface to sit as a central monitoring point for your IoT devices, whatever they’re out there doing.

HOW IT WORKS

Node-RED, as the name suggests, is built on NodeJS; a powerful server technology built on the Google V8 JavaScript engine. If you’re not familiar with NodeJS yet, you may like to read some of our previously published articles, such as “Secret Code - Node Zero”, or our projects including the “Raspberry Pi Slack Cam”, and “Network Video Event Recorder".

For this particular build, we’re assuming everything is within your own network and there are no firewalls to deal with. In a future project, we're going to cover running Node-RED on a web server on the broader internet.

For many applications, NodeJS is far advanced. This is not limited to, but definitely includes IoT. NodeJS pulls away from the traditional “request / reply” model that dominated the web server approach for many years. Now, interactions between client and server are persistent, rapid, frequent, and anything but idle. Even “regular” websites are now using web sockets, push notifications, and many other technologies that are far from idle.

Node-RED accommodates this easily, allowing flow-based programming and handling of events, whatever they may be. When running on a standard computer, you can dashboard, aggregate, and talk to other APIs. When using Node-RED on a Raspberry Pi (or other hardware-enabled computer), the power of Node-RED is truly unleashed. With full GPIO control, you can rapidly create physical actions in response to all sorts of inputs, whether they’re software or hardware in origin or destination!

PROS AND CONS

When you’re designing any type of centralised monitoring system, it’s important to recognise the potential downfalls of each method. Depending on the critical (or lack thereof) nature of what you’re doing, you can take a different approach. The data comes free (or near enough), but the redundancy of hardware can scale costs fairly quickly.

Note: for simplicity, in the topologies below, we’re going to call any data source - whether it’s an API, IoT device, or other endpoint - a “data node”. When referring to Node-RED running on the Raspberry Pi, we’ll simply refer to it as “Node-RED”.

Topology

There are several ways to design topology (that is, the way things connect to each other). With greater redundancy, comes increased reliability, but also increased cost. It's important to consider what level of redundancy you require for your application, if any at all.

Standard Topology (No Network)

While our focus for this project is on dashboarding multiple sensors using distributed sensors and data feeds, it’s entirely possible and practical to use Node-RED entirely for in-unit applications. That is, for applications where sensors and events are happening right in the Raspberry Pi itself. Although it’s not really part of our current project, it is indeed the simplest way to use Node-RED. Let’s face it, you probably want to do a lot more, and expand beyond the confines of your own Raspberry Pi. However since there are MANY single-board projects that involve data collection and display, without any external requirements, it was prudent to include it here.

There is not a great deal of difference between the setup we’ll explain, which uses external sensors, compared to using internal sensors only.

Network Node-RED Topology

The simplest connections (i.e. when using things not physically attached to the Raspberry Pi), will simply use standard IP networking such as WiFi or wired ethernet connections. This easily allows each IoT device, API feed, or whatever data source, to stream in using standard technologies.

One of the caveats of this solution however, is if your Raspberry Pi crashes; in this instance your IoT devices could be gathering data that you’re not listening to! If you’re simply measuring the moisture temperature in your window garden at home, then this probably isn’t much of an issue. However something more critical like a cooling pump for a nuclear reactor (we joke!), or a device monitoring uptime of the DIYODE website (very serious), could probably use a little more redundancy. This standard topology will be fairly robust, providing your network itself is robust.

Central Redundant Topology

Of course, one major potential problem with having all that data going through a centralised point is the risk of failure. Whether it’s power failure, hardware failure, or someone tripped over the ethernet cable, it can quickly cause a major disruption to your system.

An easy way to protect against this type of problem is to have redundant processors. In this case, two separate Raspberry Pi systems, both running Node-RED. The slave can either run in direct parallel, or monitor the master, ready to leap into action at a millisecond’s notice!

One method to retain dashboarding is to install Node-RED on a cloud service such as Amazon Web Services (AWS), which provides virtually guaranteed uptime. It’s not reliant on your Raspberry Pi or host computer; however, the downside is no GPIO control. Depending on what you’re controlling, this may not be a problem.

Excellent Redundant Topology

If whatever you’re doing is deemed “mission critical”, then perhaps a little redundancy is required. There are, however, points of failure even with the central redundancy topology previously described. This is where local cache becomes an invaluable tool.

We won’t fully explore this right now, but it’s a cool thing to imagine. Not only can your Raspberry Pi redundant units talk to each other, your individual sensors have localised datalogging also. This means that even if the network connectivity between the sensors and both Raspberry PI units is interrupted, the sensors will continue to log their data. When the connection is restored, you can either pick up where you left off, or easily code a method of retrieving any missed data.

THAT’S A BIT CRAZY ISN't IT?

Perhaps not as much as you think.

We should highlight that the last two options are really outside the scope of most projects. Central redundancy has its uses, and is fairly easily implemented. However, the “excellent redundancy” description requires so much additional code, hardware, and planning to execute.

In reality, if you truly need such a low potential failure rate or absolute uptime, you may need to re-think using consumer-grade hardware like Raspberry Pi, and start the entire planning process over again!

However it's not really that difficult to setup and get running, and the hardware investment is well within the reach of a suitably sized project.

THE COMPLEXITY/REDUNDANCY TRADE OFF

There is always a level of complexity that we're willing to engage with. If you simply have a water temperature monitor in your pool to determine whether it's nice enough to go swimming, then reduncancy isn't really something to consider. After all, if you going swimming is critical because you're an olympian, you're going to get into the water anyway.

The information is nice to have, but the data is not critical for decision making. You do however, have non-technical redundancy. Once you touch the water you're going to find out if it's nice or not!

The data you're gathering here is also non-critical to any other outcomes, especially outcomes that are likely to get someone injured, or property damaged.

However let's think about something in the middle-ground. Perhaps you have your system monitoring the water levels, to automatically top-up the pool water from your rain water tank when it falls below a certain threshold. After all, if the level drops too far you could run the risk of damaging your pool filter if the pump tries to work, unable to draw any water. This is where you can find yourself in middle ground.

At the end of the day, that would be costly to replace or repair the pump if required. Nobody is in any danger, and you can make that assessment yourself.

What if we change this water level sensor to a cattle station in the middle of Australia, where water is life and death to the cattle? In the drier months, a stock watering station could be the only source of water for miles.

In this scenario, the value of perfect function and redundancy has exponentially increased. Not only do you have the lives of the cattle at stake (not steak!), but cattle are valuable to the farmer. They're not farming them as pets! If some element of critical functionality fails, there could lead to a catastrophic event.

We should note, that we're not implying that a competent farmer wouldn't be checking these critical systems themselves as frequently as possible. After all, the cattle would be one of the most valuable things on the property in many instances.

As farms become more autonomous and rely further on technology however, these ideas have to be considered. It's not enough to implement some type of automation or technology, without appropriate checks and balances.

WHEN IT SIMPLY DOESN'T MATTER

This type of thought experiment can be useful though, in planning any type of interconnected system, and determining where the “acceptable failure rate” and cost/benefit of the solution may rest.

It's far more likely that you're simply controlling your home’s air conditioning, based on the outside air temperature, to switch it on automatically just before you get home.

In this type of scenario you can probably afford total failure. Because the worst case scenario is that it doesn’t switch on, or doesn’t switch off. You might be a little warm on a summer afternoon while the air conditioner gets to work after you get home, but there is no serious event occurring as a result.

This type of consideration can help drive system design and hardware choices to keep implementation costs down. If the application has some level of importance to it, you can easily scale up and down accordingly.

SENSOR QUALITY

It would be remiss of us to not mention that this consideration applies to the sensors you're using, as much as it does any control / display hardware.

If you simply opt for the cheapest possible sensors without consideration for the critical nature of the application, you could have a failure at the sensor, rendering even the most redundant display system useless.

Then again - the same "redundancy" theory can be applied to sensors. Perhaps two cheap sensors are indeed better than a single top-of-the-line- sensor anyway.

Either case is easily designed for, but your precise requirements will drive these decisions.

Build 1:

Lighting Controller

You will also need basic prototyping hardware such as a breadboard and jumper wires. *Quality shown, may only be available in packs.

Our first build looks at using Node-RED as a dashboard for controlling GPIO. With this basis, you can easily control all sorts of things using the GPIO on the Raspberry Pi.

BUILDING THE CIRCUIT

The circuit for this build is very straight forward. Simply follow the diagram to wire up your two LEDs and resistors.

We're only using them as representations of a dimmable lighting system, which we'll dashboard and control via Node-RED.

Take care with your connections, then power up your Raspberry Pi.

THE CODE

We’re going to be using our Raspberry Pi to run Node-RED so that we can showcase some of the GPIO functionality; however you do not require a Pi to run Node-RED. You could also run on a local computer or cloud service such as Amazon Web Services or IBM Bluemix.

THE SETUP

Let’s set up Node-RED on the Raspberry Pi.

We’ve loaded Raspbian Stretch with desktop (if you need additional help, refer to our guide in Issue 10). Node-RED comes preinstalled but lacking npm (the node js package manager), which is needed to add Node-RED extensions such as the dashboard UI, which we plan to make use of later.

To install npm. Boot your Pi, connect to the internet then run:

sudo apt-get update
update-nodejs-and-nodered

If you’d like to configure Node-RED to auto start when your Pi boots, run:

sudo systemctl enable nodered.service

We’re ready to access Node-RED. To do so, open a web browser and point to http://[address]:1880 - replace the address with the Pi’s IP, or localhost if browsing from your Pi.

Now it’s time to play!

For our first experiment we’ll create a time stamp and push it through to the debug console for viewing.

To do this, drag an inject node onto the workspace. By default the Payload will be a time stamp.

Next, drag a debug node onto the workspace and connect the two nodes by dragging between the output port of the inject node, to the input port of the debug node.

Click on the “deploy” button to save the changes to our server. Now click the “inject” button on the inject node to trigger the function. To see the output, open the “debug” tab on the sidebar.

You should now see the system time stamp.

Let’s improve this, so it’s easy for a human to read. To do so, we can add a custom function between the two nodes, which will transform the data.

There are lots of predefined functions to choose from in Node-RED, but what’s awesome is that we can also write our own JavaScript functions to suit our needs.

Drag a function node between the two existing nodes, then double click the node to edit. Enter the following code to translate the time stamp into a readable format then click “Done”.

var date = new Date(msg.payload);
msg.payload = date.toString();
return msg;

Click “deploy” to save the changes, then click the “inject” button again. You should see a nicely formatted time stamp in the debug window.

HELLO WORLD - THE BLINKING LED TEST

Next, let’s explore how we can access the Pi’s GPIO.

Grab a breadboard and connect an LED to pin 7, with a 330Ω resistor and Ground to pin 6.

From Node-RED, delete all the previously created nodes, so that we have a blank workspace to work with.

To create a timer, drag an inject node into the workspace, and edit using the following settings:

Note: Make sure you set the Payload type to boolean.

Next we’ll create a trigger node, which we’ll use to send the appropriate message to the GPIO pin, with 0 for LOW and 1 for HIGH. Edit with the following settings:

Now create a Raspberry Pi GPIO (out) node and configure with:

Finally wire them all together.

The blue dots indicate that we have changes in our editor, which have not yet been deployed to the server.

Click “deploy” and the LED should start blinking.

NODE-RED DASHBOARD UI

This Node-RED extension provides an additional set of dashboard nodes, to quickly create a live data dashboard.

To install, click the burger menu icon › manage palette.

Under the Palette menu select the “install” tab, find the node-red-dashboard and click the “install” button. We should now have access to a collection of dashboard nodes to use in our application.

To view the UI dashboard browse to http://[address]:1880/ui/

The dashboard will be empty as we have not set up any dashboard nodes yet, so let’s do that now!

HOME AUTOMATION PROTOTYPE

We will create a home automation prototype that controls two LEDs (we can pretend they are house lights). and also set up a sensor to monitor the room temperature and humidity.

LED Controls

The first LED will have a simple on/off switch and the second LED will have brightness control.

LED 1 - On/Off Switch

First let’s create a new dashboard tab to add our LEDs into. From the sidebar select dashboard then click the “+” tab button, and edit the field marked "Name" to say "Lights".

Delete the inject and trigger nodes that we created earlier, keeping just the GPIO node, as we will now use a switch button to control the LED.

Create a switch node onto the workspace and edit with:

Then click “done”. Next, wire the switch node to the GPIO pin 7 node, and click “deploy”.

Now browse to http://[IP]:1880/ui/

You should have an on/off switch for the LED.

LED 2 - Brightness Control

Connect a second LED to pin 11 with another 330Ω resistor, then connect to ground. Create a slider node, and edit with:

Next, create another Raspberry Pi GPIO (out). Configure with:

Now wire the brightness slider to Pin 11 and click “deploy”.

We can also very easily add a pretty gauge, just because!

To do so, create a gauge node and edit using:

You can easily control and display in various formats, using the same core element. Occasionally it's useful to have data displayed in various ways at the same time.

This is standard PWM control from the Raspberry Pi on the brightness pin, so the same theory applies if scaling up using a transistor to control more power.

Build 2:

Climate Monitor

Note: You don't have to have completed Build 1 first, however there are important principles covered in that build that may be useful here for using Node-RED.

One common use for dashboarding is as a data log type configuration. Text files are fine, but often we want nice pretty graphs! This solution uses MQTT via the Mosquitto service, right on the Raspberry Pi. This saves another hardware requirement, but also means if you’re already using an external MQTT service, you can easily adapt this build.

Adapting this build to use multiple wireless sensors is very straight forward, as is the modular nature of Node-RED.

In fact, you can easily combine GPIO-connected physical sensors as well as network sensors onto the one dashboard. This flexibility is really what makes Node-RED so powerful.

CONFIGURE MQTT ON THE RASPBERRY PI

Before we add the temperature/humidity sensor, we will set up a broker as a way for our two devices to communicate.

For this tutorial we will install a local MQTT broker and host it on our Raspberry Pi.

The broker is called “mosquitto” or “MQTT”. To install run the following command from your Raspberry Pi:

sudo apt-get install -y mosquitto

Mosquitto should now be installed and running, and it will automatically start when your Pi boots.

Now let’s setup a username and password to secure it.

cd /etc/mosquitto
sudo mosquitto_passwd -c passwordfile username

Then enter a password. If you make a mistake you can remove the password file and start again by typing:

sudo mosquitto_passwd -D passwordfile username

The default MQTT broker HOST address is [IP-of-Pi]:1883

You can get your Pi’s IP by typing:

ifconfig

Now we’re ready to set up our temperature sensor.

NOTE: We have covered MQTT in various projects before. It's a powerful and versatile way to create IoT communication. If you already have an MQTT broker running, and devices streaming data, you can skip the WiFi mini steps, and straight to Dashboarding The Data. However we still encourage at least reading these parts anyway, as it may provide you with new ideas.

SETTING UP THE WIFI MINI

We’re using a Duinotech WiFi Mini ESP8266 main board, with a DHT11 Shield. This is modelled on a Wemos D1 Mini, so if you are using Wemos hardware, the process should be basically the same.

Since our DHT11 sensor is already mounted on a shield made for the WiFi mini, there is actually no wiring to be done.

This means all you need to do with the hardware is connect the shield to the WiFi Mini. You can however, utilise the same sketch with any ESP8266-based device; simply take note of any pin changes for your own wiring setup.

The setup is easy - after installing the CH340 USB driver, add a board profile to the Arduino IDE and then you’re ready to upload sketches via USB.

Download and install CH340 USB Driver from http://www.wch.cn/download/CH341SER_ZIP.html

This page was not in English, however Google Chrome offered to translate for us, so we could still navigate the page and find the drivers for our OS (MAC).

This will ensure the USB can work with your computer, as it may not automatically plug and play.

Add Board support for the ESP8266 in your Arduino IDE

Go to File › Preferences › Additional Board Manager URLs and add: http://arduino.esp8266.com/stable/package_esp8266com_index.json

Now go to Tools › Boards › Boards Manager and install esp8266 by ESP8266 Community.

Next select Wemos D1 R2 and mini from the boards list. This is compatible with the WiFi Mini ESP8266 and most incarnations of this type of WiFi board.

Finally, select the USB port under Tools › Port to allow Arduino IDE to establish connections with your hardware.

Download the code package from the online resources. The code is located in the arduino-code/temperature.ino sketch within our download package.

Enter your WiFi SSID and password on lines 14 and 15, and your MQTT broker’s IP address, username and password on lines 17 to 20.

Now upload the sketch to the device using the Arduino IDE.

We will not dig into the sketch itself, as our focus is on Node-RED. But the summary of the sketch is:

Once you've successfully run the sketch, you will see the following output:

Congratulations! You now have your temperature monitor working.

Now it’s time to do something interesting with it!

DASHBOARDING THE DATA

Head back over to the Node-RED admin panel. To keep things tidy and organised we will create a new tab to store our ESP8266 sensor logic. First double click on the “current flows” tab and rename to Raspberry Pi. Next, create a “new” tab and label this one “ESP8266 WiFi Mini”.

Now that we have a blank workspace to play with, drag in a MQTT input and edit with:

TOPIC: climate/bedroom

QOS: 0

SERVER: localhost

PORT: 1883

If you set up a password for your MQTT broker, click the “security” tab and enter your username and password.

We are now subscribed to the MQTT topic but we need to do something with the messages once we receive them. You could optionally connect to a debug node to quickly test and view the incoming messages in the debug console; just remember to “deploy” to the server, in order for the changes to take affect.

Next, we will let our workflow know that we are expecting to work with a JSON object. Create a JSON node and connect it to the MQTT input.

Now, let’s write a JavaScript function to get the value of the temperature property from the JSON object.

Create a function node and edit with the following code:

msg.payload = msg.payload.temperature;
return msg;

Connect this node to the JSON node.

Now that we have the temperature value, let’s output it to a gauge on the dashboard.

Create a gauge node and modify. Edit the group. We’ll label this “Bedroom” and list under a new tab called “Climate”. Click “update” to save.

We’ve also added the following settings:

A NOTE ABOUT SIZE: The grid is 6 columns wide, so we’ve selected 2x3 for this gauge. This will save room for a 4x3 chart that we will add next, to sit next to our gauge, which will fit nicely on a smartphone screen.

The colours are adjusted to better suit cold, warm and hot temperatures.

Next we’ll create a chart to display a log of the temperature.

Drag a chart node onto the workspace and edit with the following settings:

Now wire the gauge and the chart nodes into the JSON node, and click “deploy”.

Load up the dashboard and you should now be able to see these widgets under the “Climate” tab. Now to hook up the humidity displays.

Create another function node but this time we will grab the humidity value:

msg.payload = msg.payload.humidity;
return msg;

Connect the function node to the JSON node.

Create a new gauge node and edit with:

Lastly, create another chart node with the settings:

Then connect the humidity gauge and chart nodes into the humidity function node:

Click “deploy”. Now your dashboard should show both temperature and humidity.

multi-SCREEN usage

One of the clear benefits of this dashboarded type system using Node-RED, is the ability to have multiple control points.

In traditional automation systems, a dedicated control panel is often required. The Node-RED dashboard is available to any device with a web browser which is on the same network.

ADDITIONAL FUNCTIONALITY

IMPORT/EXPORT

Node-RED has an easy import/export system, allowing you to save your project as a JSON file and share with others. Just click the Burger menu › Import › Clipboard.

For those who wish to import, we have provided an export of this tutorial in our online resources.

ADD TO SMARTPHONE HOMESCREEN

To hide the URL bar and for a native app experience, browse to the dashboard and then:

IPHONE: Safari › Add to home screen

ANDROID: Chrome › Add to home screen

Now you can open it from the homescreen, and the app will launch full-screen, without showing the pesky URL, leaving it feeling just like a native app!

REDUNDANCY

As we noted in the introduction, it's worth considering how critical your Node-RED system is to keep functioning.

While there's no specific event that would cause your Raspberry Pi or Node-RED software to stop responding, it's entirely possible.

Even the world's largest data centres use multiple servers, which are intelligently taken out of the loop when things go wrong.

Node-RED doesn't quite enable this out of the box, but with some intelligent design considerations you can make it happen, based on the level of uptime guarantee you need to ensure.

WHERE TO FROM HERE?

This project only scratches the surface of the potential within Node-RED, but should provide you with a clear and useful overview into what’s possible, and get you inspired to experiment.

Some additional functionality you can try yourself could include:

• Add a time control node and schedule your LEDS to turn on and off
• Set your temperate function to send you an alert if the temperate exceeds a certain value; yep, there’s a node for that too!
• Configures lights and a light-sensor, to gradually turn on your lights, increasing in brightness to maintain constant lighting as natural light falls.
• Trigger relays under certain conditions to automatically startup backup devices.
• Implement a "keepalive" check which simply reboots your Raspberry Pi if no response is returned (simliar to our Arduino Keepalive project using ATtiny85).

As you’re probably already thinking, you could easily take the ideas from this prototype and automate your home! The sky is really the limit with Node-RED.