How well do you understand the data your thing produces? How do we get it “out” and into human-readable form?
We talk a lot about “IoT-ing” things these days. If you are unaware of what “IoT” means, it refers to “The Internet of Things”. This very broad and vague term has sprung up over the last few years, and is typically used as an attempt to describe and encompass all the devices that have the capability to connect to the internet and have an active presence. We have seen this significant increase with the advent of the Raspberry Pi, and more notably the ESP8266, which has come to the forefront of the maker space. Thanks to this affordable, small microcontroller, we can pretty much add internet connectivity to almost anything. Of course, the consumer market also spurred this on, with the introduction of home automation (i.e., controlling things like light bulbs, speakers and garage doors with our phones; no matter where in the world we are).
Adding the element of control and being able to capture data that is available anytime on the internet, without the need to go and find a data logger to collect the information, is a fantastic step forward. If our project is monitoring something, we can now easily access this data in real-time. Of course this, in itself, presents problems. What if your device loses connection or the device that is connected gets damaged? With a connection to the internet, we can easily push our data to a server to store the information. The most bare-bones solution would be to create a database using software such as MySQL, but this means either using an API on the device itself, or writing an online API using PHP or similar, to process our data. If only there were a simpler way...
THE BROAD OVERVIEW
According to its developers, “ThingSpeak is an open source Internet of Things (IoT) application and API to store and retrieve data from things using the HTTP protocol over the Internet or via a Local Area Network. ThingSpeak enables the creation of sensor logging applications, location tracking applications, and a social network of things with status updates”.
ThingSpeak was originally launched in 2010, as a service in support of IoT applications. It has integrated support from the numerical computing software MATLAB from MathWorks, which allows users to analyse and visualise their data using MATLAB, without requiring the purchase of a MATLAB license.
HOW IT WORKS
The first thing we need to do is to set up your ThingSpeak account. Once you have created an account we need to do a couple of things before we can start programming our “thing”.
Our next job is to create a New Channel, which you will find under Channels > My Channels. After clicking on the new channel, we are presented with several options that we need to complete. To give the channel a name, enter a unique name for the ThingSpeak channel. In the next box is the description; here we enter a description of your ThingSpeak channel. We are then presented with a series of field boxes, of which we need a minimum of 1 and a maximum of 8. Check the box to enable the field, and enter a field name. Next, add any relevant Metadata and enter information about channel data, including JSON, XML, or CSV data. If you have any relevant tags, then enter the keywords that identify the channel. Separate tags with commas if you have multiple.
If your project is location-based, you can add in both latitude and longitude. We need to specify the position of the sensor or thing that collects data in decimal degrees. For example, the longitude of the city of London is -0.1275. Similarly, if you require elevation, then specify the position of the sensor or thing that collects data in metres.
If you want to link to an external site such as a website that contains information about your ThingSpeak channel, specify the URL.
You also have the option to add a video URL (i.e. if you have a YouTube or Vimeo video that displays your channel information), so in that instance, specify the full path of the video URL.
THE VIEW DATA SCREEN
With our channel set up we can look at the data tools. Click on Channels > My Channels and you will be presented with all of the channels you have created. Click on your newly created channel and you will be greeted by your data, displayed as a basic bar graph. You will notice you have a Private and a Public view. If you only want to view your data analysis in the ThingSpeak web app, then you do not need to worry about the Public tab. However, there are some great apps out there, such as ThingView, which will pull the data down and allow you to see an even greater visualisation of your data.
MORE
ThingSpeak gives you a limited number of messages for free — a mere 3,000,000 per year, in fact! This equates to 8,129 messages per day or 338 per hour, or around 6 per minute. In addition to this “limitation”, there is a message update interval limit of 15 seconds.
At this stage we don’t have any data in the system. ThingSpeak requires the use of API keys, and you will find these in the API Keys Settings. This contains the Write API Key which you will use to write data to a channel. The Read API Keys are used to allow other people to view your private channel feeds and charts. Note: Use this field to enter information about channel read keys.
The API itself, is quite simple in terms of how it works. We need to do a basic URL call, to read and write to the channel. The examples below show how this is structured:
API REQUESTS
Update a Channel Feed:
GET https://api.thingspeak.com/update?api_key=8DS5DM3Y9AK12345&field1=0
Get a Channel Feed:
GET https://api.thingspeak.com/channels/367828/feeds.json?results=2
Get a Channel Field:
GET https://api.thingspeak.com/channels/367828/fields/1.json?results=2
Get Channel Status Updates:
GET https://api.thingspeak.com/channels/367828/status.json
Now that we have an understanding of how ThingSpeak works let’s look next at actually doing something with it, and getting some data in. The following examples use a simple DHT11 sensor.
Arduino
This particular code is designed and tested on the NodeMCU, however it will translate directly to any of the ESP8266 family. Another option would be to use the Arduino with an Ethernet shield. The core code to write to ThingSpeak will work, but you will need to import the Ethernet library, and change the WiFi library object. This assumes that we have a DHT11 temperature and humidity sensor on Pin 2.
If you are using this code please make sure you have the ESP8266 and DHT libraries installed. We then define the pin that will be used for the DHT sensor, before defining which DHT sensor type we are actually using for the library. If you are using the DHT22 or similar, then change this section to that particular sensor.
Next we need to enter the SSID and password for the WiFi network we will be using for our project. Now we come to the ThingSpeak details. We need to enter two key pieces of information to enable this to work: the first is the API Key. This is generated on “API Keys” tab. To write data to our channel we need to use the “Write API Key” to ensure that nobody else can write to our data when we do not want them to.
The next line sets up the URL that we will be using to call the ThingSpeak API. It is best practise to use api.thingspeak.com but if you are having issues you can use “184.106.153.149”.
Finally, we instantiate the WiFi client and DHT objects.
#include <ESP8266WiFi.h>
#include "DHT.h"
#define DHTPIN 2
#define DHTTYPE DHT11
const char* ssid = "XXXXXX"; // SSID of local network
const char* password = "XXXX"; // Password on network
String API Key = "8DS5DM3Y9AK12345";
const char* server = "api.thingspeak.com";
WiFiClient client;
DHT dht(DHTPIN, DHTTYPE);
Our setup code has been truncated for clarity, so please refer to the full file listing for the complete code.
We begin by starting both the DHT and WiFi objects before echoing the connection information using the serial.print function.
The next section of code is the status loop. This makes sure that the system will continue in setup mode, until the WiFi has been connected and correctly established.
void setup() {
dht.begin();
WiFi.begin(ssid, password);
Serial.print("Connecting to ");
Serial.println(ssid);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");
} Our main loop is quite busy but is not all that complex. Ultimately we need to read the DHT sensor, then take that data and convert it into the URL string that ThingSpeak requires, before waiting for the set period of time, after which the process is then repeated.
The code is quite verbose but makes for easier reading. The first two lines are the declaration of the floats used for storing the temperature and humidity data before we send it. Even though we declare floats to allow decimal points, the DHT sensor will only return whole numbers. The next “if” statement is to check that we were able to read the DHT sensor correctly; if not then it throws an error.
void loop() {
float h = dht.readHumidity();
float t = dht.readTemperature();
if (isnan(h) || isnan(t)) {
Serial.println("Failed to read from DHT sensor!");
return;
} Next, we make sure the client is connected and build the API key string. This string prepares the data from the DHT into a long string ready for transmission. This string is declared and starts with the API Key variable. We then follow by adding the “&field1=” and then the data that we are going to "place" into field1. This tells ThingSpeak what data field this piece of data belongs to. We repeat this for field2 before placing some end-of-line characters.
if (client.connect(server, 80)) {
String postStr = API Key;
postStr += "&field1=";
postStr += String(t);
postStr += "&field2=";
postStr += String(h);
postStr += "rnrn"; We now have to send the data in our string to the server. We use the client.print function to send our data to the HTTP server using POST. We can see in the below section, that we use the API Key in the fourth line, before sending the string we built in the previous section in the last line.
client.print("POST /update HTTP/1.1n");
client.print("Host: api.thingspeak.comn");
client.print("Connection: closen");
client.print("X-THINGSPEAKAPI Key: " + API Key + "n");
client.print("Content-Type: application/x-www-form-urlencodedn");
client.print("Content-Length: ");
client.print(postStr.length());
client.print("nn");
client.print(postStr);
}
client.stop(); We have now sent our data to the server. Next we wait for a defined period of time before we repeat the process. For projects like this, once a minute is ample; but you can sample and send much faster. Remember if you are using a free account you are limited to a minimum 15 second wait between updates.
Serial.println("Waiting…");
delay(60000);
}Python
Note: This is not the complete code, so please refer to the included files for the full code listing.
This assumes a DHT sensor to be on GPIO23.
We start out by importing the required libraries.
We also need Adafruit’s Python library for the DHT11 sensor, which you can find at https://github.com/adafruit/Adafruit_Python_DHT.git
import sys
import RPi.GPIO as GPIO
from time import sleep
import Adafruit_DHT
import requests
This is our function for pulling the data from the DHT temperature and humidity sensor. We call this from the main section. Python handles its variables slightly differently and it returns two strings: one for the relative humidity (RH) and one for the temperature (T).
def getSensorData():
RH, T = Adafruit_DHT.read_retry(Adafruit_DHT.DHT11, 23)
return (str(RH), str(T))
This section shows how we build the actual string and send it to the ThingSpeak server. The first line is our base URL, and join it with the API Key. The system calls the getSensorData function. With this, we then construct a similar string to the Arduino C code, and then output it to the screen. With the data pushed to the server, we then send the system to sleep, and then repeat.
baseURL = ‘https://api.thingspeak.com/update?api_key=%s’ % ‘8DS5DM3Y9AKCR123’
while True:
try:
RH=’43’
T=’23’
#RH, T = getSensorData()
f = requests.get(baseURL +
"&field1=%s&field2=%s" % (T, RH))
print (f)
f.close()
sleep(15)
except:
print (‘exiting.’)
break
USING YOUR DATA
Now that we have our data in ThingSpeak, what can we do with it?
There is a myriad of ways to slice and dice your data. If you have any experience with MATLAB then the code will switch straight over. We have many options for visualisations, but the simplest is to plot our data with line graphs over time.
You can see below our uploaded humidity and temperature data over the course of several days.
The Thingspeak dashboard can also provide you with enormous insight and data visualisation across multiple channels.
ThingSpeak offers a simple and fast way of collecting your data and makes it easy to download into a spreadsheet format, for further processing or analysing. If you need a way of easily collecting large volumes of data over time, we would highly recommend using this service.