Forever playing by the rules, JSON is a powerful, yet talented container for information.

At the most basic level, programming is about the transfer of data from one point to another. Internally it can be quite simple, but there may be times when we do not know how the receiving system will require its data. So how can we get the information we need, to the place we need it to be?

At its core, programming is the input of a piece of data and then the output of that data, be it for computation or storage. Take for example, the humble calculator app on your phone or system. It serves a single purpose, takes one or two discrete or computed numbers, performs a mathematical function and then outputs them to the user. As the equations get more complex, the basic principle remains the same: take the inputs, perform equations, output results. As our programming needs grow, so too does the complexity of the data. We start to store our data on servers in databases, or SD cards attached to our microcontrollers. This comes with one inherent issue: how do we conform our data to be readable by another system or programmer at a later date?

THE BROAD OVERVIEW

Transfer of data and data protocols

We need to establish a way to get our data from one point to another. For example, we may build a wireless controller to send commands to a robot that we are working on. It is quite simple for us to send a single number – one at a time – to the receiver. We can program the receiver to accept 1 piece of data at a time; or we may add to this and start sending 2 numbers, or 15! Being that we are working with both sending and receiving the data, we are essentially developing our own data protocol. This can prove to be inherently complex as the project grows. JSON offers us a solution that will help you, and anyone else who joins the project, to develop a scalable and rich data processes.

JSON stands for “JavaScript Object Notation”, and it is a text-based, open-standard file notation that transmits text in a human-readable form. It consists of data objects that are attribute-value pairs, and array-style data types. It is one of the most commonly used asynchronous data types, and is seen as a replacement to XML in some systems. Although the J on the acronym stands for JavaScript, you are by no means forced into using this language, as it is language-independent; because of this, we can use it with nearly all modern languages.

While there are smaller and faster ways to transmit and share data, JSON grew from the need to have a real-time, user-readable communication protocol for use in modern browsers, without the use of plugins. It is a serial-based protocol that presents the data in a simple readable form.

HOW IT WORKS

JSON has several data types that can be used in its structure. The “Number” data type is a signed decimal number; it makes no distinction between integers (whole numbers) or floating point (numbers with decimal points). While JavaScript itself only uses a double precision floating point, the final interpretation of your number will be language-dependent. For example, Python recognises the number and creates the right variable type automatically, whereas C/Arduino requires explicit declaration.

The “String” data type is defined as a sequence of zero or more characters. This must be Unicode-compliant and delimitated with double quotation marks. It also supports backslash escaping syntax.

The “Boolean” data type is defined as true and false.

An “Array” is an ordered list of zero or more values, and it can be made up of any of the types listed earlier. They use the square bracket to notate and define their structure, and the elements are comma separated.

An “Object” is an unordered list that can comprise of name-value pairs. The names (or keys) are defined as strings and should be unique. Objects are defined by the curly braces and use commas to separate each pair. They use a colon to separate each name-value pair.

Let us take a look at how it works. While we are using proper JSON structure, the programming code is intended as pseudo code, as each language varies on how we access the data. Take for example, the JSON implementation of a person:

Person={
  "firstName": "John",
  "lastName": "Smith",
  "age": 25,
  "address": {
    "streetAddress": "354 York Street",
    "city": "Sydney",
    "state": "NSW",
    "postCode": "2000"
  },
  "phoneNumbers": [
    {
      "type": "home",
      "number": "02 9123 4567"
    },
    {
      "type": "mobile",
      "number": "0412 345 678"
    }
  ],
  "children": [],
  "spouse": null
}

Objects

The above example is a JSON representation of a person object, but let’s break it down somewhat, and look at the small components.

Person={
  "firstName": "John",
  "lastName": "Smith",
  "age" : "25"
}

This is a simple object. For the context of our data it’s the simplest person we can model. This is an object as it’s denoted and enclosed by curly braces. At it’s simplest, we can access the variables by addressing Person.firstName, which returns “John”, for example:

Print("My name is " + Person.firstName + " " +Person.lastName)
Results in
My name is John Smith

Arrays

Let’s say we need to store a family of people; we can create an array of objects using JSON. The people structure remains the same, but multiple objects are enclosed with square brackets.

Family=[
{
  "firstName": "John",
  "lastName": "Smith",
  "age" : "25"
},
{
  "firstName": "Jane",
  "lastName": "Smith",
  "age" : "23"
}]

In order to access the data, this becomes slightly different, as we need to reference the array index rather than the individual objects. We do this by putting the square brackets at the end of the array, placing the index inside that, and then accessing the variable we need inside the object.

Print(family[0].firstName)
Print(family[1].age)
Results in
John
23

Nesting

The flexibility in JSON comes in the ability to nest arrays and objects. The above example relies on either knowing that John is at index 0 or looping through the array, until we find the object we are looking for. We can make our JSON structure easier to use and navigate, by nesting both arrays and objects.

Family=[
John:{
  "Name": "John Smith",
  "age" : "25",
  "starsign" : "Capricorn"  
},
Jane:{
  "Name": "Jane Smith",
  "age" : "23",
  "starsign": "Leo"
}

We can then access each object by its name or key, rather than knowing its index number or looping through. Which is better? This is going to up to you, and the data that you plan to use. If you have a case similar to this, where you know what your data is, how it will be accessed, and how many rows to expect, then the named key is possibly the right solution. However, if you are retrieving or transmitting data where the length of rows or entries varies, then being able to loop through them until the end is a necessity.

Print(family.John.Name)
Print(family.Jane.starsign)
Results in
John Smith
Leo

JSON is designed with AJAX in mind, and it is one of the most versatile ways of exchanging data. So how do we make the most of it, and use it with Arduino and Python? The implementation of both these systems is quite straightforward, but we need to add some bits and pieces to the development environments first.

Arduino:

We will be using the ArduinoJson library. As usual, we have included this in the files. However, this is one of the most popular libraries for the Arduino on GitHub, and as such, there is a repository available in the Arduino extension manager. If you choose to use the library, please download or go to https://github.com/bblanchon/ArduinoJson to get the latest version. In this case, we would recommend you install via the Arduino Library Manager. To do this open up the Arduino IDE and go to the menu bar, selecting: Sketch | include library | manage library to open the library manager. Once you have this open, type in “ArduinoJson”, select ArduinoJson by Benoit Blanch. The current version at time of writing is 5.11.2.

LIMITATIONS OF NESTED ARRAYS: It's important to note that JSON doesn't have any theoretical limit for nesting. You could put an array, inside an array, inside an array, inside an array, inside a... OK, you get the idea.

While there are no theoretical limitations, reading and traversing arrays take up memory and CPU time. So when you're thinking about how to store data, and how it'll be retrieved. If you're having to nest too far, you're probably storing too much data in an inefficient way.

You should get into the habit of separating JSON arrays into related segments. They'll perform better, and your code will run smoother.

Python:

While the JSON object is built in, we do need to set up the requests module. This is only required if you elect to follow the final section of this article, with regards to getting JSON data from the internet.

If you are using a desktop version of Python then open up the command line and use:

pip3.6 install request

If using Python on a Raspberry Pi go to the command line and type:

sudo apt-get update
sudo apt-get install python-requests

This will add the requests module. You may need to just check your version of Python and machine, to ensure that the above correspond to your specific setup.

BASIC USAGE

So how do we implement JSON in our projects? In Python, it’s quite simple, but you have to remember that there is considerable difference between the Raspberry Pi and the Arduino environment. We will start by examining the basic use of JSON on Python. When you compare the code between the Raspberry Pi and the Arduino, there are considerable differences. We need to be mindful that Python, in this instance, sits on top of the Linux infrastructure and as such, all the underlying buffering and handling of the network is done outside of your control. Due to this, the Python code becomes quite compact.

Python:

import json
json_string = ‘{"firstName": "John","lastName":"Smith"}’
parsed_json = json.loads(json_string)
print(parsed_json[‘firstName’]

Results in;
John

This Python example is a simple implementation of an object and then the retrieval of one element.

Arduino:

When we examine the same code (we have excluded some parts of the general Arduino code for clarity), you can see we have some differences. We have to manually adjust the buffer size, which means that we have the power to control the memory resources much more efficiently. The final two lines are examples of pulling the data out of the JSON object, and back to a usable variable.

Arduino Parsing Program:

#include <ArduinoJson.h>
const size_t bufferSize = JSON_OBJECT_SIZE(2) + 40;
DynamicJsonBuffer jsonBuffer(bufferSize); 
const char* json = "{\"firstName\":\"John\",\"lastName\":\"Smith\"}"; 
JsonObject& root = jsonBuffer.parseObject(json);
const char* firstName = root["firstName"];
const char* lastName = root["lastName"];

Flavour of the Month

It wasn't long ago that XML was the preferred way to store and traverse data in a structured, yet flexible manner. XML was so favoured that prior to HTML5, XHTML became popular. It was more rigid than standard HTML (since in the early days of the web, HTML was the Wild Wild West of coding). However it wasn't to last forever.

JSON proved to be flexible, lightweight, and powerful. Once AJAX technologies started shuttling code around the web without a browser page refresh, standards quickly emerged.

The popular Javascript library jQuery became popular and by its very nature, favoured JSON for storing configuration variables and such, amongst many other things. Even competing libraries such as Dojo used JSON, and it quickly became part of every web developers knowledgebase.

After a few years, server-side web languages such as PHP handled JSON (or more importantly, the conversion of JSON to PHP arrays and objects, and vice versa) seamlessly and quickly. XML on the other hand, was still a little more tricky.

This is probably, in part, what drove the shift from XML to JSON universally. After all, popularity wins with most of these things. Just like XML, JSON may one day fall out of favour, rapidly replaced by the next great thing.

Real-World Example:

Arduino / IoT ESP8266-Based Weather Project

Let’s take the opportunity to put our theory into real-world practice. Up to this point, we have used some JSON data that we have developed and pulled on from our resources. Given the very nature of JSON is to enable systems to talk to one another, without understanding the underlying APIs we will pull some JSON weather data and parse it with both the Arduino and Python.

There are many weather data sources on the internet and they vary in function, usability and price. For this demonstration, we will be using openweathermap.org. They offer data from all over the planet, and it is easy to access via their API, which of course is outputted in JSON. Openweathermap.org not only supply current weather data, but can also serve 5-day and 16-day forecasts, historical data, and even weather warnings. Depending on your usage requirements and loads, there is a cost involved; but for us makers, there is a free option that provides us with the opportunity to get the current weather and five-day forecast from any city in the world. You will need to sign up for their free account in order to get an API key that will enable you to access this additional information.

For this article and demonstration, we will be using their sample call, which provides data about London. Please be mindful that while this code will work with any account, you will need to swap out the sample URL with the actual API URL and your API key. The code in these examples is the start of a much bigger project. You may have seen the smart mirror project with underlying OLED or LCD screens that display the current weather, a forecast and read your daily schedule – that data has to come from somewhere, and there is a good chance its a JSON data feed.

Here we have the complete code for both Arduino and Raspberry Pi (C/Python). You can see that there is a considerable difference in the amount of code required for each to enable the use of JSON with the Arduino required to do a large part of the heavy lifting. Running either will give almost identical outputs.

#include <ESP8266WiFi.h>
#include <ArduinoJson.h>
    
const char* ssid     = "SSID of local network";
const char* password = "Password on network";
    
WiFiClient client;
char servername[] = "samples.openweathermap.org";
String result;

The first part here includes the two headers we need to use in this project. The first is the ESP8266WiFi library as we are using the NodeMCU/ESP8266 for network duties. Next is the ArduinoJSON library, even though there is a bit of work to getting the coding running there is a substantial abstraction layer at work here. The library helps out with these underlying processes. The next two lines are for the SSID and password for your particular network before we initiate the WifiClient. Next, we determine two variables we will be using in servername and result.

void setup() {
  Serial.begin(115200);
  Serial.println("Connecting");
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
  delay(500);
  }
  Serial.println("Connected");
   delay(500);

For the setup routine, we take the created Wifi object and connect to the network. The system performs a "while" loop, pausing for half a second before continuing on. The system will not progress until a connection is present. Once connected it outputs the message connected.

if (client.connect(servername, 80)) { 
    client.println("GET /data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1");
    client.println("Host:samples.openweathermap.org"); 
    client.println("User-Agent:ArduinoWiFi/1.1");
    client.println("Connection: close");
    client.println();
}
else {
    Serial.println("connection failed");
    Serial.println();
}
  while (client.connected() && !client.available()) delay(1);
  while (client.connected() || client.available()) {
    char c = client.read();
    result = result + c;
}
client.stop();

While this looks like a big chunk of code it is used to call our JSON object URL. The next section is simply to identify if the connection has not worked before the last set of while loops. These loops read the returning string of CHAR's and read it into the result array. We now have our data set in one big chunk.

result.replace('[', ' ');
result.replace(']', ' ');
Serial.println(result);
char jsonArray [result.length() + 1];
result.toCharArray(jsonArray,sizeof(jsonArray));
jsonArray[result.length() + 1] = '�';
StaticJsonBuffer<1024> json_buf;
JsonObject &root = json_buf.parseObject(jsonArray);
if (!root.success())
{
    Serial.println("parseObject() failed");
}

Next, we traverse the variable make some char changes to ensure we don't have anything in there that will give us trouble. We then create a new array that is the size of the results array to process the JSON object. We then traverse the CHAR set and create our structure JSON object, root.

  String city = root["name"];
  float temp = root["main"]["temp"];
  Serial.print("The tempreture in ");
  Serial.print(city);
  Serial.print(" is ");
  Serial.print(temp);
  Serial.println(" degress celsius");
}

Once successful we can now call on the root object to pull our data out of the objects. For example, by specifying root["name"] we get the name of the city, but when we call root["main"]["temp"], we get the temperature element of the main array.

Python:

import requests
import json
url = 'http://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1';;
r = requests.get(url)
parsed_json=json.loads(r.content)

Python is much more straightforward to implement the same piece of code as a lot of work is done for you in the library; similarly, Python will deal with all the memory access functions, so it makes the process much simpler. Above we import our two library's, requests and JSON. We then specify the URL we need and send the output to the variable r. We then call the library to parse the contents of r and created a structured JSON object in parsed_json.

print("JSON Payload")
print(json.loads(r.content))
print("nThe tempreture in "
      + parsed_json['name']
      +" is "
      +'{:03.2f}'.format(parsed_json['main']['temp']-273.15)
      +" degress celsius")

This final section of code is a read function to access and display the data similar to the Arduino example. The output for both is the same;

ARDUINO:

Output
JSON Object
The temperature in London is 7.73 degrees Celsius

You can see with there are considerable differences in the code required to get the JSON stream and process it into a usable entity.

import requests
import json
url = ‘http://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1’
r = requests.get(url)
parsed_json=json.loads(r.content)
print("JSON Payload")
print(json.loads(r.content))
print("nThe tempreture in "
      + parsed_json[‘name’]
      +" is "
      +’{:03.2f}’.format(parsed_json[‘main’][‘temp’]-273.15)
      +" degress celsius")

Python:

Output
JSON Object
The temperature in London is 7.73 degrees Celsius

You can see with there are considerable differences in the code required to get the JSON stream and process it into a usable entity.

WHERE TO FROM HERE?

The weather data itself, as well as the availability of worldwide data for you to use, sets up some cool projects ideas. JSON data itself will allow you to have a standardised format to use in many other projects, not to mention what other streams are available for you to use. Check out all the public JSON “things” on thingspeak!

For anyone interested, it's worth exploring the database engine MongoDB. Mongo takes JSON to a whole new level, as part of the NoSQL movement. Rather than storing in traditional rows and columns (effectively like a very large spreadsheet), MongoDB uses what's called document storage. Effectively you have a record ID, but what's in that record is entirely flexible. Of course, some consistency helps store and retrieve data within your software, but the terms are entirely open.

MongoDB documents don't contain plain text, they contain JSON, or more accurately BSON. BSON is effectively a binary-encoded version of JSON. It does provide some additional functionality, but it's exceptionally fast.

One major problem that exists in many technologies is transfer and crossover of data from one language or format, to another. It usually involves iteration through the data and converting it, one key/value pair at a time, to suit the new format. This takes time and processing power, yet many systems do this for every single data read or data store cycle.

MongoDB by its very nature, helps remove this process. It also means you work with your data in a very consistent way, improving development times. While it may be a strange feeling to work with the first few times, without the contraints of table-style storage, it can free you to make better decisions for databasing. Though it's important to realise, not everything should break free of a table. If it's tabular data, then a tabluar database style is probably more suited to the data anyway. We have seen the same argument play out in HTML too with the arguments for and against using tables. Ultimately some of it's purely preference, and perhaps development time considerations too.

With the likes of NodeJS rapidly gaining on historically dominant technologies, it's becoming clear that for now, Javascript and JSON are going to be a part of our code skills for some time. They're powerful, flexible, and fast.