Web apps move the processing from a server, to the browsing device. This allows complex interaction with simple devices!
When you’re creating interactive or IoT devices, it’s important to consider the devices’ processing power. An Arduino, for instance, can serve very simple data, however, serving complex code from an Arduino can use more memory than you have available. This is where a web-app comes in.
WHAT IS A WEB APP?
As with just about anything from the internet, there is often a lot of differing opinion around what is a web app, what is a website, and things in between.
Realistically however, anything that performs calculations or interactivity within the browser itself, without communicating to the server, can really be deemed a web app.
HOW DOES A WEB APP WORK?
The major difference between a simple web page, and a web app, is the amount of information which is served and processed on the server. Not in terms of volume, but arguably proportion. Web apps are generally capable of a huge amount of data processing before even making a single request to the server, however, it’s not mandatory either.
TRADITIONAL SERVER SIDE PROCESSING
Web apps are indeed a product of available technology. Traditionally, websites relied entirely on Server Side Processing (SSP). The web server itself would serve pages, do any and all processing of user data, and return new pages based on requirements. The processing performed in-browser would be exceptionally minimal.
TO WEB APP OR NOT TO WEB APP
... that is the question. Not every instance will warrant a web-app, and you must consider a few things first:
- Is there a use for offline modes?
- Is the server capable of serving complex multimedia?
- Are there any bandwidth limitations?
Actually, there are many considerations - but they’ll become more apparent as we go.
To put this into a practical sense... if you’re creating a news website, or a Twitter digest, there’s not a whole lot of offline content you can browse. It’s all live from servers, and having offline capability doesn’t really yield you much.
However, if you have a checklist application or task manager, the data isn’t updating so frequently, so there are many scenarios where using the app offline then syncing to the server later, is exceptionally useful.
It also takes a huge burden from the server itself, as we’ll review further later in our examples.
THE MANIFEST
Perhaps the biggest step forward was when browsers stopped requiring things to be on the broader Internet. Once a web browser was free to function when there was zero network connectivity, things started to get REALLY interesting. This opened the door for true web-app functionality.
Now, an intelligent web app could make everything happen in-browser, store results and changes in the localStorage or other methods (explained in more detail later). The data is stored in the browser itself, and the web app continues to function normally. Generally, once network connectivity was available again, the data would be synced to a traditional web server.
When you’re sitting in an office with 5G WiFi connected to a faster-than-necessary internet connection, this is rarely a thought. However, in areas where mobile reception was flakey, or non-existent, the ability to roam with your web-app, then return to land-connected WiFi opened the door to opportunities that were previously non-existent.
Here’s an example of a manifest file for HTML5:
CACHE MANIFEST
/styles.css
/scripts.js
/default_image.jpg
You then save the manifest file as something like “mymanifest.appcache”, and then add the name in the HTML file you’re using. For instance:
<html manifest="mymanifest.appcache">
This will then allow caching of your ‘must-have’ files, so the app can work offline entirely.
While the manifest file itself is required to be on the same host as the HTML file (a requirement that exists in many systems for security), you can link to external resources. This is great if you’re using a library such as jQuery, and rather than hosting locally, you’re linking to the Google CDN (content delivery network) version. However, there’s usually some security issues to resolve to do this reliably, so we’ll cover this another time.
INGREDIENTS OF A WEB APP
It’s difficult for us to find a place to start here, since web apps require a “full stack” approach. That is, they require knowledge of HTML, JavaScript, and server side processing.
However, we can take you through some important aspects and history of the different technologies that come together, to make your web app work.
HTML
HyperText Markup Language is the fundamental structure of most pages served to a web browser. If you have never experienced it before, it’s a formatted language that provides the ability to describe different content, so that a web browser can display it.
Since the foundations of the internet are rooted in the sharing and distribution of technical papers, HTML itself isn’t designed for complicated layouts and clever designs that we’re now used to browsing. Outside of images and some annotation abilities, there were few functions which related to visual styling.
For those of you who haven’t seen HTML before, here’s a basic heading and paragraph marked up in HTML.
<html>
<h1>Hello World!</h1>
<p>This is HTML, and while it’s not very pretty, it’s very functional.</p>
</html>
This will display something like below.
However, HTML has evolved with the times too. While it’s a slow process, HTML5 (the current standard) brought with it a host of technology which caters for modern websites. Elements for handling video, vector graphics, and many more items which used to require complicated plugins and technologies to make work.
This improvement in standard made developing web apps much simpler, while improving cross-browser compatibility for phones, tablets, and desktops alike.
CSS
Cascading Style Sheets (CSS) are a powerful tool for improving the look and feel of your web app. They drive virtually all visual styling on the internet that’s not an image or video.
Before CSS was popular, HTML was intermingled with styling attributes for layout, colour, and position. This wasn’t a big deal in the very early uses of the internet, since websites were simple and graphics minimal. However, things changed very rapidly.
CSS brought an intelligent way to style HTML objects, with an abstracted layer from the HTML.
Originally HTML catered for a limited number of styling attributes in the markup such as the example below:
<div width="140" height="80" border="1" color="#ff0000">
Hello World!
</div>
This would create a box 140 x 80 pixels wide, give it a border, and set the text-colour to red. Fairly straight forward and adequate for the early Internet.
Soon, more advanced styles were being implemented, though usually in a very clumsy way. The STYLE attribute allowed endless addition of styles using a slightly different syntax. We could now achieve the same box in a more concise way.
<div style="width:140px; height:80px; border:1px solid black; color: red;">
Hello World!
</div>
Wait - that’s not any more concise! Naturally, the transition to this new type of styling syntax simply implemented more flexibility. What it didn’t do, however, was create consistent code.
This is where CSS truly took a revolutionary step to create uniform, flexible, repeatable styling, with concise code.
Now you could simply define style rules, and apply them to your HTML. For instance:
<style>
.stdBox {
width: 140px;
height:80px;
border:1px solid black;
color: red;
}
</style>
<!-- the above would generally be put in the <head> section, but presented together for simplicity -->
<div class="stdBox">
Hello World!
</div>
As you have probably gleaned from the code provided, you can now create as many boxes as you want simply by using the class attribute.
One step further is for all styles to go into their own file. The file then becomes cached (i.e. stored in memory) in the web browser, so it doesn’t need the information from the web server on each load. This speeds up page loading, reduces avoidable data transfer, and makes everything run better overall.
We’re not trying to educate you in CSS just yet, but you can now see the benefits CSS has for simplifying HTML and making uniform styles.
JAVASCRIPT
JavaScript was (and arguably is) the gateway to movement on the internet.
It wasn’t all smooth sailing, however. The early internet had a love-hate relationship with JavaScript. While it provided some fantastic functionality not possible with static HTML, there was a lot of distrust.
In the 90s, many people had Javascript switched off as a security measure. As developers, we had to accommodate for script-disabled browsers for many years.
These days, JavaScript is the backbone of Web 2.0. The world’s most popular websites are all now JavaScript dependent.
Google search works perfectly with JavaScript disabled, but you do lose all the nice features such as auto-suggest/auto-fill that we take for granted.
Even our very own DIYODE Magazine website will run as best as it can without JavaScript, however, a warning is displayed. Some items such as the checkout, rely on JavaScript for various features, and cannot function without it.
In reality, JavaScript is now such an accepted part of websites that no-script notices / scriptless functionality is merely a courtesy. It’s no longer expected that sites are no-script compatible for 100% function when JavaScript is disabled.
In line with this popularity, various JavaScript libraries became popular as a method to simplify what can be complex code, and at the time, resolved many cross-browser challenges developers faced also. While they still do handle some of this, browsers are generally less finicky than they were in years past.
For our examples here, and the code examples, we’ll use jQuery. jQuery is arguably the most popular JavaScript library, and takes much of the hard work out of using JavaScript. It only requires the inclusion of one JavaScript file, and opens up a host of functionality.
AJAX
Asynchronous JavaScript and XML (AJAX) changed things further. It’s worth noting that AJAX doesn’t always have anything to do with XML, however, AJAX has been adopted colloquially for the functionality. If you’re returning plain text, JSON, HTML, or anything else, it’s still generally acceptable to call the action AJAX.
Traditional server requests from a web browser came in two forms; GET and POST (there are others, but we’ll stick with these for now).
A “GET” request was in the form of a URL (eg. loading a page), and a “POST” request was generally submitting form data (eg. a contact form). These requests typically required an entire page reload. Depending on the action taken and the data provided or requested, you would be served a different page (or modified page) compared to the previous.
The system was stable and reliable, however, you spent most of your time online waiting for pages to load - especially in the dialup days. And often, the items you were loading weren’t different to the ones you just loaded!
AJAX does away with this overhead, but handling GET and POST data independently of the page load. Even if you didn’t notice it, you would have observed this functionality. Posting a comment to Facebook and having it appear, endless scroll (where new content is loaded automatically when you reach the bottom of a page), or updating your cart contents and the total recalculates instantly. These are almost always examples of AJAX in action.
As the name suggests, AJAX relies on JavaScript being available. So while the technology existed for AJAX for quite some time, it didn’t really become popular until JavaScript itself was more accepted and more browsers allowed JavaScript by default.
LOCALSTORAGE
LocalStorage is a relatively recent addition to the web app toolkit, but it can play a powerful part in a great user experience.
Typically it was tough to store much data in the user’s browser. You’ve no doubt heard the term “cookies”, and these were the most fundamental way to retain a user’s session information. It meant that when you refreshed the page, or restarted your computer, the server still knew who you were. Your preferences, cart contents, and other personalised aspects of the site were retained.
LocalStorage makes storing and using data a breeze. Unlike Cookies, which are designed to be handled by the server, LocalStorage really is focused on the in-browser activity. The data is persistent unless its explicitly deleted, and you can store up to 5MB, which is loads of storage! Cookies, by comparison, can hold a maximum of under 4kB depending on the browser.
Similar to LocalStorage is also sessionStorage. It’s a slightly more restrictive version of LocalStorage, but is specific to the browser tab and automatically cleared when the user closes the window. This is particularly useful for in-page information and preferences, but not terribly useful for information required on the next visit.
For most web apps, LocalStorage will be the logical choice for persistent data. It’s accessed and managed by JavaScript.
SERVER SIDE
For many DIYODE readers, your own IoT device is going to be your reason for developing a web app. In this context, it’s not a traditional web server that creates the server endpoint, but an Arduino or Raspberry Pi.
In this example, we’re going to use an Arduino. The process is virtually identical for any network-capable Arduino, however, we’re using an Ethernet shield and an UNO. It's a simple pairing for experimenting in this area, since there's no Wifi challenges to deal with - simply plug in and play. If you’re using a WiFi enabled board, you may need to ensure libraries and board configurations are setup correctly for your particular hardware.
In order to get your Arduino to act as a web server, it’s fairly straightforward. Ethernet versus WiFi is only marginally different. For Ethernet you only need this code:
#include <SPI.h>
#include <Ethernet.h>
// Enter a MAC address and IP address for your
controller below.
// The IP address will be dependent on your
local network:
byte mac[] = {
0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED
};
EthernetServer server(80);
void setup() {
// Open serial communications and wait for port to open:
Serial.begin(9600);
while (!Serial) {
; // wait for serial port to connect. Needed for native USB port only
}
// start the Ethernet connection and the server:
Ethernet.begin(mac);
server.begin();
Serial.print("server is at ");
Serial.println(Ethernet.localIP());
}
Congratulations! You made a web server!!! OK, so perhaps the celebration is a little premature. It might be a legitimate web server, but it still doesn’t do anything just yet.
If you had serial monitor open, you should have been returned an IP address, as the following image.
If you didn’t receive an IP address, there are generally two causes. The first; you forgot to connect your Ethernet board to the network! The second; you may not have DHCP (Dynamic Host Configuration Protocol). That’s basically the software which takes care of issuing the IP addresses. Virtually all home networks will run DHCP since it effectively enables plug n play networking. Without it, manual configuration is required. We’ll cover static IPs another time, since dynamic IP assignment will cover virtually everyone for this example.
OK, so our server initiated, but we didn’t tell it to do anything. Now we’ll tell our web server to actually handle incoming connections, at a rudimentary level. We add this loop, since we want it to accept connections forever, not just on startup.
void loop() {
EthernetClient client = server.available();
if (client) {
Serial.println("new client");
boolean currentLineIsBlank = true;
while (client.connected()) {
if (client.available()) {
char c = client.read();
Serial.write(c);
if (c == ‘n’ && currentLineIsBlank) {
client.println("HTTP/1.1 200 OK");
client.println("Content-Type:
text/html");
client.println("Connection: close");
client.println();
client.println("<!DOCTYPE HTML>");
client.println("<html>");
client.print("<p>Hello World</p>");
client.println("<br />");
client.println("</html>");
break;
}
if (c == ‘n’) {
currentLineIsBlank = true;
}
else if (c != ‘r’) {
currentLineIsBlank = false;
}
}
}
delay(1);
client.stop();
Serial.println("client disconnected");
}
}This is really the bare minimum of a functional server, returning some text when you visit its IP address. It will simply display “Hello World”.
At the same time, the serial monitor will spring to life, dumping out data on what’s going on. This isn’t really useful normally, but provides you with some insight into how it all works.
This code is also included in the Digital Resources for this ‘article basic_server.ino’.
VARIABLE RESPONSES
Building on what we’ve just discussed, it’s not difficult to provide conditioned responses (or indeed, perform different actions), based on the request.
Just as you visit different URLs on websites, you can do the same on an Arduino, by providing data in the URL. The most straightforward way to do this is by modifying the URL.
There are really no limits to what you can put into the URL, but it’s always sensible to keep things as simple as possible.
For now, we’re going to create two URL endpoints on the Arduino. /on, and /off. Simple enough to try a few things.
In the digital resources, grab the ‘iot_server.ino’ sketch and load it into your Arduino IDE.
On line 53, you’ll notice some new code:
if (strstr(linebuf, "GET /on") > 0) {As you can glean from reading the code, we’re telling the Arduino to scan the GET portion of the request. GET is the request method that all web browsers use to fetch information. In this case, we’re simply looking for the URL “/on”. The GET portion is everything after the IP address (or more generally, on the internet, the domain name). This means you can easily expand it.
You may wonder why we don’t do a precise match such as:
if (linebuf == "/on") {The reality is that we could, except we would have to strip out any other information that comes along also. Often the full string will contain other information such as the request method (GET), and the protocol (HTTP/1.1), plus a whole bunch of information. This can rapidly get in the way of our logic.
If you have a huge number of cases to account for in your code, then it’s probably worth experimenting with this, as it will indeed be faster to process. However, for a handful (even a few dozen) different URL handlers, this method works perfectly fine, and will provide resilience against future changes.
You’ll see in our code that we handle two URLs.
if (strstr(linebuf, "GET /on") > 0) {
client.println("<!DOCTYPE HTML><html>
<head></head><body>
Switching On</body></html>");
}
if (strstr(linebuf, "GET /off") > 0) {
client.println("<!DOCTYPE HTML><html>
<head></head><body>
Switching Off</body></html>");
}Now it’s worth adding that we have no “default” handler here any longer. Requesting simply the IP address will return nothing. So you may like to add a line such as:
if (strstr(linebuf, "GET / ") > 0) {
client.println("<!DOCTYPE HTML><html>
<head></head><body>
Arduino Server!</body></html>");
}For requests which request the IP (or domain name) alone, the forward slash is implied and sent by default. Requesting http://YOUR_IP or http://YOUR_IP/ will yield exactly the same results, as the browser adds the slash anyway. There also must be an empty space after the slash in our logic, otherwise “GET /” will match to every URL. The space ensures that it matches empty URL only.
HARDWARE ACTIONS
From this point, it’s remarkably easy to handle hardware changes based on the URL requests also. It’s no different to any other GPIO process.
Connect an LED to pin 7 and GND as shown in the image (you'll need a 220-330Ω resistor to protect the LED in series with pin 7 and the LED).
Often with an Ethernet shield, you can’t see the onboard LED, so adding one makes this progression a little easier to follow.
Open up 'iot_server_with_LED.ino’ from the digital resources. You’ll notice on lines 55 and 59, we’ve added some digital writes. This is rudimentary action, but demonstrates how easy hardware interfacing from the web server becomes.
Load the code onto your Arduino and wait for the server to initialise. Now if you visit YOUR_IP/on or YOUR_IP/off in your web browser, the LED will switch on and off appropriately!
Once you have switched your LED on and off a hundred times, you can move on (OK move on any time, but it’s fun, right?!).
SIMPLE RESPONSES TO COMPLEX QUESTIONS
OK, so we took something of a roundabout way to finally reach this point - but now we’re ready for the core web app to do its thing.
As you saw from our previous examples, serving even the most simple HTML takes reasonable effort. But we only served a few lines of HTML - imagine if it was hundreds, or thousands of lines! Our previous examples use around 50% of an UNOs available memory - you can see the problem, can’t you?
With a device like an Arduino UNO, you will quickly run out of memory. Handling the requests is tough enough - but if you’re asking it to return complex HTML as well?
This is really where a web app will play an important role in IoT; offloading the heavy lifting from the IoT device (that is, in this case, the Arduino), and putting the browser to work.
The biggest change that we’re going to make is we have to pre-design HTML, then load it into the browser.
First, let’s look at our fundamental web app design. There are three parts to this basic web app:
- IP configuration
- On button
- Off button
Simple!
We’ll cover just how we create JavaScript events and things in the next installment, but here’s the JavaScript we’re using (utilising the jQuery library). But this web app works, and is fun to mess around with.
You’ll need to load your Arduino with the appropriate code ‘iot_server_web_app_responses.ino’. This has a few modifications to ensure the browser is happy. However, it’s important to note... it’s built for machines! We’ve removed the HTML output from the on / off buttons. Our web app doesn’t need them, and we’re preparing to do some more intelligent responses later. Once you’re running, enter the IP address for your server (provided via serial monitor) into the web app.
Now you should be able to click/tap on the LED ON and LED OFF buttons, and see your LED switch on and off. Awesome!
Behind the scenes, in the JavaScript file (main.js if you download the web app HTML package), there’s a good deal of code. However, here’s where the magic happens:
function get( action, id ) {
if ( !ip ) {
showMessage( "error", "Set IP address" );
return;
}
var url = "http://" + ip + id,
ajaxTimeout = 10000,
$.ajax( {
url: url,
type: "GET",
contentType: false,
timeout: ajaxTimeout,
complete: function() {
}
} );
}This code essentially checks for an IP address (and throws an error if it doesn’t find one), then builds an AJAX request to send based on the button which has been pressed. In this case, we’re not sending anything back from the Arduino, so we’re just sending an empty response.
In the HTML, we created buttons with data references, which the JavaScript picks up on. If you check out the index.html file, you’ll notice things like this:
<button data-id="/on">LED ON</button>
As you’ve probably figured out, we’ve kept things simple, and the data-id is the URL that is checked on the Arduino! Now you have the basics, you can experiment with adding new buttons, and handling them on the Arduino too. Go nuts!
WHERE TO FROM HERE?
Now you understand the fundamentals of creating a basic web-app, you can easily expand the functionality to create advanced user interactions, with very simple devices!
Next month, we’ll go into more advanced web app techniques. We’ll cover more around how JavaScript is attached to the HTML, and send responses from the Arduino and handle them in our web app. We’ll also implement manifest files, and demonstrate some offline / resync capabilities for web apps.