Streaming real-time data from Arduino using AJAX and persistent connections

I decided to include an AJAX example that works with streaming data from Arduino in the last version my DHCPLite library. I already showed how the library allows you to point your browser to http://arduino/ and get "Hello, World!" back. Now I wanted to actually read the data from a digital or an analog input and show the result as a number and paint it as a circle (using browser canvas).

The model is simple: a request to a URL like "http://arduino/A1" returns an HTML page that includes JavaScript code that periodically retrieves data from URL "http://arduino/A1=", which returns the actual value read from analog pin 1. Similarly, requests for A# and D# retrieve and return current values read from those pins. The result looks like this:

I have been able to send requests every 40-50ms; I also added 100ms delay between response and the next request to avoid overloading the server.

Before we look at the client-side code, I'll describe the tricks I applied on the server side that some may find useful.

First of all, when I was testing performance on subsequent requests, I noticed that there was occasionally a large gap between receiving a response from Arduino and sending a subsequent request. My original server code was closing a connection after each response and I realized that this was causing additional traffic to close and re-establish the connection. As all modern browsers support persistent connections I simply removed RedFly.socketClose(sock) call and added Content-Length HTTP header to the response to indicate the length of the message (to let the client know when to stop reading the socket). This change had significant effect on performance and the delays between the requests almost disappeared.

There is one thing to keep in mind though: while the persistent connection delivers better performance, the connection is indeed persistent and another client may not be able to connect to the same server on the same port. [Update 2011/12/20] To allow multiple connections you may need to open several sockets as in:

h1 = RedFly.socketListen(PROTO_TCP, 80);
h2 = RedFly.socketListen(PROTO_TCP, 80);
h3 = RedFly.socketListen(PROTO_TCP, 80);

[Update 2012/2/25] It turned out that things are a bit more involved than just opening multiple sockets on the same port. This can only been done when the first socket gets connected (not ahead of time; if you try to run the code above, you will get INVALID_SOCKET in h2 and h3). Here is the sequence of steps to go through in more detail (as per Andreas Watterott, the CTO of the company that designed the RedFly shield, who was super helpful in troubleshooting this and other issues):

  1. Open the first LTCP socket in module (for example port no. 8001)
  2. Socket handle returned for this socket would be 1.
  3. Connect this socket to the remote peer socket
  4. You can now open the second socket in module with the same port no. 8001
  5. Socket handle returned for the new socket would be 2
  6. Connect this socket to another remote peer socket

The second improvement was in removing multiple RedFly.socketSend... calls. I initially had something that looked like this:

RedFly.socketSend("HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n");
RedFly.socketSend("Content-Length: ");
RedFly.socketSend(...sending content length...);
RedFly.socketSend("\r\n\r\n");
RedFly.socketSend(...sending actual content...);

and so on. You get the idea. It turned our that it was a bad way to send the data as there were still noticeable delays between different HTTP requests. When I was reviewing the sniffed traffic in Wireshark, I noticed that there was a large number of duplicate and re-transmitted TCP packets. I cannot say what was causing them, but, since it was an ad-hoc connection, there was no router involved and both client and the server were sitting close to each other. I switched to building a buffer with the response message and sending it all in one packet, and those retransmissions and the delays they were causing disappeared.

The last thing that tripped me on the server side was related to sscanf function. I wanted to be able to parse URLs like /A1 and /D3, but wanted to have a different handler for /A1= as I wanted the first one to return an HTML code and the second one to return the actual value. I initially used a simple and straightforward code:

if (sscanf(out, "GET /%1[AD]%2d= HTTP", &mode, &pin) == 2)....

to check if I indeed get two parameters. But for some reason, this sscanf call returned 2 even when I tested it against "GET /A1 HTTP". I (mistakenly) assumed that it will return 0 when the equal sign fails to match, but it simply returned the number of matches that happened by that time. I added one more parameter to avoid this issue: if (sscanf(out, "GET /%1[AD]%2d= %1[H]", &mode, &pin, &ignore) == 3).... Note that you can use %1[AD] to match 'A' or 'D'.

The client side has all the usual elements: (1) the code is executed from the onload handler (<body onload='doit();'>) to make sure that the HTML elements are fully loaded as this is needed for references to the canvas element to work, (2) AJAX calls are implemented using XMLHttpRequests (I didn't bother with using alternative methods for old IE versions), and (3) repeated calls are made using setTimeout method to avoid making a new call from the same callback and growing the call stack; getMore() function used in the setTimeout call should be a global one, otherwise setTimeout will not see it.

In most cases you wouldn't need to worry about implementing your own functions to handle AJAX requests as these functions are provided by all popular JavaScript toolkits, but this wasn't an option in my case. As this HTML is returned by Arduino and I need to make it work on ad-hoc networks without Internet access, there is no way I can fit jQuery in 32k available to me. In fact, there are only 2k of SRAM that are available, but there is also program memory that can be used to store string constants, and this is exactly what I used. First, I compressed the code HTML/CSS/JavaScript code using HTMLCompressor as it was one of the few compressors that could handle HTML, CSS, and JavaScript in one file and compress it correctly. It reduced the size of the message from 1842 bytes to 1143. Then I split that message into several fragments, as the code would still need to use RAM to send it and I didn't have 1143 bytes available to send it in one piece. This is something to keep in mind while using PROGMEM strings: even though you can store large chunks, you may need enough RAM to work with them.

The last thing of interest is an example of using canvas functions to draw something in the browser:

// do this one time and store generated context
var canvas = document.getElementById('out'); // get canvas element
canvas.width = canvas.height = SIDE;
var ctx = canvas.getContext('2d'); // get canvas context for 2d drawing
// do this every time you need to draw something
ctx.clearRect(0,0,SIDE,SIDE); // clear if there is already something
ctx.beginPath(); 
ctx.arc(SIDE/2,SIDE/2,radius/10,0,Math.PI*2,true); // draw a circle
ctx.fillStyle = '#002D80'; // pick a color to fill the circle
ctx.fill(); // fill the circle

And here is the complete version of the uncompressed code;

<html>
  <head>
    <title>Arduino</title>
    <script type='text/javascript'>
      var SIDE = 200;
      var DELAY = 100;
      var request = new XMLHttpRequest();
      function getUrl(url, callback) {
        request.onreadystatechange = function() {
          if (request.readyState == 4) {
            callback(request.responseText);
            request.onreadystatechange = function() {};
          }
        }
        request.open('GET', url, true);
        request.send(null);
      };
      function doit() {
        var info = document.getElementById('info');
        var canvas = document.getElementById('out');
        canvas.width = canvas.height = SIDE;
        var ctx = canvas.getContext('2d');
        function circle(radius) {
          ctx.clearRect(0,0,SIDE,SIDE);
          ctx.beginPath();
          ctx.arc(SIDE/2,SIDE/2,radius/10,0,Math.PI*2,true); // shrink radius
          ctx.fillStyle = '#002D80';
          ctx.fill();
          info.innerHTML = radius;
          getMore = function () { getUrl(window.location.pathname + '=', circle); }
          setTimeout('getMore()', DELAY);
        }
        circle(0);
      }
    </script>
    <style type='text/css'>  
      html, body { width: 100%; height: 100%; }  
      html { overflow: hidden; } 
      body { 
        margin: 0px; 
        font-family: Verdana,Geneva,Georgia,Chicago,Arial,Sans-serif,'MS Sans Serif';
      }  
      #info { 
        position: absolute; padding: 4px; left: 10px; top: 10px; 
        background-color: #FFFFFF; border: 1px solid #002D80; 
        color: #002D80;
        opacity: 0.8;
        -moz-border-radius: 5px; -webkit-border-radius: 5px;
      }
    </style>
  </head>
  <body onload='doit();'>
    <canvas id='out'></canvas>
    <div id='info'/>
  </body>
</html>

2 Comments

Do you happen to have the completed code somewhere? I'm trying to get realtime info and messages from my ardunio to my web page without patchube or other external service, and have been googling everywhere, to no avail. thanks

Hi Jonathan, I do have a complete example included with the library: examples/DHCPLite/RedFly/RedFly.pde. This includes all the code you need to get the example described on this page working (although if you are not using RedFly shield, you may need to change the socket calls).

Leave a comment

what will you say?
(required)
(required)

About

I am Paul Kulchenko.
I live in Kirkland, WA with my wife and three kids.
I do consulting as a software developer.
I study robotics and artificial intelligence.
I write books and open-source software.
I teach introductory computer science.
I develop a slick Lua IDE and debugger.

Recommended

Close