Web-Based Real-Time Application Analysis

99f6aeec9715bb034bba93ba2a7eb360
0
Nick 102 Nov 03, 2009 at 15:00

Hi all,

To analyze a real-time application you typically have to write a custom solution. Think about a developer console, or a transparent head-up display (HUD). But that can be quite a bit of work even it you try to keep things simple. The poor man’s option is often to use a log file to output diagnostic information, and a configuration file for input parameters. But that’s limited to just text and clearly not interactive…

So what I’d like to present here is a solution that is as powerful as a console & HUD, but almost as simple as a log & configuration file. The key is to use existing web technology. By letting a browser take care of the interface rendering you get a lot of possibilities, just by writing a bit of HTML code. To make it interactive, you need an embedded server that will serve the web page and responds to input.

So were do we find a simple embeddable server? Actually, it’s really easy to write our own!

The first thing we need to do is to open a network socket and make it listen for TCP/IP connection requests. In Windows, we can use Winsock for this (similar API’s exist for other platforms). Note that you’ll have to link in Ws2_32.lib before you can make use of Winsock. The code for opening a socket looks like this:

// #include <winsock2.h>
// #include <ws2tcpip.h>

SOCKET listenSocket;

WSADATA winsocksData;
WSAStartup(MAKEWORD(2, 2), &winsocksData);   // Initialize Winsocks 2.2

addrinfo hints = {0};
hints.ai_family = AF_INET;         // Internet Protocol (IPv4)
hints.ai_socktype = SOCK_STREAM;   // Sequenced, reliable, byte stream
hints.ai_protocol = IPPROTO_TCP;   // Transmission Control Protocol
hints.ai_flags = AI_PASSIVE;       // Listen for connection requests

addrinfo *info;
getaddrinfo("localhost", "80", &hints, &info);   //   localhost, port 80 (HTTP)

listenSocket = socket(info->ai_family, info->ai_socktype, info->ai_protocol);   // Create socket
bind(listenSocket, info->ai_addr, (int)info->ai_addrlen);   // Bind it to the address
listen(listenSocket, 1);   // Start listening for one connection

This may seem a little complicated at first, but this is just some one-time initialization code. Note that sockets are extremely powerful and you can use practically the same code for lots of other communication protocols between various applications and devices. So now that we have a socket that listens for incoming TCP/IP connections on localhost, we have to actually wait till a connection request arrives. This will happen when you open a browser and type the URL [url]http://localhost[/url], possibly followed by a path to a resource. Waiting for an incoming connection is done by the accept function:

SOCKET clientSocket = accept(listenSocket, 0, 0);

Note that this will freeze execution. Or more precisely, it suspends the current thread until a connection is accepted. So in an actual real-time application you will want to put this and the remaining embedded server code in its own thread. Note also that the accept function returns a new socket. That is to allow multiple clients to connect to a single server, each with their own independent data stream. Here we’re accepting only one client though.

Once accept returns, you can start exchanging data with the browser! You only need to know two more functions; recv and send. As the names already imply, recv will wait for a request from the client, and send can be used to reply. All communication happens using simple byte strings, and the HTTP protocol uses messages that are fairly easy to read. So let’s receive our first request message:

char request[65536];
int bytesReceived = recv(clientSocket, request, 65535, 0);

This function again freezes until the client actually sends something. But when you type for instance [url]http://localhost/panel[/url] into your browser and hit enter, the accept function will first establish a communication session and right after that the recv function will return the following text in the request buffer:

GET /panel […]

The […] is a bunch of additional information the browser decided to send us, like the browser name and language, and what formats it accepts.

So let’s reply with “Hello World!”:

send(clientSocket, "HTTP/1.1 200 OK\r\n"
                   "Content-Type: text/html; charset=UTF-8\r\n"
                   "Content-Length: 25\r\n"
                   "\r\n"
                   "<html>Hello World!</html>", 104, 0);

You should now magically see the words Hello World! in your browser!

The first line of the response message tells the browser that the request was understood and this message contains the reply. The second line tells it the reply consists of html code, encoded using UTF-8 (a popular ASCII extension that can also represent Unicode). This is followed by the length of the body. Next is another carriage return and newline to separate the headers from the body of the message. And finally we have the HTML code (not standards compliant, but most browsers will take it anyway). The send function also takes the client socket and the total message length as a parameter.

From this point forward the possibilities are almost endless. HTTP and HTML are fairly simple so I’ll leave it up to you to decide what you want to do and explore tutorials and documentation. To make things interactive you probably want to create a form with a submit buttom. This is what SwiftShader 2.0 does. Just run the demo and point your browser to [url]http://localhost:8080/swiftconfig[/url] (and have a peek at the HTML code if you’re not sure how it’s done). With a form the browser will send the server a “POST” request, containing the enabled checkboxes and selected drop down box elements. Beware that some browsers will send that in a second message. After parsing just resend the entire page with appropriately selected controls.

Note that the essential code is really just a few dozen lines, hardly any longer than for reading and writing configuration and log files. You now have all the power of HTML at your fingertips to create any interface you like. You can even change your application’s behavior remotely (try with your smartphone)!

You can also get back diagnostic information from your application. However, when you use forms you’ll have to click a button to get updated information. That can be annoying, especially when you’re waiting for a specific event to happen. Also, forms require redrawing the entire web page. So isn’t there a way to automatically get regular updates, without reloading the entire page?

Absolutely, but for this we’ll need JavaScript. So make sure you’re fully comfortable with using HTTP and HTML first. The key to dynamically update your web page is the use of the XMLHttpRequest JavaScript API. It can be used to make an HTTP request from within JavaScript, and the response can be plain text or XML. The latter case gave birth to a suite of techniques known as Ajax, but here I’ll keep things simple by using the plain text version to create a framerate counter. Here’s the essential JavaScript code for making a request:

var xhr;
function request()
{
    xhr = new XMLHttpRequest();
    xhr.open('POST', 'http://localhost/panel/fps', true);
    xhr.onreadystatechange = update;
    xhr.send();
}

Notice that we’re making a “POST” request. This avoids any browser caching. Next comes the URL for our framerate service. The last parameter to the open function specifies that we want to make an asynchronous request. This means the browser won’t halt waiting for the server’s response. Instead, it will call the onreadystatechange function when it receives something. Here we set that function to our “update” function, which I’ll explain below. Up to this point no actual request has been made yet, we’ve just set up our XMLHttpRequest object. The send function sends it off to the server. Here’s the update function, responsible for refreshing part of the web page with the data we receive:

function update()
{
    if(xhr.readyState == 4 && xhr.status == 200)
    {
        document.getElementById('fps').innerHTML = xhr.responseText;
        setTimeout('request()', 1000);
    }               
}

The XMLHttpRequest object will call the onreadystatechange function (which we’ve associated our update function with) multiple times before the complete response has been received. That’s why we first have to check the readyState and status properties to make sure that we can use the response. The plain text sent by the server will be available in the responseText field.

So we want to assign that string to a certain HTML element where the framerate should go. This can be achieved by navigating the Document Object Model. The DOM is a tree-like structure representing all the elements of an HTML (or XML) document. In this case we locate the element that has the “fps” id and we overwrite the HTML contained within that element with the server response containing the framerate as a string.

Finally, we also make the script call the request function again after 1000 milliseconds. This way you’ll see the framerate get updated automatically every second. Note that this is a ‘pull’ approach. If the sever has some updated data sooner it can’t send that to the browser before it asks for it. However, an alternative solution is to immediately make the next request, and have the sever send it when ready. Anyway, on a local network you can have hundreds if not thousands of updates per second without stressing anything.

So finally lets put it all together:

#include <winsock2.h>
#include <ws2tcpip.h>
#include <math.h>
#include <string>

class Server
{
public:
    Server();

    ~Server();

private:
    enum Status
    {
        OK = 200,
        NotFound = 404
    };

    void loop();
    std::string page();
    std::string fps();
    void send(SOCKET client, Status code, std::string body = "");

    SOCKET listenSocket;
};

Server::Server()
{
    WSADATA winsocksData;
    WSAStartup(MAKEWORD(2, 2), &winsocksData);

    addrinfo hints = {0};
    hints.ai_family = AF_INET;
    hints.ai_socktype = SOCK_STREAM;
    hints.ai_protocol = IPPROTO_TCP;
    hints.ai_flags = AI_PASSIVE;

    addrinfo *info;
    getaddrinfo("localhost", "80", &hints, &info);

    listenSocket = socket(info->ai_family, info->ai_socktype, info->ai_protocol);
    bind(listenSocket, info->ai_addr, (int)info->ai_addrlen);
    listen(listenSocket, 1);

    loop();
}

Server::~Server()
{
    closesocket(listenSocket);
    WSACleanup();
}

bool match(char *url, char *string)
{
    return strncmp(url, string, strlen(string)) == 0;
}

void Server::loop()
{
    SOCKET clientSocket = accept(listenSocket, 0, 0);
    char request[65536];
    int bytesReceived;

    do
    {
        bytesReceived = recv(clientSocket, request, 65535, 0);

        if(bytesReceived > 0)
        {
            request[bytesReceived] = 0;

            if(match(request, "GET /panel "))
            {
                send(clientSocket, OK, page());
            }
            else if(match(request, "POST /panel/fps "))
            {
                send(clientSocket, OK, fps());
            }
            else
            {
                send(clientSocket, NotFound);
            }
        }
    }
    while(bytesReceived > 0);

    closesocket(clientSocket);
}

std::string Server::page()
{
    return
    "<!DOCTYPE HTML PUBLIC '-//W3C//DTD HTML 4.01//EN' 'http://www.w3.org/TR/html4/strict.dtd'>     \n"
    "<html>                                                                                         \n"
    "    <head>                                                                                     \n"
    "        <title>Panel</title>                                                                   \n"
    "    </head>                                                                                    \n"
    "    <body>                                                                                     \n"
    "        <p>FPS: <span id='fps'>-</span></p>                                                    \n"
    "        <script type='text/javascript'>                                                        \n"
    "            var xhr;                                                                           \n"
    "            request();                                                                         \n"
    "            function request()                                                                 \n"
    "            {                                                                                  \n"
    "                xhr = new XMLHttpRequest();                                                    \n"
    "                xhr.open('POST', 'http://localhost/panel/fps', true);                          \n"
    "                xhr.onreadystatechange = update;                                               \n"
    "                xhr.send();                                                                    \n"
    "            }                                                                                  \n"
    "            function update()                                                                  \n"
    "            {                                                                                  \n"
    "                if(xhr.readyState == 4 && xhr.status == 200)                                   \n"
    "                {                                                                              \n"
    "                    document.getElementById('fps').innerHTML = xhr.responseText;               \n"
    "                    setTimeout('request()', 1000);                                             \n"
    "                }                                                                              \n"
    "            }                                                                                  \n"
    "        </script>                                                                              \n"
    "    </body>                                                                                    \n"
    "</html>                                                                                        \n";
}

std::string Server::fps()
{
    char buffer[256];
    sprintf(buffer, "%f", 45.0 + 15.0 * sin(timeGetTime() / 1000.0));
    return buffer;
}

void Server::send(SOCKET client, Status code, std::string body)
{
    std::string status;
    char header[256];

    switch(code)
    {
    case OK:       status += "HTTP/1.1 200 OK\r\n";        break;
    case NotFound: status += "HTTP/1.1 404 Not Found\r\n"; break;
    }

    sprintf(header, "Content-Type: text/html; charset=UTF-8\r\n"
                    "Content-Length: %d\r\n"
                    "Host: localhost\r\n"
                    "\r\n", body.size());

    std::string message = status + header + body;
    ::send(client, message.c_str(), (int)message.length(), 0);
}

int main()
{
    Server server;

    return 0;
}

You’ll have to link with Winmm.lib for the timeGetTime function. This application produces a sine shaped “framerate”. Just run it and open [url]http://localhost/panel[/url]. In this case the HTML code does pass conformance validation. Also, if the server gets an invalid request it will respond with status code 404. This may occur when some browsers make a request for the vaficon.

That said, this code may still not work under all circumstances. The XMLHttpRequest API has only recently become available on all popular browsers, so you may want to update your browser to the latest version or look for backward compatible implementations. I also didn’t add any kind of error checking, so if you need this to be more robust check the documentation of the used APIs. And of course you may want to extend this with convenient methods to extend the HTML code with certain controls and asynchronously updated diagnostic information. You can even update images to create things like graphs…

Either way I hope I’ve convinced you that with relatively little code you can get access to a wide range of possibilities to diagnose and control you application with a neat interface. Please let me know if you use this anywhere, or if you know any other web based technology that is useful during development!

Cheers,

Nicolas “Nick” Capens

12 Replies

Please log in or register to post a reply.

99f6aeec9715bb034bba93ba2a7eb360
0
Nick 102 Nov 03, 2009 at 15:28

A big thanks goes out to the people in the Framerate in a browser thread for helping me get started with asynchronous updates!

6aa952514ff4e5439df1e9e6d337b864
0
roel 101 Nov 03, 2009 at 15:52

Very inspiring, thanks!

Offtopic: I never bothered to do more Javascript than an occasional assignment for my study, but I’m very surprised that it allows (or forces?) things like “xhr.readyState == 4”. Now that is ugly!

2b97deded6213469bcd87b65cce5d014
0
Mihail121 102 Nov 03, 2009 at 20:01

Oh my God, so application analysis is what it was all about :> Although I really like the nice tutorial on implementing quick-and-dirty dynamic web servers I really don’t think it’s worth it given that creating another form with .NET (for example) takes two lines of code. As always though, you manage to demonstrate a very large number of concept within a few lines and you do it very clearly. And I’m serious on it, it’s not just blabla nice job blabla.

99f6aeec9715bb034bba93ba2a7eb360
0
Nick 102 Nov 04, 2009 at 14:01

@roel

I’m very surprised that it allows (or forces?) things like “xhr.readyState == 4”. Now that is ugly![/sub]

Yeah, the specification defines symbolic names for these constants, but no browser actually implements them. I don’t think it’s really that ugly though. Whether it’s a number or a symbolic name you still have to look at the documentation to write the correct code. Also, you could define the symbolic names yourself, but what exactly would you win by doing so?

You may want to add a clarifying comment though…

99f6aeec9715bb034bba93ba2a7eb360
0
Nick 102 Nov 04, 2009 at 15:45

@Mihail121

Oh my God, so application analysis is what it was all about :>

Actually it can be used for more than that. For instance you could create a test suite that can be accessed from all over the world (checking status and starting new test runs). And I’m sure there are many other creative uses.

But yeah, personally I’m using it to analyse and control real-time applications. :happy:

I really don’t think it’s worth it given that creating another form with .NET (for example) takes two lines of code.

Sure, in many cases that’s a quick and easy solution. But it has some limitations. First of all you need a visual framework like that, which isn’t always an option. Also, it wouldn’t work remotely (or at least not with two lines of code). And when you’re creating just a component instead of a complete application you often lose control over keyboard and mouse input.

2b97deded6213469bcd87b65cce5d014
0
Mihail121 102 Nov 04, 2009 at 16:14

@Nick

Actually it can be used for more than that. For instance you could create a test suite that can be accessed from all over the world (checking status and starting new test runs). And I’m sure there are many other creative uses. But yeah, personally I’m using it to analyse and control real-time applications. :happy:

Yes, it certainly adds another perspective to browsing and abstracts the idea of “what” is being browsed. In that sense, it’s an innovative and therefore interesting work. Besides, it’s implementable with a few strokes.

Sure, in many cases that’s a quick and easy solution. But it has some limitations. First of all you need a visual framework like that, which isn’t always an option. Also, it wouldn’t work remotely (or at least not with two lines of code). And when you’re creating just a component instead of a complete application you often lose control over keyboard and mouse input.

I was about to point at X11 before reading the “not with two lines”. After reading, I agree, although technologies are evolving fast and it’s certainly possible (using a decent framework) to achieve the same although it will not reach that many people as your solution. I’m personally interested in trying it out with Lynx. :w00t:

6aa952514ff4e5439df1e9e6d337b864
0
roel 101 Nov 04, 2009 at 16:32

@Nick

Yeah, the specification defines symbolic names for these constants, but no browser actually implements them. I don’t think it’s really that ugly though. Whether it’s a number or a symbolic name you still have to look at the documentation to write the correct code. Also, you could define the symbolic names yourself, but what exactly would you win by doing so? You may want to add a clarifying comment though…

Ok: I skimmed through your code, and thought: “what on earth could these weird numbers mean?”. Then I read the documentation. If the line would have been something like if (xhr.readyState == XMLHttpRequest.DONE && xhr.status == HTTP.OK) I’d probably not even noticed that line.

Ceee4d1295c32a0c1c08a9eae8c9459d
0
v71 105 Nov 04, 2009 at 17:17

I see this as a tremendous debugging utiliy, you run your engine on the main pc and with an internet connection you can debug on another pc connected trhough the net , both local or global, brilliant…

99f6aeec9715bb034bba93ba2a7eb360
0
Nick 102 Nov 23, 2009 at 16:22

Writing the HTML ‘page’ as a string can be a little annoying. You have to add quotation markts to every line and you have to run the application to see the result of every change. So here’s a little addendum to try and facilitate that:

Using Visual C++, you can easily add resources to your project, so instead of writing the HTML as a string just store it as a resource. This way you can easily view and edit it within the IDE. The code to locate and load the resource looks like this:

HMODULE module = GetModuleHandle(0);   // Current process
HRSRC html = FindResource(module, MAKEINTRESOURCE(IDR_HTML1), RT_HTML);   // Locate resource "IDR_HTML1"
HGLOBAL resource = LoadResource(module, html);   // Load it into memory
const char *data = (char*)LockResource(resource);   // Get raw data pointer
int size = SizeofResource(module, html);
std::string string(data, size);

send(OK, string);

Note that you can also always load the HTML from a file, but the above approach has the advantage that it’s still completely embedded so you don’t have any additional files floating around. The resource is always there.

The only disadvantage compared to construcing the page directly as a string is that it’s not that straightforward to create dynamic content on-the-fly. However, you can easily just add placeholders that you can search for and replace once you’ve loaded the HTML resources into the string. One suggestion is to use the <%…%> tags, also used by ASP, which get ignored by the browser. Note that ASP, as well as PHP, are powerful server-side scripting languages meant exactly for generating web pages dynamically.

28a3d09a07d47c82382fea33a37cfb00
0
cknopp 101 Feb 20, 2010 at 16:54

Please forgive this total n00b question, but would it be possible to use this technique to add a simple GUI to a program like OpenFOAM? Since I use all Win7 OS’s on my computers, I was looking at picking up a used server rack (local high-school sale, 10 quad core blades) and installing the Linux OS. If I could use this great technique to access the server cluster with a GUI to run simulations over my LAN, it would be great!

Heck, at that point, I may even be able to allow others to log in and use my server remotely!

Please, if you think this is a possibility, you can email me directly to ask any questions or give any advice!

THANK YOU!

Chris

cknopp@gmail.com

31541bc66110f7a0f9b4fece3274b27f
0
afton 101 Feb 22, 2010 at 06:50

hey If I use this technique as well get the server cluster with a GUI to run simulations over LAN, would it be great?

99f6aeec9715bb034bba93ba2a7eb360
0
Nick 102 Feb 22, 2010 at 12:34

@cknopp

Please forgive this total n00b question, but would it be possible to use this technique to add a simple GUI to a program like OpenFOAM? Since I use all Win7 OS’s on my computers, I was looking at picking up a used server rack (local high-school sale, 10 quad core blades) and installing the Linux OS. If I could use this great technique to access the server cluster with a GUI to run simulations over my LAN, it would be great!

Absolutely! Lots of sever applications have web interfaces. And as I tried to make clear it’s not all that difficult to get started.

Heck, at that point, I may even be able to allow others to log in and use my server remotely!

Sure, but beware that doing the opposite, making sure certain people do not get access, is more challenging.

Please, if you think this is a possibility, you can email me directly to ask any questions or give any advice!

The best advice I can give is to get coding. ;) Get the above code running (under Windows), modify it for Linux, and then integrate it with OpenFOAM.

Good luck!