I’ve been playing around with concepts to alert users in real time of information they need to do their jobs. In this post I describe what I’ve achieved so far.
There is a variety of technology available to display information. These days it is more and more common to use web based systems as a primary system interface. These are built on XHTML and JavaScript, typically, and are on the whole pretty good. Where they fall down is in how agile they can be. Having an entire page refresh – some of which is just headers and footers, or navigation menus, takes time – for both networking and processing on server and client. A good way to resolve this is to use RESTful service calls via AJAX. These can be to fetch data for local processing and display (typically via JSON) or through rendering just part of a page (As I’ve previously called HTML Fragments).
Now you have built an AJAX powered, uber fast web interface you need to get hold of the data. One of MarkLogic’s strengths is in alerting. This means you can register a search as a reverse query. So rather than say ‘Give me all tweets that are sent from, or mention, Surrey’ you instead say ‘Tell me when a new tweet arrives that are sent from or mention Surrey’. This means you can go away and do some other work and be alerted when new data arrives. This is great when MarkLogic is used to alert some other system, such as a Business Process Management (BPM) system, but falls down when it comes to a web interface.
This is because HTTP is not built for server to client communications, but rather client requests and server responses. There is no way for a web server to ‘reach out’ to a user’s browser. There are ways around this such as polling. JavaScript has long had the ability to poll a web server with a request. Say, every 5 minutes. This may well be fine for most things, like receiving email notification every 5 minutes rather than when they arrive, but is not ideal in more reactive organisations. This could be trading fraud where billions of pounds are at stake, or in the defence and intelligence space.
Thankfully, HTML5 and it’s associated standards has a solution for this. This is an extension to HTTP called WebSockets. This is a connection that remains open between the client and server. Keeping the connection open means the server has a conduit to send information to a browser. Typically this will be received and actioned via JavaScript. This is ideal for MarkLogic’s uses. We can fire off a request to a client. The only downside is that currently the MarkLogic app server does not support WebSockets.
Fear ye not, there is a solution! Ye olde faithful Node.js is there to help! This is a very, very lightweight event oriented server architecture. Designed to be low latency and low on its IO use, a Node.js server can be constructed that has an app server waiting for an alert to be fired from MarkLogic. This alert is then sent down the conduit between the client browser and Node.js. This is possible as Node.js has a few API available that allow the construction of a WebSockets server.
Having Node.js as a middle tier sounds like extra complexity, but actually it could make client code neater. You can have many user interfaces ‘login’ to your WebSockets server. This in turn may register itself as a listener on behalf of the user, and set up the alerts in MarkLogic automatically when a user logs in. This removes MarkLogic REST code from the client browser, making that interface more responsive and simpler. It also means you can control exactly the communication from your MarkLogic server to any clients. This separation is desirable in high secure or very remote client installations.
Hopefully this has given a flavour. I have some sample Node.js code for such a server. If you’re interested in seeing it, then leave a comment!
Adam's Deep Technology Blog 