Every new platform has a killer application that sets it out from the crowd. Situational Awareness, I believe, is the killer app for NoSQL databases with real time alerting, like MarkLogic. In this post I explain what Situational Awareness is, how it is different from what has come before, and why MarkLogic is uniquely suited to solve this type of business problem.
What is situational awareness?
It has long been a mantra of Enterprise Software companies that they will deliver to your employees the right information at the right time, so they can make better decisions, faster. Situational Awareness takes this concept further by bringing together both internal (Enterprise) data and external data. This may involve some data transformation, analysis or aggregation. This results in useful information (or intelligence) rather than a data stream, which can be immediately acted upon. The secret sauce within all of this is that this information is directed to those individuals or systems that need it in as near to real time as possible.
Practically this means providing an individual with everything they need in order to function on a minute by minute basis. This could be in Air Traffic Control, a Nuclear Power Plant or the Battlefield. Anywhere that the perception of the environment is time critical. This differs from Business Activity Monitoring or Business Intelligence in that it is much more tactical rather than strategic, and the information flow and environment is changing all the time. Also, the information presented doesn’t just allow a better overview of the situation, but rather is essential for an accurate analysis of the environment and current situation to take place.
How is it different from searching / BI / BAM?
As I’ve already mentioned, Business Intelligence (BI) provides a longer term, strategic view of a situation. It may also allow you to ask questions like ‘What if we had done X instead of Y?’. It is very much intended and designed for a post-game, historical querying approach. Business Activity Monitoring (BAM) on the other hand gives you an up to date summary of the current state of play. It is typically used to measure a team’s performance against a set of Key Performance Indicators (KPIs). It may also be interactive and allow for re-tasking and alerting when particular KPIs are exceeded. A typical example is that of your team’s average work completion time, or pending work items. If a persons work queue goes over 20, say, then the manager is alerted or work is reassigned dynamically by the BAM tool. Although this tool could be accessed by those completing the actual work, typically it does not provide them with any information that would allow them to complete their work quicker, but rather information on how well they are doing completing work. BAM is inward looking.
Searching on the other hand allows an end user to access and explore information. If a MarkLogic database is used that receives streamed data in real time, then the searching can be over all information as it arrives. This is because unlike traditional combinations of database and search engine, MarkLogic indexes information on commit – you don’t have to wait for the search engine index to catch up. This is unique to MarkLogic. This gives users access to up to the minute information they need to carry out their work. This could be internal information or external feeds. Typically it’s a combination of both.
Where Situational Awareness applications differ is in the experience of consuming that information, and the aspects to the information your queries are based on. As we’ve already seen, time and location are key to Situational Awareness applications. This is inherent to everything you do. Perhaps you’re receiving positional readings from a vehicle/robot. You’ll only want the latest reading. Maybe there are many robots. You only care about those in a particular area that are performing a particular function. Every query is time and space related. Typically this will be done transparent to the user. The application will ensure that only current information in that user’s area of interest is shown.
The user interface is also typically drastically different. It will need to be tailored specifically to the requirements of the users. Perhaps this is a map based display with overlays for current readings. These could be icons, or heatmaps, or contours – but always visual. Perhaps instead there is a head up display or camera phone or tablet showing an overlay of information. This would use the phone’s camera to determine direction and show the base layer – the current world. All information will be overlaid. You’ve seen these apps already – showing what direction and distance the nearest donut shop is in. Situational Awareness apps do this, but with live streaming data. So maybe the level of pedestrian traffic or road closures overlaid, or the number of donuts left at each nearby shop.
What is needed to power situational awareness?
A user will not type in a search like a traditional application. They will want alerting to all information of interest by the system, and this will be displayed as it arrives, in real time. A user of a Situational Awareness application neither knows nor cares that a database with search underpins their application. An underlying database that is collating information in real time is vital. An alerting capability is also key. More importantly, only alerting the right information to the right people. This is where I believe MarkLogic has an edge. MarkLogic’s reverse queries and alerting allow you to use all of the advanced searching capabilities to specify what information is of interest to you. The database then itself matches incoming documents against this, and when they arrive alerts you to their presence.
You may also need to transform incoming information to different forms. Perhaps you need to extract or enrich incoming data. This could be to identify key terms such as people, places and organisations. It may be to associate additional information with these, such as longitude and latitude with a place name. Perhaps you need to aggregate this new information with recent similar reports to determine whether an event has happened, or is imminent, prior to alerting. Perhaps you’re doing this analysis to see if the information is worth keeping in the database at all. This information triage is common in scientific communities where Petabytes of information is being received, but is hard to store and analyse. Certainly aggregation prior to alerting is of use where individual readings or reports are in and of themselves insufficient to form a decision.
You then need to get the information to the right people or systems. Systems are easy as they are typically always turned on. People are more difficult. They are on the move on their local machines or devices. You need some mechanism for them to login and say ‘send all my alerts here’. This can be accomplished using a HTML5 app built with websockets. These always keep a connection open, and are ideal for funneling alerts down. The traditional model instead would have that user poll the server every few seconds or minutes for updates. This is computationally intensive on the database, and eats bandwidth on the client.
You then need to be able to display the information. How you do this really depends on the application. Typically it will be visually rich. This could be a 3D display much like military computer games you can buy. (I always liked Total Annihilation myself!) Perhaps it’s a map overlay. There are many excellent mapping libraries out there that will make creating this pretty easy. Maybe it’s a head up display within a set of special glasses. It could even be an audio feed – “He’s behind you!” blaring out in your ear perhaps.
What can be built?
Really the only limit is your imagination. These applications will only be adopted if they work the same way the people doing the job do. You can’t take their concentration off of their job and on to your software. It must enhance their experience and how immersed they are in the task at hand – not distract them from it, no matter how information rich it is. Below are a few ideas I had. Perhaps you know of other examples? Please add them to the comments section, below.
- Social app taking feeds from your twitter and facebook accounts to determine where your friends are. Wondering why someone is late? Open your tablet and point it in the direction of their house. You may see that they have tweeted “I’m late” from “Doncaster Road”. This could be overlaid with a walking time estimate from their current position.
- Live flood alert map showing overlays of current met office weather station information of rain fall. This can be matched with historical information of rainfall and duration and when floods happened in each area to give a live view of where flooding is about to occur.
- Social information analysis of intent overlaid by topic and location. This gives a live sentiment map. This has already been done by companies like SGI and their DataRaptor with MarkLogic product.
- Land monitoring application. See how your agricultural land is doing. How moist is the soil, how sunny is it, temperature overlaid with last feed and sowing information to show any issues live with your crops across a large area, and predict when crops will be ready for harvest based on previous years’ data
- Defence applications – overlay current information on people, resources, and live sensor data and intelligence reports. Alert those in the area to relevant information as it arrives
- Astronomy – just like in Star Trek, show a 3D room size visualisation of star readings as they come in during an evening’s observations. Have this interactive so you can zoom in to a particular area. See 3D representations of your observational data, and be able to use this to find and highlight the underlying complex data from observations.
Adam's Deep Technology Blog 