I’ve updated my MLDB JavaScript database driver for MarkLogic! New search functionality included, but mainly the hot news is that MLDB now runs within a browser app! Also many widgets are being released today. Read on for more…
As many of you may know I developed my MLDB JavaScript REST Server wrapper API in order to power a Node.JS situational awareness application I was developing. It occurred to me at the time that this same API could be used within the browser to simplify the task of application developers who need to build a new application on MarkLogic.
I’ve now gotten around to allowing MLDB – the same JavaScript file – to be used within Node.JS as well as within the browser. All the underlying connection and ajax call work is done for you – you just need to call a function and pass in a callback function.
Of course we already have some widgets as part of our application builder tool in MarkLogic. Customising this requires you to know XSLT and do things like customise search result output. I got to thinking though that an average web developer new to MarkLogic will definitely know HTML, CSS and JavaScript. They are familiar with the concept of Widgets thanks to libraries like jQuery and Prototype.js. They aren’t familiar with altering an existing MarkLogic application – event one built with app builder – and they certainly don’t want to spend a lot of time in a complex application editing an XSLT file.
I’ve designed the MLDB Widget API to be very lightweight and easy to use. Coupling between components is kept to a minimum – in fact everything uses a lightweight event subscribe/unsubscribe/publish model, passing around REST API JSON documents – so you can even mix and match a RAW REST app and an MLDB widget.
So what have I developed? Well, as this page gets published I’ll be presenting a 15 minute lightning talk at MarkLogic World 2013 in the Mont-Royal 2 room on the fourth floor of The Cosmopolitan Hotel at 1330 local time. Please come along to chat about this to me.
What’s new in MLDB Core?
Structured search has been added to the MLDB object, along with persisting or loading search options to/from the server. This makes dynamically creating an interactive application very easy. I’ve also supported the /v1/values endpoint in the REST API. This allows you to lookup lexicon values or view co-occurence results and frequencies (see widgets below). There’s a whole bunch of other API functionality which you can read all about on the Core API WIKI Page. Logging has also been improved.
The Core API also dynamically checks whether it’s working in Node.JS or a Browser, and automatically configured the underlying connection wrapper. Browser wrappers include jQuery, Prototype.js and the browser’s own XMLHttpRequest object. The jQuery one is the most tested, but I want to be as agnostic as possible to encourage maximum use.
I’ve also included a few functions to convert from XML documents to JSON documents much like you can do on MarkLogic on the Server side. This is for instances where the server configuration cannot be influenced by the application developer, so that MLDB can handle this transparently. You can use these in your own custom widgets too if you wish.
There have been a whole host of other bugfixes though, so if you’ve tried the API before and found it lacking (bah, unlikely!) then please try it again and let me know. Log any issues you may find.
What’s new in MLDB Widget API?
Everything, it’s brand new! I’ve created a few widgets to showcase what can be done. These are vanilla HTML + JS so they’re really simple to integrate to your REST Server application. Here’s the list:-
- Error widget – for connection errors, search option errors, on screen
- Simple search bar using MarkLogic search grammar
- Search Facets
- Search Results
- Results paging control
- Results sorting control – driven by the JSON search options
- Search page – combining all of the above on to a single page (you could even have multiple of these if you wished!)
- MarkLogic HighCharts widget – extract fields from JSON search results and combine them using aggregation functions on the client to display charts
- Co-occurence – using MarkLogic’s co-occurence features over range indexes, view the most popular co-occurences – E.g. Actor vs. Genre of Film in same document, across all documents (totalled by frequency)
- Google Kratu table widget for visualising search results
Prove it
Below are a set of screenshots with explanation text to show you what has been developed.
This shows the searchpage widget in use. I’ve linked all range constraints defined on the database, as you can see in the facets. I’ve selected the Animals facets, so any empty facet sections are automatically hidden. You also see that the sort options are driven by the underlying search options JSON definition. Building these search options is easy using our search options builder.
Also shown somewhat subtlety is the uber cute Hurt-Bat. Awww bless! This makes the point that I can combing JSON results and XML results, and register custom plugins to the search results widget to render the results. For SVG images I’m showing a small version of the image, and the URI. For an Animal JSON document I’m showing the name of the animal, and a custom sentence combining the name, family, and summary fields from the documents.
Paging is also catered for, as is editing the search bar. Each of these widgets is loosely coupled through event handlers and listeners, but they do affect each other. So clicking on a facet alters the search bar text, and re-executes the search. Same with sorting. Also, editing the collection facet in the search bar will also update the facets widget. Cool huh?
The above shows the co-occurence widget configured on to different co-occurence definitions, within the same search options object. There are many documents with movie title, year, main actor, and genre listed. I’ve used this data and the range constraints defined on them to generate co-occurence results of Actor vs. genre, and actor vs. Movie Year. Very quick, very simple to do.
The above, very long, image shows 8 instances of the HighCharts widget running of the same structured search results object. This is executed on page load, and drawn dynamically. It also shows using aggregations, including mean averages – so if multiple temperature results exist on the same month, the widget has been configured to average them together using a mean. All the configuration options on HighCharts can be accessed.
The above image shows the Google Kratu widget linked in to the JSON content of a set of search results. Very quick way to display arbitrary JSON content
In Summary
MLDB now exposes much more of the REST API. By version 1.0 I will support 100% of the REST client API for both MarkLogic 6.0 and 7.0 (when released!) This can be used within either a NodeJS or Browser application to store, retrieve, and modify content, as well as access advanced functionality like geospatial search and co-occurence. I’d recommend this to build any custom server, like Jabber clients or HTML5+websockets apps (I’ve done these internally!)
The MLDB in the Browser change, along with the widgets I’ve created, will allow you to develop a REST application in just JavaScript and HTML, with the odd splash of CSS (although I have sensible defaults). You can create a search page in just a few lines of JavaScript code. You don’t have to be familiar with the inner workings of MarkLogic to do this – using the helper objects and MLDB core API abstracts out all the underlying communications and REST endpoints, allowing you to concentrate on creating your applications and not writing plumbing code.
How do I get it?
For Node.JS simply do a npm install mldb to get the 0.6 version, right now. For a browser app you can download the tar.gz or the .zip distribution files. You can also clone the MLDB repository to get a Roxy hybrid application allowing you to quickly deploy a new application, and create the REST instance, content and modules database without coding anything – check out the mldbwebtest folder for the example app with the above pages configured on /mldbtest/search etc. URLs.
Happy application development!
Copyright notice
The Hurt Bat image is licensed under Creative Commons here is it’s reference in full: By Ecelan (Own work) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0-2.5-2.0-1.0 (http://creativecommons.org/licenses/by-sa/3.0)%5D, via Wikimedia Commons
It can be downloaded here: http://commons.wikimedia.org/wiki/File:Murci%C3%A9lago.svg
Adam's Deep Technology Blog 