I’ve been developing MLDB in isolation for the last couple of months, so I thought it would be useful to put a roundup of what I’ve delivered in that. I also thought it may be instructive to show what you get out of the box in MarkLogic 6’s Application Builder too.
App builder as we often call it is a wizard driven web based tool for getting up and running quick with a search application on MarkLogic. Once you have configured your indexes and loaded content (probably using Information Studio’s wizard tools for that), you use App builder to build a quick application on top of it.
The process is quite straightforward – you review the indexes on your database given them readable names, you define the facets available for your application, you specify how search result summaries should look, then you say which widgets (high charts graphs, google maps) you wish to use, then you preview the layout of this. When done, you deploy the whole app to a new REST API server that the wizard can create for you.
Here’s a screenshot of the search config and indexes page in App Builder for some publicly available NHS prescription information:-
Also the layout page showing a couple of widgets, and one of these widget’s configuration:-
This can be used to create a working, valuable search application like the one below that shows a chart of the total number of prescriptions across a range of prices:-
Widgets in app builder
The App Builder provides a standard set of search functionality – search bar, facet navigation, sorting, results pagination, and the result list itself. It also allows you to embed 1 or 2 visualisation widgets between the search bar and the results information. These can be HighCharts based graphs or google maps, with heatmaps. The data for these visualisations is gathered from aggregations defined in the search options or from facet information.
This search page is a great starting point. There are a couple of downsides. The first is that App builder doesn’t yet support Path Range indexes that were added in V6. The second is that the use case is for a search app only – there’s no support for getting the data in or doing something with the search results (E.g. download an Excel summary)
Customising app builder
Adding customisations is supported within a ‘custom’ folder, but unless you know what you are customising exactly it’s hard to get your head around. The search results are rendered using XSLT, and interaction is handled in a complex controller.js file.
This is an overarching widget containing several other widgets. Below you see the default layout. This includes a search bar at the top, facets on the left. You see for the results a results pager, sorter and the results themselves. These are clickable too if you provide an app URL to handle navigation in the results widget.
This page is intended to be very similar to the App Builder default app. How results are rendered is customisable – for example in the above most animals are JSON documents, whereas the bat is an SVG XML document. These renderers are pluggable at runtime.
I can also link a chart to the results. At the moment this is limited to only the search results on the page because the graphs are linked to JSON docs’ contained properties rather than to search values or facets summaries. I’ll change this in future though. I’ll also allow embedding in the search-page widget too. Below is a couple of charts linked to a search bar as a simple example.
There are a host of different charts available. Below are a few examples. MLDB is a thin wrapper over the HighCharts library that comes with MarkLogic, so any chart you can create should ‘just work’. Famous last words. Here are a few examples:-
In this example I’ve put in movie information. Two co-occurences are provided – one of main star against movie type, the other of star against year of release. You see here that Jim Carrey mainly works in comedy, whereas Sean Astin has worked in a variety of films (Goonies, LoTR, and recently in an episode of NCIS!)
File upload (Docbuilder)
One of the things we tend to overlook in pre-sales is the addition of new documents. We tend to bulk load sample docs using command line tools. Fine for a demo, but not great for an actual end user on a production system.
I’ve added a docbuilder widget. The idea behind this is that you can build up a HTML form to create a new document by adding fields and linking them to XML or JSON fields. All from simple function calls on a widget object – no nasty form code required. At the moment though I’ve only had time to add basic file upload and permissions controls – so you can specify a single file, it’s URIs pattern, collections, and it’s read and update properties. All works fine and dandy:-
Markings / adding facts (triples)
Something that may occassionally be useful is embedding security markings within an XHTML document. Maybe most of a report can be read by some people, but the section detailing next steps is classified at a higher grade. You can do this in MarkLogic by using an XQuery transformation on all get and search operations. This effectively redacts sections on the fly. Very neat feature.
In MarkLogic 7, which is currently in Early Access, we support storing triples in the server. These are facts in subject-predicate-object triples. E.g. Adam Likes Cheese. Getting to a point where these are extracted and linked is non trivial. I decided to use any entities identified in the XHTML (E.g. by Temis, Smartlogic or by a simple DB trigger in my examples) and use their existence in the same paragraph to suggest triples.
So from the above you see that Abraham Troublemaker and Animal Freedom are mentioned in the same paragraph, so my code suggest that A Troublemaker is a member of the Animal Freedom organisation. The user can remove, modify or add to these facts. Clicking on done adds security to the relevant document sections, but also stored the facts in to MarkLogic, and maintains a link from those fact graphs back to their originating document and section.
SPARQL (facts query) widgets
Having triples stored is one thing (see above), but querying them is another. There is a W3C standard called SPARQL. This is kinda like SQL but is a lot more flexible. Complex queries may be hard to understand though. I’ve created a widget so you can define these queries using object hierarchies. In the below example I search for any people that are members of an organisation with the name Animal Freedom.
As you can also see, you see a summary of results. Clicking on one of these views all the facts for that entity in the triple store. An extension I’ve not yet done is clicking a ‘related content’ button on this facts list so you see the list of documents that these facts were inferred from. Each of these three is a separate widget (Indeed the content results widget is the same one as on the search page example, higher in this article – showing re-use of widgets).
It is sometimes useful to view a collection’s child collections. For now I’ve created a simple list. A tree structure could also be useful. No image yet, sorry! It’s just a simple list of names anyway right now.
Kratu is a handy little widget for visualising JSON results – it can take any results and view them in a tabular format. I’ve created a thin wrapper widget that links JSON search results to Kratu. You can choose either the documents’ content (default) or the search parameters (URI, match score for each doc – for example)
I’m currently working on a set of widgets for Records Management using the Document Library Services (DLS) functionality of MarkLogic. I can already declare search results are records. I will extend this to support listing of RM collections, and RM rules.
The world is your oyster when it comes to UI – and everyone’s needs are different. I have found it best to be very flexible in providing a granular set of widgets that can be used together using a common data model (MarkLogic REST JSON objects). I have already created a set of apps much quicker using re-usable content rather than coding from scratch in XQuery. I can then spend my time on novel functionality for each of our clients.
If what you want is a quick search app over new data, then try App Builder. This will get you a visualisation quickly at the cost of long term extensibility later on.
All the above functionality is available for MarkLogic 6 except any semantic/triple functionality which requires ML 7 Early Access. You can get MLDB from here. Read the README text at the bottom of the page.