Just a quick summary of what I’ve been up to since preparing for my session at MarkLogic World, and subsequently, including semantic widgets…
I’ve built various improvements for the version 0.6 (April 2013) build of MLDB – my JavaScript MarkLogic API. The main one was around getting MLDB working directly within the Browser. This enables developers to now built web apps in JavaScript and HTML only, with no middleware layer, directly against MarkLogic.
Browser access
I’ve re-worked the internals of MLDB so that you can use it directly in the browser app. This involved checking whether the app was running in a browser or not. I just check for the presence of ‘window’. I’ve written several wrappers for various AJAX libraries to handle communication. This includes jQuery, Prototype.js, XMLHttpRequest and, more recently, XMLHttpRequest2. I’ve tested the jQuery one heavily as Roxy (the framework I use to create REST apps) comes with it built in. Unfortunately, it makes file upload of binary files a nightmare.
I’ve recently moved to XMLHttpRequest 2 as most browsers support it these days. This makes binary file upload a breeze. had to add a check to enable JSON upload though, weirdly, but that now works well. This wrapper seems to work a factor of times quicker than jQuery too, which is interesting in itself. I recommend this wrapper for MLDB now.
Browser UI Widgets
The natural step after ensuring the library can be used in the browser, is to create some reusable UI widgets. I had developed the following for MarkLogic World:-
- Search bar, pager, results, sorter, facets – and wrapper for all these in a single page
- Co-occurence widget
- HighCharts graphs based on search results
- Google Kratu for JSON result visualisation
I’ve now added to this:-
- List collections underneath a parent collection
- Security markings within XHTML content (as well as at content level), highlight known objects (people, places, organisations), and suggest facts based on proximity of these things, and save to triple store
- Doc builder widget – Forms creator helper widget. Currently includes file upload and permissions controls. Will expand so you can create any XML or JSON doc visually in a few lines of JS
- Document Library Services (DLS) – our content versioning and purging/holds library. Currently includes widgets for declaring search results as named record group, listing record groups, and listing documents within a group
These are all re-usable, and you can add multiples of each to a page if needs be.
Search Helper classes
People have commented (correctly!) that building our search options documents and structured searches is a complex thing to do. I’ve found this too. I’ve now completed (yes, completed!) a fully featured search options builder. This has sensible defaults and supports all the options within the search options configuration. This can be created from an MLDB instance by doing new db.options(). You then call methods on this.
I’ve also started a structured search builder. This currently supports basic range query options and geospatial search features. Eventually this will support all common options. There are more configurations possible in the query than in the options, so I expect to be adding these over time as new scenarios appear.
Semantics
MarkLogic version 7 Early Access was released a month ago. I’ve started taking a look at this, and the REST API extensions, to build a real world scenario. As I’ve mentioned above, the document markings widget will extract known XML tags (configurable) and link these E.g. <placename> to the #name field of a placename IRI. This enables me to suggest triples based on identified entities within a document, and their proximity. This makes semantically enriching your content a breeze.
I’m currently working on a sparqlbar like a searchbar widget. This will enable you to select what you want to find, and provide basic search terms. So you would select ‘Any Entity’, ‘Facts’ or configure your own like ‘People’, ‘Places’, ‘Organisations’. Below this you will be able to add one or more conditions. So choose a relationship class and select an entity IRI (interactively) that it must link to. The resultant list will be based on the /v1/graphs/things URI but be presented in a consistent style, and browseable interactively.
Eventually I’ll update that so you can nest conditions. So you can say ‘Where person is a member of an organisation, and that organisation type is X’. I’ll also support drag and drop query building, and saving of sparql query configurations. Much like we do at the moment for content searches.
Eventually I want to merge the content search world with the semantic search world. So if you search for matching things, you can view both facts and content related to those things. Conversely, when you search for content, you can see selected facts about entities within that content.
Documentation
Based on feedback from our own Eric Bloch, I’ve been busy documenting every method and object in the core and browser APIs. Not a small task! I’ve also built JSDoc documentation. This is all now available on GitHub Pages. This will be the source for all documentation in the future. It currently includes API doc as well as code samples and some tutorial links to my blog. I’ll continue to add more as I go.
It’s worth mentioning that the GitHub project, as well as the mldb-browser downloads, include samples page code named page-*.js. These can be added to a REST API html page to see samples for the widgets I’ve built. You can just download the Roxy app though and install that to get all the samples working.
Universality
I’ve done some work to support non JSON documents. The search results now support XML results returned in a JSON search. You can also do a save() and a get() on XML and binary documents and it works fine. I may add more functionality as and when I need to. I won’t support internal calls in XML as well as JSON though – if MLDB and the widgets are hiding the underlying mechanism anyway, I don’t see why support it. I’ll only support XML in information you are storing and retrieving from MLDB, not for things like search options, structured queries, and the like. (Aside from alerts of course)
General Bug Fixes
I’ve found a host of issues, mainly around handling different document types, and fixed them. Unfortunately my Node.JS test framework is broken. I’m probably going to replace this with Mocha so I get better output anyway. I’ll spend some time on the Node.js version in the next month. I’ve received some great feedback and suggestions I want to incorporate in to the Node.js version.
What the future holds
There’s some tidying up I want to do. I may move the mldb-xhr2.js code in to mldb.js itself to make it more obvious that its required. I’m also probably going to rename mldb to mljs, and to MarkLogic JS in all the documentation. Be on the lookout for both the ‘mldb’ and ‘mljs’ tags on my blog from now on.
I’m definitely going to support more options around creation and ingestion of documents, and general management – like viewing information, versions if a DLS document, and properties, as well as download and download-as transformation options. This will complete the lifecycle of functionality – create, search, view info, and download, and delete.
Node.js additions have been lacking. I’m going to retest all my Node.js apps against the current version, and rewrite them for some upcoming blog posts. This should help people get started.
I know others will be using MLJS (MLDB!) soon too, so I need to ensure building an app is as good as, and hopefully better than, using our own Application Builder. I’ll need to refine my search widgets (especially the sorter and results viewer), and graphing capabilities over facets as well as search results themselves.
I also want to add more semantic UI capabilities for V7. All the demos I saw at MarkLogic World used code in the Query Console rather than a UI. Not the best for customer demos. I’m creating widgets so we can paint a picture of content workflow for any customer, rather than a feature-function demo. I really want to allow visualisation and exploration of these facts stored in the triple store, and overlay content information transparently too.
Generally though I’ve now proven you can create an entire full featured web application in MLJS alone with no XQuery support code. Making this pluggable and re-usable has saved me days of effort already. I hope others start to use and experience this too. Please give me your feedback on MLDB/MLJS, I really want to hear from you!
You can get MLDB from the GitHub site here. Read the README instructions at the bottom of the page.
Adam's Deep Technology Blog 