I’ve been thinking a lot these past two years about how we build NoSQL database apps, and how we expose their functionality. Here’s a few of my disjointed thoughts on the subject…
Haven’t we been doing this for years?
Yes and no. Relational databases have allowed apps to be built directly on top of them for years. This included simple data entry, shopping basket management, and reporting/dashboarding type UI experiences.
These originated by being integrated in the databases themselves. These days there exists three tiers in an application stack – the database, middleware where business logic resides, and a front end interacting with the business logic (add item to cart vs. add row to cart table in database).
These applications used the same query language as all the relational databases pretty much talked the same lingo and had the same features. This isn’t the case in NoSQL world. MarkLogic Server does a lot more than simply store, retrieve and run aggregates over data.
This depth of functionality means you have to use native functionality, necessitating a deeper level of coupling between UI code and the database than is usually apparent.
Why it’s different
MarkLogic Search is a classic example of where attempting to be too loosely coupled falls down rapidly. MarkLogic has a free text grammar. This is like a google style text box with a query that is parsed on the server.
This is easy to represent in a loosely coupled way. Where it falls down is when you need to have a single page of multiple widgets all contributing different aspects of the query, and the wider query settings.
Which page you are on, how many results to show per page, sort order, facets selected, geospatial area on the map selected, and parsing and representing the results on a map/results list/facet area/info box area for related content.
As soon as you do these other things then a simple text field doesn’t cut it. You all of a sudden need to understand the different type of facets – which one is selected and its type determines the type of query to submit. Geospatial constraints, and the result pins and heatmap shown on a map, all require special handling too.
Trying to loosely couple a very specific backend system function is a nonsense. You end up writing lots of ‘plumbing code’ trying to be an architectural purist. This is the web UI equivalent of performing heavy ETL (Extract, Transform, Load) when consolidating data in to a single relational model.
You’re trying to force an artificial intermediate structure on your code when in rality its much simpler and quicker to just adopt the native search formats and semantics of the underlying platform.
Where portlets had it right
Back in t’day Java Portlets and Portal Servers were all the rage. You could take a variety of portlets developed by different people and plug them in to one web UI. Later versions could even share data amongst themselves.
Portlets were pre-packaged bits of functionality, sometimes also referred to in the web UI they show as ‘widgets’. You could have an email inbox widget, news widget, notifications, calendar meetings list – all in a single user configurable page.
They were also easy to install – just upload a glorified zip file. They had an edit screen for modifying settings, and a view screen for displaying their content.
This made it very simple to create your own customised interface for your exact user interface needs.
I used to work at FileNet, an Enterprise Content Management (ECM) and Business Process Management (BPM) company, now part of IBM. We accidentally stumbled upon a very popular user interface.
The strategy at FileNet was to create a set of basic portlets to show how functionality and lower level APIs could be incorporated in to other systems. To showcase this FileNet developed a simple Java web app that acted as a thin portlet container and security housing, called the FileNet Workplace.
This was intended just to provide a testbed for the portlets. A user inbox, shared work queue, e-form filling, document folder listing, and content search set of widgets. Nothing mind blowing at all.
The FileNet Workplace ended up being the most heavily deployed application in FileNet! Companies loved the fact it came working out of the box – they could login and rearrange and reconfigure these widgets and have an app ready in no time.
They knew they could take the portlets and integrate it with others in a Portal Server, but they didn’t bother. It had all the functionality of the underlying FileNet platform they were interested in.
This resulted in customers buying in to the full FileNet ECM and BPM stack because they had access to the Workplace to see the functionality in action.
These customers knew an independant portal server with business rules abstracted was a good architectural choice. They didn’t code this from scratch though because the FileNet Workplace provided exactly the functionality they needed with enough configuration to tailor it for their needs.
Crucially the Workplace required no systems administration and especially no coding. It was by far and away a key success. Probably because of the tight coupling of the UI to the underlying system, rather than in spite of it.
Using these prebuilt and somewhat configurable widgets (portlets) gave customers a way to get instant value with minimal cost. It was a no brainer to deploy the Workplace application.
Where architectural rules for rules sake hurts innovation
There is a religious battle going on right now on the Internet. It is largely hidden from most people, but its been waging for a number of years.
The web UI world knows their is a need to apply standard patterns but they go through new frameworks like locusts. Barely 6 months goes by without a group of developers moving from one web UI framework to another claiming that the new one solves all the problems.
They claim they will solve all the code copy/paste/alter customisations that make code re-use impossible. They claim the frameworks provide fast responsive UIs whilst simultaneously making apps easier to develop. They also claim a consistent set of framework rules that all people apply makes code easier to pick up, understand, share and maintain.
They are all smoking crack.
No single web ui framework has even a large minority let alone majority of uptake. None. Various excuses are made for them. Claims of inertia and break through frameworks being talked about most are made to justify pouring years of effort in to re-creating the same apps in a new way for a particular framework.
The problem is by using these frameworks you enforce a degree of separation between your code and the underlying system that doesn’t need to be there. Binding a web component to a variable, and the same variable to a result or parameter to a remote service sounds like a good idea. But when that variable holds data objects that are only ever used with and understood by that underlying system, what’s the point? You’re just adding a layer of complexity, not to mention creating slower code.
Promises are another lie. You call a service and say ‘when it returns run this code’. What if many loosely coupled elements relies on this code? You need effectively an event publish/subscribe model. The same as you would by passing a callback anyway. So you basically are using promises to make your code look nicer through avoiding a single pair of curly braces. Why?
Promises, $(‘#someelement’) and $(‘.someclass’) are particular evils I would love to see eradicated. Sure they make developers code shorter – but at a massive cost of increased code base and slower processing time. DOM operations are positively encouraged even though we all know they are the slowest thing you can do in a browser.
So you’ve adopted the latest UI framework standard fad. You’ve re-architected your code to use its conventions. You now have code loosely coupled to your backend system and tightly coupled to a web UI framework.
If you’re building a search app then its talking to just the search engine. You’ve basically increased your plumbing at the cost of a lot of flexibility. When the backend system changes you now have more work to do than you would have done if you weren’t using a framework. Oops.
This happened recently in MarkLogic. Version 7 introduced the concept of a combined query – this single REST call take a search options configuration document, a text query, and a structured complex query (E.g. for geospatial and complex nested OR/AND query terms).
In MLJS, my javascript library tightly integrated to MarkLogic, this took just a few lines to support. (More lines because I was supporting both version 6 and 7, with platform detection).
If I had instead adopted a set of services, promises, bound variables, and ui directives I would have had a lot of plumbing work to do. As it was my contributeFacet() and doSimpleQuery() and doStructuredQuery() functions didn’t change – just the code within them. No need for 7 levels of indirection.
The lies of code re-use
Many of these jesus frameworks claim eternal life through maximising code re-use. In my experience developer teams using web frameworks take a copy of the old code and re-jig it for their new needs, breaking old apps in the process.
This means new services, bound variables, configuration options – and no backwards compatibility or testing with previous applications.
This means they end up with several versions of the same widgets and service code, none under effective cross-company source control, and very new development becomes precisely that – new development code. And a waste of time and effort.
Development sustainability
A single page web application should not take 25 man days to create. Let alone 50 or 75 days.
I don’t care how nice the frickin’ thing looks. Software companies cannot afford to have a team of 20 people coding essentially the same thing over and over again. It’s like building a car but creating a different design for every individual care you create. Its insanity.
It’s also unsustainable. Instead of coding apps for people you should create re-usable code or add new features and provide the configuration options to control how they work.
You should empower those that need to use the code, not sit down for a month and code every new application for them. If a piece of software becomes very popular you cannot afford the delay in creating demonstrations.
You also cannot afford to train everyone that needs to create a simple demo on being the best web UI framework and javascript developer they can be. This takes time in itself, and just slows down your ability to respond to customer demand in both the short and long term – not increase it. Coding is hard and takes more time than simply configuring an existing piece of functionality in an easy to use web form.
A much better approach is to empower these people by making a configurable – not code customisable – interface available to them. This is where the MLJS Workplace has a clear advantage over any web ui framework.
Benefits of tighter integration
By more tightly coupling code to the underlying system you have a common interface reference – that of the underlying system. Adding new features means supporting more of the backend system. Adopting native JSON objects, terminology and parameters means the code is re-usable across all projects built against that product.
By creating re-usable widgets at sensible levels of abstraction but all working on the same underlying message format (E.g. a JSON search results object), these widgets can be independently built whilst providing an absolute guarantee that they will work together.
No need to invent your own ‘service’ for your new project, or set of bound variables – just use the native data structures the system provides.
This cuts development time drastically and ensures better re-use of code. It also means newly developed widgets are instantly pluggable.
I can create a new search application in minutes by configuring a set of search widgets and their common search context that defines the shared search settings and results.
If I need a new feature like displaying a particular search result in a particular way then I write a single object with a couple of functions. This is picked up at run time and works straight away.
If I want a new option on my map widget I add a configuration variable to it with a description. This is then picked up in the administration user interface immediately, whilst not breaking any older descriptions of application pages from previous demo builds.
The MLJS Workplace
I created the MLJS Workplace application to provide a drag/drop web UI for creating MarkLogic based applications. This has over 30 web widgets that are tightly integrated to MarkLogic and work in a single easy to use, professional looking application.
There are no dependencies on any outside web UI frameworks, which means minimal code base, less layers of indirection (and thus less debugging), and faster demo creation times.
I can create a 6 page web application in 5 minutes covering search, geospatial map views, semantic query, graph exploration, document viewing, and a dashboard of analytics. All with zero code.
This is impossible with any web framework you care to mention today.
Tight integration isn’t a sin when it’s done for a good reason. Enforcing architectural choices that increase development time and complexity with zero user visible business value is the real sin.
For more information on the MLJS Workplace have a look at the playlist on YouTube:-
http://youtube.com/playlist?list=PLWoOSZbmib_e43iLpUURq7MEGBKRb13Id
Adam's Deep Technology Blog 