I am frequently asked if MarkLogic is really a search engine. It is easy to debate whether MarkLogic fits the classic definition of a search engine. In my opinion, this is the wrong question. The question you should be asking is “Does MarkLogic enable great search experiences?” The answer is undeniably Yes.
MarkLogic comes with all of the standard search capabilities like: keyword search, synonyms, fuzzy search, hit highlighting, sorting, faceted navigation and relevance. These are the basic features that every search engine should have. MarkLogic checks the box on every one of these and more.
The fact that MarkLogic can do all of the basics makes it just like all of the other search engines on the market. What sets MarkLogic apart is that it is not just a search engine. MarkLogic combines some of the best features of search with a fast performing XML database. This combination allows MarkLogic to offer features that traditional search engines lack. Four of the most important differentiators are:
- multi-level searching,
- editable search results,
- schema flexibility,
- and simplified architectures.
MarkLogic allows for multi-level searching. Most search engines require you to flatten out the data for search results. MarkLogic is an XML database. As a result, information can be stored in a hierarchical format and queried at multiple levels. This is particularly important in more complex search experiences. For example, if you are searching large documents, you may want to show the documents that contain your search term along with the sections of the documents that have that term. Normal search engines would require you to create multiple collections or a complex search screen. MarkLogic handles these situations naturally.
MarkLogic’s database features allow you to create applications with editable search results. Our architects call it a “Live” search tool as opposed to a “read only” search tool. Traditional search engines are designed to be read only. Edits to existing search data require re-indexing. Solution providers like Avalon create special indexing routines to allow for updates to content. These solutions are not real-time and they are not simple. Fields can be updated or added to a MarkLogic database at any time, transactionally, with full ACID protection. This flexibility allows us to create a number of really interesting search applications that would have been much more difficult with standard search engines. For example, we have created tools that allow end-users or administrators to “tag” one or more search results (similar to the functionality in Flickr). In other applications, we have created search screens where the users can edit the search results without leaving the screen. Adding these cool features to our search applications is much easier with a combined database and search engine.
As an XML database, MarkLogic provides schema flexibility for storing and querying information. Our developers and our clients love MarkLogic because it is easy to add new fields to the index. Traditional search engines typically require administrators to delete and reload the data in order to add specific fields. In extreme cases you have to re-index an entire data set. MarkLogic’s schema flexibility becomes even more important when you are working with techniques like entity extraction. Text Analytics tools can identify people, places and things within unstructured text. Through this process our clients often find interesting things they want to include in their search applications. MarkLogic makes it easy to run text analytics against unstructured documents and include the entities in the search results. Traditional search engines add a great deal of complexity to the process and do not allow for changing structures.
Our architects like MarkLogic because of its simplified architecture. The next time you meet with your search engine vendor, ask them for a physical architecture diagram of one of their larger implementations. At a minimum you will have a database or file system to store documents and data, a search indexer, a search server, and a web server. Large data sets get even more complicated. Search results have to be clustered and replicated. You will need multiple indexers and search servers running. You will also likely need more than one web server and application server for your front end application. MarkLogic is a database server, search engine and applications server in one tool. It also has built in replication. This means fewer servers and less complexity in your dev, test and prod environments.
One final reason to use MarkLogic to power your search applications is that MarkLogic is not just a search engine. Traditional search engines are very powerful, but they are expensive and limited to search-based use cases.
- Want to publish thousands of documents to your website or mobile devices. Some of the largest publishers in the world use MarkLogic to do this every day.
- Want to build an application that allows users to build reports on the fly by combining sections from other documents. Those same publishers use MarkLogic offer custom publishing solutions.
- Want to create a central repository tracking all of your digital assets. We are working with three different customers using MarkLogic as a central repository across all of their content management systems.
- Do you need a tool to capture unstructured information for your Big Data solution. MarkLogic does this for numerous government customers.
At the end of the day, when your management asks you how much you spent on your search solution, it is nice to say that the tool you bought does more than just search.
In fairness, MarkLogic may not be the best solution for an organization that is looking to build a vanilla search intranet that indexes content from numerous secure repositories. Search engines like Endeca, Autonomy, Vivisimo and Lucene/Solr were designed for these types of solutions. If, however, you need to build a powerful search application that will change over time, MarkLogic is a great choice. It offers many valuable features that are not available in any other search engine.