The Case for High-Definition Search

We are witnessing a strong shift today from general-purpose search engines, crawling the web, to more specialized ones, which are domain-aware.

One of the early signs was eBay. As early adopters of the web became marginalized, because casual users would not create their own web site and pages to sell their used stuff, eBay captured a significant part of these searches.

But towards the end of the 20th century, a significant number of valuable transactions were difficult to drive to eBay, with their simple search engine parameters: car purchases.

So eBay created eBay Motors, to offer advanced features to search for cars. eBay had become "car-aware." It suddenly knew about model years, car makes, models, trims, body type, color, engine type, mileage, etc.

Today, a hefty amount of search done on the web is performed on specialized web sites. eBay drives now only a fraction of car purchases, because other platforms have specialized in these, and provide "high-definition" search, using a wealth of useful parameters.

Meanwhile, "low-definition" platforms struggle to drive consumers towards them. Of course, high-definition search is only possible is you have high-definition modeling of the domain, something which is lacking in general-purpose platforms. But also, general-purpose platforms are already big, and because of their lack of abstraction layers, and extensive use of the human-wave approach, they cannot evolve, and remain stuck in a limited set of features.

Relational Databases have not helped. When you have ambition for your platform, and prepare for growth, you need to care for performance. Efficient search in a relational database is limited to mono-dimensional search, search on a single parameter. This is because they have never evolved much beyond simple binary search trees. Following the rise of maps, and latitude and longitude-based search coming with it, they had to stretch towards bi-dimensional search (search on two criterions instead of just one) with geographical indexes. But who cares about searching only on two parameters?

Alongisde its high-definition modeling, Metaspex proposes an "instantaneous" multi-dimensional search, with no dimensional limit. Indexes are created in only one line of specification. This programming gem performs as well as mono-dimensional search when only one search parameter is selected, and blows all databases out of the water when more than 2 parameters are selected.

Just for fun, we created a 3-dimensional index (allowing to search on 3 criteria at the same time) over a billion products (Amazon has 12 million, 350 if we include third-party products). The index searches in microseconds and occupies less than 59 GB memory... It has more than 2.8 times the whole Amazon, and fits on a single conventional machine that you can buy on... Amazon.