The search system within the Widen CollectiveⓇ allows our 500,000+ users to find the digital assets they need. Across every Collective site the search system's job is to store all of uploaded asset information in a format that can be searched in a wide variety of ways; so that it can return results that are accurate, complete, and in compliance with the platform's governance structure. All in a matter of milliseconds! This is a highly complex system that has gone through several major overhauls over the past 15+ years to keep pace with the exponential growth of data that Widen handles. This post is the tale of that evolution over time, and the steps we take to ensure its ability to scale into the future.
The early years of search (the 2000s)
Under the hood, our search system has always been powered by a popular open source project called Apache Lucene. Lucene organizes enormous amounts of information in a way that facilitates rapid retrieval and display of search results. It is like the engine of a Ferrari — and will get you from point A to point B very quickly! However, you still need a frame and many supporting systems (the rest of the car), not to mention a driver to steer things in the right direction.
The frame has taken many forms for us over the years. Initially, we built it ourselves. It worked well, and Lucene got us lightning-fast results. But it required a good deal of maintenance that took time and focus away from building the core application experience.
Fortunately, lots of pre-built frames have since emerged that make working with Lucene much easier. And we've gone through a lot of them. We used a system called Compass for a while which allowed us to remove a lot of problematic code and focus on our core domain. Later we switched to Hibernate Search to achieve better performance. It included an impressive massively parallelized indexing system that allowed us to get customer sites fully indexed and ready for search much more quickly. Over time, however, even this system could not keep up with the demands being placed on it. For some customers it took over 30 hours to index their content (sync their content with the search engine).
A search system overhaul in 2014
This led to the need for a major overhaul of our search infrastructure, and in 2014 we were an early adopter of a new Lucene-based technology called Elasticsearch, from Elastic. We rebuilt all of our indexing logic from scratch. With this change, search performance improved dramatically and even the largest customers could be fully indexed in less than an hour!
While this was a huge improvement, new problems emerged. This was the first technology we used to house data for all customers in a single cluster of machines. (Note: each customer has their own “index” which is separate and distinct from other customers, but the machines running all the indexes work together to power the experience.) This meant that if there was a problem with the cluster, it took down search for all customers at once. Not good!
Over the last several years Elastic has put a huge amount of energy into fixing these stability problems and continuing to increase the performance and features they offer. And as a company, Elastic has been wildly successful— conducting an IPO in October 2018 to become publicly traded. As these improvements have become available we have put considerable time and effort into upgrading our systems to make use of them.
Preparing the next evolution of search
In 2018 we formed a team dedicated to search and analytics. Their first major project was a significant upgrade to the latest version of Elasticsearch, as part of a larger push to make our search infrastructure ultra fast and always available.
As that work has progressed for the past year, we have finally reached a point where these changes are rolling into production for our customers! The performance looks incredible and we have much greater confidence in the reliability of the new system to serve our customers’ needs consistently – no matter what gets thrown at it.
Behind this confidence is a new partnership with a company called Bonsai. The purpose of their whole operation is running Elasticsearch clusters. They are the experts at this technology and are actively monitoring and addressing any issues that emerge 24/7. Our search team communicates with Bonsai on a regular basis to ensure that we are using the technology as efficiently as possible, and that any changes are done with care and minimal downtime.
Another change that comes with this upgrade is that our customer base is being spread across multiple search clusters, each specially tuned to the needs of those customer populations. Not only will this make the experience faster and more efficient, but it will also ensure that if any issues do emerge, they won't impact everyone at once.
With our greatly improved new search system ready to roll, we will begin augmenting our search capabilities to make content more discoverable and usable based on user activity. Expect a new experience down the road as we add greater structure to our data and the supporting systems, and capabilities to make the right content always available at your fingertips!