Deane Barker
Dec 7, 2010
(4 votes)

Integrated Lucene Search for EPiServer

We recently implemented a site that had more PDF files than pages, in a ratio of about 6-to-1.  Sadly, this process revealed some of the shortcomings in EPiServer’s default search architecture.

The biggest problem we found is that there are two search algorithms at work.  Pages are indexed via a custom-built tokenizer, with their keywords stored in a database table.  Files, on the other hand, are indexed and searched via Lucene.Net.

Given that there are two separate methodologies, it becomes somewhat impossible to mix results.  When using the SearchDataSource, what happens is that all page results are returned first, then all file results come after.  This is a significant issue for my client.  Given their mix of content formats, visitors are often looking for a PDF file first, and an HTML page second.

There is a “magic” property on all search result pages called “PageRank.”  We were excited when we found this, because we figured we could use LINQ to order by this property.  Unfortunately, this property is fairly crippled because all files have the value of “500.”  We reflected through the DLL and found that this is hard-coded and not changeable.

When we brought this to their attention, EPiServer offered to do a hotfix, but we weren’t sure it would help – even if this value wasn’t always the same, how do we know if the PageRank value for files has a directly proportional relationship to the PageRank value of the page results?  These two values are calculated completely differently, and we have no way of knowing if they compare to each other in any meaningful way.  (ex: if a page result has a PageRank of 500 and a file result has a PageRank of 1000, does it follow that the file is twice as relevant than the page?)

So, the bottom line is that we needed to get pages and files indexed and searched using the same algorithm.  This way, we can compare them are know that the relative values have some relationship.

Files are indexed via the EPiServer Indexing Service, and this offered the fewest options to change.  So, we decided to keep that as-is, and simply created another Lucene.Net index for pages.  Then, we abandoned EPiServer’s SearchDataSource control, and implemented a new searching API that uses raw Lucene.Net calls to query both sets of indexes – those from files (created by the EPiServer Indexing Service) and pages (created by our new plugin).

Code to search looks like this:

LuceneSearchManager search = new LuceneSearchManager();
search.SearchIndexes = "SitePages,SiteGlobalFiles,SiteDocuments";

LuceneSearchResultCollection resultCollection = search.Search("episerver");

SearchResults.DataSource = resultCollection.Results;

In this situation, “SitePages” is an extra VPP created just to store the Lucene index of pages.

Preliminary tests have been quite good.  Both files and pages are indexed and searched using the same methodology, and result sets are ordered by rank, regardless of whether they’re page or file results.  The API supports paging, and also supports EPiServer’s security model (the actual enforcement of security was modeled after how EPiServer does it in SearchDataSource – it catches an expected exception at one point, which I’m not in love with, but it works).  PageTypeNames and all ancestor IDs back to the Start Page are stored, and can be used for searching and filtering as well.

Additionally, since you have low-level access to the underlying Lucene.Net index, you have the full power of Lucene.Net at your disposal, including search-time field boost to bias results.

This has not been deployed to production yet, but will likely see the light of day in Q1 2011.  Consider it an alpha release.  If you implement, we welcome all feedback.

Lucene Search Download

Dec 07, 2010


Dec 7, 2010 11:17 AM

Nice solution!

We are aware that the mixed-search model is a pain-point and we will adress it. This will happen after the release of EPiServer CMS 6 R2.

Dec 8, 2010 08:03 PM

Nice work. :)

Aug 11, 2011 04:00 PM

Hi Deane!

I'm currently testing Lucene via the EPiServer FTS module and the facade code from EPiServer Nuget (ref.
For comparison I'd very much like to check out your solution but the download link is no longer working. Can you update the link?

Please login to comment.
Latest blogs
Configured Commerce - Introduction to Long-Term Support (LTS) Releases

First off, for those who have not had a chance to meet me yet, my name is John McCarroll, and I am the Technical Product Manager for the Optimizely...

John McCarroll | Sep 29, 2023

Auto-translate with OpenAI in Optimizely CMS

You can now auto-translate content using your favorite online AI service, inside the old trustworthy Episerver.Labs.LanguageManager!

Tomas Hensrud Gulla | Sep 29, 2023 | Syndicated blog

Vulnerability in CMS 12 shell module configuration

Introduction A potential security vulnerability has been identified in Optimizely CMS 12, triggered by a certain shell module configuration. To be...

Magnus Rahl | Sep 28, 2023

AI-Assistant: The 'Change Tone' Shortcut

The AI-Assistant for Optimizely is constantly evolving, adjusting, and transforming to meet your digital needs, providing a cutting-edge advantage...

Luc Gosso (MVP) | Sep 27, 2023 | Syndicated blog