Viktor Sahlström
Aug 22, 2016
  7188
(4 votes)

Episerver Find stemming

One of the things that makes search challenging (and really interesting) is language handling. Often, spoken languages differ from what a programmer is familiar with, since the rules are more comparable to a never-ending set of exceptions than actual rules. To properly analyze a string of text, the system must understand all of these exceptions.

A great feature of Episerver Find is stemming. Stemming reduces an inflected word to its root form (a.k.a. stem), for example, "fishing", "fished", and "fisher" have a root word of "fish." If the root word is determined, it can be used to return the full set of related items, thus improving retrievability and relevancy of search results.

Episerver Find uses snowball stemmers shipped as the default stemmer with Elastic search. This stemmer handles the general rules quite well but does not handle all special cases. In many languages, this works very well (English, for example). But depending on the complexity of the language and the maturity of the stemmer, this is not always enough. Swedish is a case where the default stemmer is not always perfect in execution. Often, the default stemmer creates a conflict that, in turn, causes unexpected search hits.

As an example, consider the Swedish words “bananen” (the banana) and “banans” (the race tracks). Using normal Swedish stemming rules, they would both be stemmed down to “banan.” In this case, any search also stemmed down to “banan” would give both results even though half of them are not relevant.
 
To fix this, a list of exceptions has been added to the Find stemming. We have started with Swedish and will look at additional languages going forward. The new algorithm recognizes that “bananen” and “banans” are different words even though their stem is the same. Hence, it creates unique tokens from them so the search engine can distinguish them at query time. This is a great improvement to search relevancy in many cases. One thing that remains to be solved is the case of “banan” (banana) and “banan” (the race track). In this form, the words are spelled exactly the same and cannot be distinguished without looking at the context. For these cases, search results are returned for both words.

To keep the list updated we would love users and partners to let us know if they find searches that results in weird results. 

Aug 22, 2016

Comments

David Tellander
David Tellander Mar 1, 2017 11:47 AM

Hi Viktor, 

I started a thread in the find forum, before I saw this post, with some examples of words that results in a lot of false positives because of stemming. It seems most of these words gets stemmed to common words that should be stop words according to this snowball stop word list

/David

Per Atle Holvik
Per Atle Holvik Apr 24, 2018 01:53 PM

Hi Viktor,

Is there an exception list for Norwegian? If yes, are you perhaps using this one? http://snowball.tartarus.org/algorithms/norwegian/stop.txt In that case, the search for "oversette" (translate) does not seem to be recognizing "over" as a stop word, at least not in Find 9.6.

/Per Atle

Please login to comment.
Latest blogs
The missing globe can finally be installed as a nuget package!

Do you feel like you're dying a little bit every time you need to click "Options" and then "View on Website"? Do you also miss the old "Globe" in...

Tomas Hensrud Gulla | Feb 14, 2025 |

Cloudflare Edge Logs

Optimizely is introducing the ability to access Cloudflare's edge logs, which gives access to some information previously unavailable except throug...

Bob Davidson | Feb 14, 2025 |

Comerce Connect calatog caching settings

A critical aspect of Commerce Connect is the caching mechanism for the product catalog, which enhances performance by reducing database load and...

K Khan | Feb 14, 2025

CMP DAM asset sync to Optimizely Graph self service

The CMP DAM integration in CMS introduced support for querying Optimizly Graph (EPiServer.Cms.WelcomeIntegration.Graph 2.0.0) for metadata such as...

Robert Svallin | Feb 13, 2025

PageCriteriaQueryService builder with Blazor and MudBlazor

This might be a stupid idea but my new years resolution was to do / test more stuff so here goes. This razor component allows users to build and...

Per Nergård (MVP) | Feb 10, 2025

Enhancing Optimizely CMS Multi-Site Architecture with Structured Isolation

The main challenge of building an Optimizely CMS website is to think about its multi site capabilities up front. Making adjustment after the fact c...

David Drouin-Prince | Feb 9, 2025 |