dada
Apr 3, 2020
  9836
(3 votes)

Why is my Find indexing job freezing and dying?

I recently had a support case which I spent maybe a few hours too many on.

The symptom was a find indexing job running for about 5 minutes, web app froze up and restarted causing the indexing job to fail prematurly. After looking into application logs, cpu and memory usage (which looked good overall) we could see the app were being restarted .

We did profiling with some hints that the scheduled job thread was aborted. We then took a memory dump 4 minutes into the indexing job.

In the memory dump we could see StackOverflowException, OutOfMemoryException and ThreadAbortException.
After some more digging we found a call to a custom IsSearchable() for the same content over and over.
This behaviour was due to self-referencing content and some custom logic that made it index the same content over and over.


For future cases I'm gonna use something like below to more easily identify errors like this without profiling or memory dumps.
Run the indexing job and check the logs for 'This content was recently indexed' and it might give you a hint on what it's getting stuck on.

[InitializableModule]
[ModuleDependency(typeof(EPiServer.Web.InitializationModule))]
public class InitializationModule2 : IInitializableModule
{

protected static readonly ILog _logger = LogManager.GetLogger(typeof(ContentIndexer));

public void Initialize(InitializationEngine context)
{

    var previouslyIndexedObjects = new Queue<int>();
    int maxLookBack = 1000;

    ContentIndexer.Instance.Conventions.ForInstancesOf<IContent>().ShouldIndex(x =>
    {

        _logger.Info(string.Format("Attempting to index content ID: {0 } Name: {1}", x.ContentLink.ID.ToString(), x.Name.ToString()));

        if (previouslyIndexedObjects.Contains(x.ContentLink.ID))
        {
            _logger.Error(string.Format("WARNING! This content was recently indexed: {0 } Name: {1}", x.ContentLink.ID.ToString(), x.Name.ToString()));
        }

        if (previouslyIndexedObjects.Count() == maxLookBack) { previouslyIndexedObjects.Dequeue(); }
        previouslyIndexedObjects.Enqueue(x.ContentLink.ID);

    return true;
});


...

Apr 03, 2020

Comments

Please login to comment.
Latest blogs
Configured Commerce - Introduction to Long-Term Support (LTS) Releases

First off, for those who have not had a chance to meet me yet, my name is John McCarroll, and I am the Technical Product Manager for the Optimizely...

John McCarroll | Sep 29, 2023

Auto-translate with OpenAI in Optimizely CMS

You can now auto-translate content using your favorite online AI service, inside the old trustworthy Episerver.Labs.LanguageManager!

Tomas Hensrud Gulla | Sep 29, 2023 | Syndicated blog

Vulnerability in CMS 12 shell module configuration

Introduction A potential security vulnerability has been identified in Optimizely CMS 12, triggered by a certain shell module configuration. To be...

Magnus Rahl | Sep 28, 2023

AI-Assistant: The 'Change Tone' Shortcut

The AI-Assistant for Optimizely is constantly evolving, adjusting, and transforming to meet your digital needs, providing a cutting-edge advantage...

Luc Gosso (MVP) | Sep 27, 2023 | Syndicated blog