Why is my Find indexing job freezing and dying?

I recently had a support case which I spent maybe a few hours too many on.

The symptom was a find indexing job running for about 5 minutes, web app froze up and restarted causing the indexing job to fail prematurly. After looking into application logs, cpu and memory usage (which looked good overall) we could see the app were being restarted .

We did profiling with some hints that the scheduled job thread was aborted. We then took a memory dump 4 minutes into the indexing job.

In the memory dump we could see StackOverflowException, OutOfMemoryException and ThreadAbortException.
After some more digging we found a call to a custom IsSearchable() for the same content over and over.
This behaviour was due to self-referencing content and some custom logic that made it index the same content over and over.

For future cases I'm gonna use something like below to more easily identify errors like this without profiling or memory dumps.
Run the indexing job and check the logs for 'This content was recently indexed' and it might give you a hint on what it's getting stuck on.

[InitializableModule]
[ModuleDependency(typeof(EPiServer.Web.InitializationModule))]
public class InitializationModule2 : IInitializableModule
{

protected static readonly ILog _logger = LogManager.GetLogger(typeof(ContentIndexer));

public void Initialize(InitializationEngine context)
{

    var previouslyIndexedObjects = new Queue<int>();
    int maxLookBack = 1000;

    ContentIndexer.Instance.Conventions.ForInstancesOf<IContent>().ShouldIndex(x =>
    {

        _logger.Info(string.Format("Attempting to index content ID: {0 } Name: {1}", x.ContentLink.ID.ToString(), x.Name.ToString()));

        if (previouslyIndexedObjects.Contains(x.ContentLink.ID))
        {
            _logger.Error(string.Format("WARNING! This content was recently indexed: {0 } Name: {1}", x.ContentLink.ID.ToString(), x.Name.ToString()));
        }

        if (previouslyIndexedObjects.Count() == maxLookBack) { previouslyIndexedObjects.Dequeue(); }
        previouslyIndexedObjects.Enqueue(x.ContentLink.ID);

    return true;
});


...