Find Indexing Job doesn't Finish - Error while copying content to a stream.

Vote:
 

We recently noticed the Find search index job doesn't finish in a client's integration site running dotnet 6. There isn't much logged but the last thing logged appears to be:

warn: EPiServer.Find.Cms.ContentIndexer[0]
      Indexing failed (http error), attempt 1 out of 3: EPiServer.Find.ServiceException: Error while copying content to a stream.
       ---> System.Net.WebException: Error while copying content to a stream.
       ---> System.Net.Http.HttpRequestException: Error while copying content to a stream.
       ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host..
       ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
         at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 count)
         --- End of inner exception stack trace ---
         at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 count)
         at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
         at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
         at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
         at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
         at System.IO.Stream.Write(ReadOnlySpan`1 buffer)
         at System.Net.Http.HttpConnection.WriteToStream(ReadOnlySpan`1 source)
         at System.Net.Http.HttpConnection.Write(ReadOnlySpan`1 source)
         at System.Net.Http.HttpConnection.ContentLengthWriteStream.Write(ReadOnlySpan`1 buffer)
         at System.Net.Http.HttpContentStream.Write(Byte[] buffer, Int32 offset, Int32 count)
         at System.Net.Http.ByteArrayContent.SerializeToStream(Stream stream, TransportContext context, CancellationToken cancellationToken)
         at System.Net.Http.HttpContent.CopyTo(Stream stream, TransportContext context, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpContent.CopyTo(Stream stream, TransportContext context, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpMessageInvoker.Send(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
         at System.Net.HttpWebRequest.SendRequest(Boolean async)
         at System.Net.HttpWebRequest.GetResponse()
         --- End of inner exception stack trace ---
         at System.Net.HttpWebRequest.GetResponse()
         at EPiServer.Find.Connection.JsonRequest.GetResponseStream()
         at EPiServer.Find.Api.Command.GetResponse[TResult](IJsonRequest request)
         --- End of inner exception stack trace ---
         at EPiServer.Find.Api.Command.GetResponse[TResult](IJsonRequest request)
         at EPiServer.Find.Api.CommandBase`3.Execute(IJsonRequest request)
         at EPiServer.Find.Api.CommandBase`3.Execute()
         at EPiServer.Find.Client.IndexDirectly(IEnumerable objectsToIndex, Boolean deleteLanguageRoutingDuplicatesOnIndex)
         at EPiServer.Find.Cms.ContentIndexer.IndexWithRetry(IContent[] contents, Int32 maxRetries, Boolean deleteLanguageRoutingDuplicatesOnIndex)

My gut asays it's just waiting, but the decompiled code shows it should never exceed 10 seconds. There are some 413 errors logged too, but this shouldn't increase the wait time too much.

Has anyone else experienced this? Is there a good way to capture the traffic or troubleshoot it? In the past I would set up a proxy with fiddler and capture the traffic locally, but I don't beleive that is as straight forward with dotnet 5+.

#303515
Jun 14, 2023 22:23
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.