Try our conversational search powered by Generative AI!

Huge indexing queue?

Vote:
 

Hi,

In tblBigTable, I have a huge amount of items marked with StoreName = 'IndexRequestQueueDataStore' (currently about 845 000 items, and it grows by roughly a hundred new items every hour depending on user activity in the communities.

What is responsible for handling these items, and what might cause it to stop processing the queue?

#63399
Nov 19, 2012 12:31
Vote:
 

There are currently four load-balanced web servers (EPiServer CMS 6 R2 Enterprise) in the environment. One of the servers is configured to be the search server, another is a search client, and the other two aren't configured to use FTS yet.

This is the section from web.config for the server acting as an FTS search server:

<episerver.search active="true">
    <namedIndexingServices defaultService="clientIndexingService">
      <services>
        <add name="clientIndexingService" baseUri="http://hostname-of-FTS-server/IndexingService/IndexingService.svc" accessKey="secret-key" />
      </services>
    </namedIndexingServices>
    <searchResultFilter defaultInclude="true">
      <providers />
    </searchResultFilter>
  </episerver.search>
  <episerver.search.indexingservice>
    <clients>
      <add name="secret-key" description="Search clients" ipAddress="ip-adresses-of-servers,127.0.0.1/8" allowLocal="true" readonly="false" />
    </clients>
    <namedIndexes defaultIndex="default">
      <indexes>
        <add name="default" directoryPath="\\fileserver\share\Indexes" readonly="false" />
      </indexes>
    </namedIndexes>
  </episerver.search.indexingservice>

The server configured as a search client has the same episerver.search section, but nothing configured in the episerver.search.indexingservice section (except the defaultIndex attribute).

I can see that something changed in the queue after configuring FTS, since new items are added with the new service name in the NamedIndexingService column, but new items never seems to be removed from the queue.

I am about to configure the other two servers so they also use these (FTS client) settings.

#63420
Nov 19, 2012 17:50
Vote:
 

Using dotPeek I've been able to find out a little about what is responsible for handling the indexing request queue.

The indexing service itself (the server side) is responsible for processing the queue. Which is pretty obvious. :)

However, I have no idea if it actually does what it is supposed to be doing... If I open Windows Explorer and look in the indexing directory, I lots and lots of files being created and then deleted again. They are named something similar to _3h5c8.del, and have different file extensions, such as .cfs, .frq, och some other extensions.

However, is the index for Lucene stored on disk? I.e. is a file supposed to increase in size? The only files whose name doesn't change is segments.gen and write.lock, and they are 1 KB and 0 KB each.

#63764
Nov 28, 2012 19:53
Vote:
 

EPiServer FTS is causing more problems on the server configured as an indexing server... For some reason it doesn't complete the requests, but leaves them "hanging". It doesn't take long before the maximum concurrent request limit is reached (5000 concurrent requests) and the site returns an error message instead. I think it's time to remove FTS from all the servers. :)

#64024
Dec 07, 2012 12:07
This thread is locked and should be used for reference only. Please use the Episerver CMS 7 and earlier versions forum to open new discussions.
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.