Search statistics: where do they come from

Vote:
 
Hello all,
 
We have migrated to CMS-12 in INTE & PREP, but not yet PROD.
  
Our primary site has a search textbox that features autocomplete.  We only had a few search suggestions show up in INTE & PREP.  I checked the Search & Navigation dashboard which confirmed those few I noticed.
  
I am not understanding a few things and have questions:
  • Are search statistics tranferred when importing a CMS-11 database to CMS-12?
    • Does it not work for cross-environment (cross-subdomain) database imports?
    • Will I lose historical statistics when importing CMS-11 PROD to CMS-12 PROD?
  • Shouldn't any search for a phrase/word show up in the statistics?  If so... 
    • Is it automatic?
    • How long does it take to show up?
    • Are there minimum requirements?  If so, is it configurable?
  • Were there any code and/or configuration changes in CMS-12 related to generating statistics?
  • Does the Clear Index function clear statistics as well?

Since we do have a few search phrases in our dashboard, it seems that the process might be working.  But new phrases aren't showing up.  Online, the Autocomplete - Opti Support Doc mentions sources of suggestions, including “Automatically generated, based on previous searches, that resulted in at least one hit”.  This is just not happening for us.

I have scoured our CMS-11 & 12 codebase for relevant differences, and there is nothing that stands out.
  
My big fear is that we move to PROD and our search phrase suggestions are gone.
  
Any help or insight would be much appreciated!
Thanks, Kevin
#315469
Edited, Jan 11, 2024 19:13
Vote:
 

Search statistics are part of the core ElasticSearch instance that's hosted as a service, they aren't held in the database. 

You can see this as when you hit the search stats area in the admin, you'll notice it making calls to a proxy URL e.g

/EPiServer/Find/proxy/_stats/query?from=2024-01-11T11%3A00%3A00Z&to=2024-01-12T

Any of the /proxy/ calls are actually just forwarded directly to the service as they are just standard elasticsearch queries and none of the laters on top.

Statistics will show up as long as you're using the track call at the end of your search code that tells the service to track this information. Also not everything is tracked, things like autocomplete and did you mean are only tracked if they have hits (e.g navigations to the page) and there's a 90 day retention on that data. There's no configuration as it's just part of how elasticsearch works.

So the key to keeping this is going to be making sure you're on the same find index. With a CMS 12 .NET Core migration you get new DXP infrastructure, I think on some of ours this included a new s&n index but that might have just been a cluster move for performance. So check with your CSM and technical reps to make sure, there's no way to transfer so it'll only work that way.

Although you might be able to write a job or code to fool it and repop after deployment with quieres and hit tracking.

#315513
Jan 12, 2024 10:36
Vote:
 

I just checked with Optimizely and they said you should be able to keep the existing index, therefore I'd expect stats to work 

#315514
Jan 12, 2024 10:43
Vote:
 

Statistics is not in your DB but rather in your index - whenever you call .Track() on your query it will send that to the server. so as long as you keep the same index and if you don't clear statistics, you will be able to keep it when you "import"/upgrade your DB

#315517
Jan 12, 2024 12:01
Vote:
 

Thank you Scott for your detailed answer, and Quan for following up.

Our code has and always had .Track() as part of the search queries.

However, within our INTE & PREP environments, no new statistics are being registered.  Since it should be automatic, what could prevent them from being added?

Also, how can I make sure I keep the same index during migration?

#315519
Jan 12, 2024 14:28
Vote:
 

I hadn't recently debugged our search function, but decided to try.  I have a SQL Server database I exported to use for localhost development, which I'm sure if it makes a difference or not.

It turns out that it is throwing and catching exceptions when trying to track statistics.  I am using all the latest .NET 6 Epi packages.  Debugging this, I tracked this error to:

  • from decompiled EPiServer.Find.Statistics.TrackExtensions.cs
    • line 152: calls GetAdditionalTrackInformationBody
  • from decompiled EPiServer.Find.Personalization.PersonalizationTrackConventions.cs
    •  inside a loop
      • line 232: throws and catches a NullReferenceException
        • method: GetRefCodeFromSearchHit
        • code: _contentRepository.Get<IContent>(contentLink); 
    •  finally
      • line 178: throws and catches a NullReferenceException
        • method: GetAdditionalTrackInformationBody
        • code: ??

Sample log entry:

2024-01-12T20:28:25.556660236Z fail: EPiServer.Find.Personalization.PersonalizationTrackConventions[0]
2024-01-12T20:28:25.556667736Z       PersonalizationTrackConventions could not get reference code from search hit.
2024-01-12T20:28:25.556673836Z       System.NullReferenceException: Object reference not set to an instance of an object.
2024-01-12T20:28:25.556678936Z          at EPiServer.Find.Personalization.PersonalizationTrackConventions.GetRefCodeFromSearchHit(SearchHit`1 hit, String& refCode)
2024-01-12T20:28:25.556685636Z       System.NullReferenceException: Object reference not set to an instance of an object.
2024-01-12T20:28:25.556692037Z          at EPiServer.Find.Personalization.PersonalizationTrackConventions.GetRefCodeFromSearchHit(SearchHit`1 hit, String& refCode)

Honestly, I don't know what this could possibly be from, and I am usure when it started.  Any ideas?

#315530
Edited, Jan 12, 2024 19:11
Scott Reed - Jan 15, 2024 18:01
I'm not sure, could there be issues with your data? If it's working on production I'd suggest restoring the content back over in the DXP pass portal. Else maybe some convention you'd set up different? Maybe check the startup and appsetting.json for differences per environment. Also if you think it should work, raise a support ticket
Kevin Gainey - Jan 16, 2024 14:51
Thinking the same, last week I did restore production data back. Unfortunately, the same result. The conventions are close to identical.
Regarding a support ticket, I created one nine days before I posted this, and has been slow going.
Scott Reed - Jan 19, 2024 10:56
Have you fully cleared and reindexed your data using the S&N admin area? It could be old data in the index
Kevin Gainey - Jan 19, 2024 13:24
Yes, to clearing and reindexing. I've never cleared the statistics fearing I'd lose the few historical working terms I have.

Scott, part of your comment from Jan 15 refers to restoring content from production. How does one copy a CMS-11 production database to a CMS-12 database using the PAAS portal? It was the original source.
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.