Blog posts by Stellan Danald

Blog posts by Stellan Danald2017-08-01T11:51:05.0000000Zhttps://world.optimizely.com/blogs/stellan-danald/ Optimizely World Running Find and Commerce on Azure/DXC? Read this...<!DOCTYPE html> <html> <head> </head> <body> <h2>Introduction</h2> I recently ran into an issue which required a lot of time and effort to solve, and wanted to share my findings in case someone else runs into similar issues. <h2>Background</h2> A client of ours is currently running a sizable Episerver Commerce site hosted in Azure by Episerver DXC. This particular site is one of the most integration heavy sites I've had the pleasure to work with recently. As with many commerce sites, this one too is integrated with both a PIM and an ERP among other back-end systems. With well over 15K products and 150K variants, needless to say, it is a challenge to keep all information up to date on the site, with both new and updated products, variants and even inventory. To manage this, we have created an API that constantly receives product updates from the PIM and scheduled batch jobs that update the inventory from the clients ERP system. In the best interest of the end user (and for best performance), we put all this information in Episerver Find. When products/variants or inventory are updated, Episerver Find events are triggered which adds the related products or variants into the Find Indexing Queue (stored in the DDS in the CMS database). Any of currently running Episerver instances will process the queue as soon as there are any and update the document in Find, usually within seconds.   <h2>The Issue</h2> After an internal effort by our client to improve the quality of the metadata of the products on the site, they came to us with some strange findings. After products/variants were imported from the PIM, they noticed that a few of the variants were missing information which they knew were in the PIM. And stranger yet, this wasn't happening all the time. This particular subset of nested information is currently feeding an alternative navigation feature on the site, which made it very apparent when the product/variant they were looking for wasn't listed. <h2>Related findings</h2> After realizing this was not a data/user issue, I did some additional digging and found the following things: <ul> <li>If I reimported the same product, the same variants that were previously incorrect were correct, but other variants were incorrect.</li> </ul> <ul> <li>After manually re-indexing the product and variants from an Admin Plug-in we built, everything looked correct on the site. The difference here is that the content is being indexed by the instance I'm currently on.</li> </ul> <ul> <li>To add even more to this mystery, there was nothing in the error logs that indicated that anything was wrong.</li> </ul> <ul> <li>This issue only occurred in the production environment. And we couldn't replicate this in any other environment.  </li> <li>Page Types that were explicitly excluded in our Find client conventions still ended up in the index.</li> </ul> <ul> <li>When comparing the same Find document when it was incorrectly indexed, and later a correct one, I noticed that some of our properties which had nested conventions were missing the "$$nested" part of the name in the Find index.</li> </ul> <ul> <li>Knowing how an incorrect document looked in Find, I could now query it directly to get the exact number of failed products and variants, which showed that this issue actually had affected 5-10% of the entire catalog. </li> </ul> <h2>Hypothesis</h2> My initial thought, however unlikely, was that one (or more) instances were not indexing correctly or wasn't initialized. Obviously, since all instances in Azure are running the same codebase, it wasn't completely logical, but either way, there are things I can do to confirm this theory, which I wanted to do before submitting a ticket with Episerver Developer support. <h2>Steps taken</h2> The first thing I did was add additional logging and logging levels to our EPiServerLog.config, to make sure everything is initialized in the intended order. <pre class="language-xml"><code><logger name="EPiServer.Framework.Initialization"> <level value="Debug" /> </logger> <logger name="EPiServer.Commerce.Initialization"> <level value="Debug" /> </logger></code></pre> After deploying this, the logs clearly showed that our Find client conventions were initialized correctly on every single instance. At this point, we even re-indexed the whole site (which takes 2-3 hours), and everything seemed to be working. I queried the Find index and saw that the number of failed documents were counting down. However, the day after the deploy, the number had gone up again.  Still not convinced that this wasn't the root cause, I wanted to know which instance(s) that were failing, so I added an "IndexedBy" property to the product and variation models. <pre class="language-csharp"><code>public string IndexedBy => EnvironmentUtility.GetEnviroment();</code></pre> <pre class="language-csharp"><code>public class EnvironmentUtility { public static string GetEnviroment() { var environment = $"{Environment.GetEnvironmentVariable("WEBSITE_SITE_NAME")}:{Dns.GetHostName()}:{Environment.GetEnvironmentVariable("WEBSITE_INSTANCE_ID")}"; return (environment.Length > 2 ? environment : Environment.MachineName); } }</code></pre> Intriguingly enough, this did confirm my theory. I could see that all incorrect Find documents were indexed by the same Azure instance(s)! Even though the logs looked fine, I updated our initialization module containing our Find conventions so that they were executed at the very end of the initialization process, but that still didn't help. I also tried setting the nested convention using a typed implementation, instead of an interface, which also didn't help. <h2>The Solution</h2> Obviously, this was a harder nut to crack than I initially thought, so at this point, I had reached out to Episerver developer support. Thanks to the work I had already put in, I did get a confirmation that this was a bug (which later turned out to be two) that will be fixed in future releases of Find. The final step I took (with the support of the Epi dev team), was instead of implementing our Find conventions in an IInitializationModule, we let it inherit from IConfigurableModule, and add the following <pre class="language-csharp"><code>public void ConfigureContainer(ServiceConfigurationContext context) { EventedIndexingSettings.Instance.IndexingQueueBatchSize = 0; if (Logger.IsEnabled(Level.Information)) Logger.Log(Level.Information, "Setting IndexingQueueBatchSize to 0 #find"); }</code></pre> And then adding the following lines to the Initialize() method  <pre class="language-csharp"><code>var batchSize = EPiServer.Find.Configuration.GetConfiguration().IndexingQueueBatchSize; EventedIndexingSettings.Instance.IndexingQueueBatchSize = batchSize; if (Logger.IsEnabled(Level.Information)) Logger.Log(Level.Information, $"Setting IndexingQueueBatchSize to {batchSize} #find");</code></pre> <h2> Root Cause</h2> As you probably can tell by the code above, the root cause was that WHEN and IF there are items in the Find Indexing Queue, when an Azure web site (re)starts, it will start processing the queue before our Find conventions (which are commerce dependent) were completely initialized(!). At that point, it doesn't help to re-apply the conventions (I built an admin plug-in to try that as well). This is probably one of those edge cases where you have to have a site that is constantly updating the Find index, while utilizing CMS, Commerce and Find and is hosted in a load balanced environment, but please rate and/or comment if you find this article interesting or helpful! Thanks! Stellan Danald Solutions Architect @ Making Waves P.S. Have you checked out the EPiServer Slack Community? <a href="https://episervercommunity.net/">https://episervercommunity.net/</a> - Read about it at <a href="/link/c26f2ae236034e47882ee706a3875a18.aspx">https://world.episerver.com/blogs/stellan-danald/dates/2017/6/episerver-slack-community/</a> An additional tip* If you want to see how many references are in your Find indexing queue, run the following SQL script in your CMS database <pre class="language-csharp"><code>SELECT COUNT(1) FROM [dbo].[tblBigTable] WHERE [StoreName] = 'EPiServer.Find.Cms.IndexingQueueReference' SELECT COUNT(1) FROM [dbo].[tblBigTableIdentity] WHERE [StoreName] = 'EPiServer.Find.Cms.IndexingQueueReference'</code></pre> </body> </html>2017-08-01T11:51:05.0000000Z

Blog post

Episerver Slack Community<!DOCTYPE html> <html> <head> </head> <body> For the last couple of years, we have been utilizing <a href="https://slack.com/">Slack</a> as our primary communication tool. It's fast, searchable, direct, (practically) spam free and lets you effortlessly collaborate easier in or across offices without cluttering your inbox. Not too long ago, I've started to join other slack teams/communities within my field or with my friends. What appeared to me is there currently is no Episerver Slack community. I've seen several other targeted at other CMS's, like SiteCore, Drupal etc., but not Episerver, which somewhat surprises me considering the huge effort Episerver has put into building a community. Ever since I started working with the Episerver platform, I've always thought that the community was really great. They have frequent meetups all over the world, Blogs, the world.episerver.com portal with the forum and support pages, the EMVP initiative to encourage the Episerver community, Episerver Ascends, the recently started <a href="/link/4d36bbc1d75940b5ab798c8924d423f5.aspx">Episerver Coders webinars</a> and other sales and/or dev related events. In our internal Slack group, we do have #episerver channels where anyone can ask the MW epi developers random Q/A, which is not only a great brain-trust to quickly tap into, but also a simple way to help junior developers to ramp up quickly. I don't mean to devalue the dev-to-dev forum on world.episerver.com, but imagine the vast amount of knowledge and experience we could gather if we get some, or even most, of the distinguished Episerver minds together in a tool that I believe many of us are already using. It's a natural step and will make the Episerver community become even closer! To kick this off I've started both the Episerver slack community, set up a dedicated website for easy (and automated) registration, created a bunch of more or less obvious channels/topics related to Episerver related products/features. <img src="/link/13d6d656f7654a28a4377550f2268b06.aspx" alt="Image epi-heart-slack.png" /> It is totally free, non-exclusive and not affiliated with the company Episerver. Register now at <a href="https://episervercommunity.net">https://episervercommunity.net</a> Code of conduct can be read <a href="https://episervercommunity.net/codeofconduct">here</a> Happy coding and see you there! Stellan Danald, Solutions Architect @ Making Waves </body> </html>2017-06-26T02:49:10.0000000Z

Blog post

Episerver Find - Conditional Sorting?A client asked me if it was possible to push a specific variation to the top of a list that is coming from Find. I immediately replied "Of course!", since like most things working with development, almost everything is possible; It's just a matter of time/effort, and I thought it would be a quick change. When I first looked into the task, I noticed that the list was set to order by the variations SortOrder and not score, so boosting wasn't really an option. <pre class="language-csharp"><code>SearchClient.Instance.Search<VariationContent>() .Filter(x => x.Parents.Match(refId.ToString())) …… // Other visibility conditions, facets/terms/skip/take etc .OrderBy(x => x.SortOrder) .GetResult(); </code></pre> My next step was to see if there are any possibilities do to a conditional sort, as in sorting with filters, or perhaps using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html">custom script sorting</a> which you can do with <a href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a> that <a href="http://find.episerver.com/">Episerver Find</a> is based upon. After a lot of googling and plowing through a lot of interesting blog posts and questions on <a href="/link/6c9478a8761c41d88dfc32e9ef56e714.aspx">world.episerver.com</a>, I couldn't find anything that was related to my task. Next step was to do some reflection of the Episerver.Find assembly. What I found there was that it IS possible, IF you were sorting by nested objects, using the following method in the Episerver.Find.NestedSortingExtensions class <pre class="language-csharp"><code>.OrderBy(enumerableFieldSelector, itemFieldSelector, filterExpression)</code></pre> Unfortunately, there aren't any existing methods that does what I was looking for, e.g. <pre class="language-csharp"><code>.OrderBy(fieldSelector, filterExpression)</code></pre> Taking another look at the <a href="/link/8b77b657138f45cfb78b2b583905101e.aspx">Find SDK</a>, I stumbled upon <a href="/link/e11091db0e424a3c92eb39ae12b58a24.aspx">.Include(filterExpression, boostMultiplier)</a>, which, together with the extension <a href="https://gist.github.com/lindstromhenrik/9779858">.ThenByScore()</a>, written by <a href="/link/d99f9563c9ff482c98393641f4e03dc5.aspx?page=1">Henrik Lindström</a>, that we already use, would help me with my task. .Include(filterExpression, boostMultiplier) allowes me to: <ol> <li>Search for my specific item, using the regular .For(query) with the .InField(fieldSelector) and .Filter(filterExpression)</li> <li>Get the rest of the variations with the same boosting score (since they don't get a hit in the .For), that I can then order using the field SortOrder, just like before.</li> </ol> The final solution was <pre class="language-csharp"><code>SearchClient.Instance.Search<VariationContent>() .For(itemId) .InField(x => x.ItemId) .Filter(x => x.Parents.Match(refId.ToString())) .Include(x => x.Parents.Match(refId.ToString()), 1) …… // Other visibility conditions, facets/terms/skip/take etc .OrderByScore() .ThenBy(x => x.SortOrder) .GetResult(); </code></pre> I'm sure there are multiple ways to achieve the same results, but this was the simplest way I could do it and it might help someone else. Cheers! / Stellan Danald, Solutions Architect @ <a href="http://www.nansen.com">Nansen</a> 2016-12-06T02:53:35.0000000Z

Blog post