Blog posts by Matt FitzGerald-Chamberlain2025-12-11T01:30:00.0000000Zhttps://world.optimizely.com/blogs/matt-fitzgerald-chamberlain/Optimizely WorldData Imports in Optimizely: Part 2 - Query data efficiently<p class="">One of the more time consuming parts of an import is looking up data to update. Naively, it is possible to use the PageCriteriaQueryService to query a each page, one at a time as it is needed. While this does work, it can add significant overhead as it requires potentially thousands of queries.</p><p class="">Instead, since virtually all pages are going to need to be accessed, it is more efficient to preload all of the pages using batch APIs and then query them from the in-memory data.</p><p class="">The key here is using the ListContentOfContentType method of IContentModelUsage to get all of the content references using the content type, and the GetItems() method of IContentRepository to batch load items.</p>
<pre><code class="language-csharp">var allPageRefs = _contentModelUsage
.ListContentOfContentType(communityContentType)
.Select(x => x.ContentLink.ToReferenceWithoutVersion());
var allPages = _contentRepository
.GetItems(allPageRefs, new LoaderOptions { LanguageLoaderOption.MasterLanguage() })
.OfType<ImportedPage>()
.GroupBy(x => x.LegacyID)
.ToDictionary(x => x.Key, x => x.First());</code></pre>
<p class="">Then, to load the page for editing, it’s just a matter of looking up the ID in the dictionary.</p>
<pre><code class="language-csharp">if (allPages.TryGetValue(importedData.Id, out var page))
{
page = (ImportedPage)page.CreateWritableClone();
}
else
{
page = (ImportedPage)_contentRepository.GetDefault<ImportedPage>(importFolder).CreateWritableClone();
}</code></pre>
<p class="">Some considerations:</p><ol data-rte-list="default"><li><p class="">This can use a significant amount of memory as it is loading a potentially large segment of the database. In my testing, this can even be multiple GB of data. However, RAM is plentiful and in my testing it is well within the amount of available memory on Optimizely DXP.</p></li><li><p class="">Loading all of the items takes a couple of seconds. However the alternative is much more. The PageCriteriaQueryService can take a significant amount of time to query each time it is used, which can add up to many minutes of additional time, just querying pages.</p></li><li><p class="">ListContentOfContentType() may return older/other versions of content, not necessarily the latest published version. This is why in the example I stripped the version with ToReferenceWithoutVersion(). </p></li></ol>2025-12-11T01:30:00.0000000ZBlog postData Imports in Optimizely: Part 3 - Only save when necessary<p class="">Saving to the Optimizely database is generally the most time consuming part of an import. The import can be sped up by saving as infrequently as possible.</p><p class="">The first time the import is run, it will be a worst case scenario as every item needs to be created in the database. However, subsequent runs where not all of the data is changing can be much faster.</p><p class="">The simplest option is to just check if a property is changing and only saving if it is different:</p>
<pre><code class="language-csharp">var needsSave = false;
if (page.Title != importedData.Title) {
page.Title = importedData.Title;
needsSave = true;
}
if (needsSave) {
_contentRepository.Save(page, SaveAction.Publish | SaveAction.Patch, AccessLevel.NoAccess);
}</code></pre>
<p class="">This can be tedious when there are dozens of properties to update. We can improve this by creating a function to perform this check for us.</p>
<pre><code class="language-csharp">private static void UpdateProperty<TPage, TProperty>(TPage page, string propertyName, TProperty newValue, ref bool needsSave) {
var currentValue = page.Property[propertyName];
if (currentValue != newValue) {
page.Property[propertyName] = newValue;
needsSave = true;
}
}
// Update the import method
UpdateProperty(page, nameof(ImportedPage.Title), importedData.Title, ref needsSave);</code></pre>2025-11-03T00:15:06.0000000ZBlog postData Imports in Optimizely: Part 1 - Writing efficient data imports2019-05-28T15:13:10.0000000ZBlog post