Per Magne Skuseth
May 29, 2013
  7891
(8 votes)

EPiServer Find: Bulks, please!

Are you using Find and index lots of custom data? Improve performance by Indexing lists of objects, instead of one by one, as shown in the example below.

   1: // Not optimal 
   2: List<MyObject> objects = GetObjectsFromSomeWhere();
   3: foreach (var o in objects)
   4: {
   5:     client.Index(o);
   6: }
   1: // Better! (obviously)
   2: List<MyObject> objects = GetObjectsFromSomeWhere(); 
   3: client.Index(objects);

 

By doing this, you will significantly reduce the number of calls sent to the Find index, thus increase the general performance and decrease time taken to index.

This is fine as long as your list of objects isn’t too large, (depending on object size), but what if you have a list of 10 000 items? Or 100 000 items? Trying to index all of them at once will most likely result in a timeout error from the service. To solve this, you should split up the list and index the objects in bulks.  A simple way to do this is to create an extension method, like so:

   1: public static void IndexBulks(this IClient client, IEnumerable<object> objects, int bulkSize)
   2: {
   3:     while (objects.Any())
   4:     {
   5:         client.Index(objects.Take(bulkSize));
   6:         objects = objects.Skip(bulkSize);
   7:     }
   8: }

The extension accept a list of objects and a bulksize, and is used like this:

   1: client.IndexBulks(objects, 50);

Numbers
Indexing 1000 objects –  time taken:

  • One by one: 8 minutes, 13 seconds.
  • Bulks of 50: 4 minutes, 29 seconds. 
  • Single large bulk : As expected, the service timed out.

 

Happy indexing!

May 29, 2013

Comments

Marcus Granström
Marcus Granström May 30, 2013 10:25 AM

Very nice post Per Magne.

Thanks for sharing

Frederik Vig
Frederik Vig Oct 31, 2013 03:56 PM

Tip from Henrik Lindström: the more the better
as long as you keep below 50mb per request.

This is when calling the index method.

Henrik Fransas
Henrik Fransas Sep 27, 2018 08:46 AM

Thanks for this!

Please login to comment.
Latest blogs
Optimizely PIM - Variant Management Overhaul & Rollout

Optimizely PIM has completely overhauled variant management with improved usability and support for variant swatches. In order to enable this...

Arthur Vander Voort | Feb 4, 2023

You Just Got Vectored! SVG Image Formats

 If you're reading this, then you've come across a need that nearly all Opti developers encounter in their careers; You need to display a vector...

Greg J | Feb 3, 2023 | Syndicated blog

CMS Core 12.12.0 delisted from Nuget feed

We have decided to delist version 12.12.0 of the CMS Core packages from our Nuget feed. As a consequence, we are also temporarily delisting version...

Magnus Rahl | Feb 3, 2023

How Optimizely Web Experimentation enables fast validation of design and UX changes

Ever wondered how to increase conversion rates and quickly validate design/UX alternatives to find the most impactful changes? I wrote a practical...

Jacob Pretorius | Jan 31, 2023