Per Magne Skuseth
May 29, 2013
  7788
(8 votes)

EPiServer Find: Bulks, please!

Are you using Find and index lots of custom data? Improve performance by Indexing lists of objects, instead of one by one, as shown in the example below.

   1: // Not optimal 
   2: List<MyObject> objects = GetObjectsFromSomeWhere();
   3: foreach (var o in objects)
   4: {
   5:     client.Index(o);
   6: }
   1: // Better! (obviously)
   2: List<MyObject> objects = GetObjectsFromSomeWhere(); 
   3: client.Index(objects);

 

By doing this, you will significantly reduce the number of calls sent to the Find index, thus increase the general performance and decrease time taken to index.

This is fine as long as your list of objects isn’t too large, (depending on object size), but what if you have a list of 10 000 items? Or 100 000 items? Trying to index all of them at once will most likely result in a timeout error from the service. To solve this, you should split up the list and index the objects in bulks.  A simple way to do this is to create an extension method, like so:

   1: public static void IndexBulks(this IClient client, IEnumerable<object> objects, int bulkSize)
   2: {
   3:     while (objects.Any())
   4:     {
   5:         client.Index(objects.Take(bulkSize));
   6:         objects = objects.Skip(bulkSize);
   7:     }
   8: }

The extension accept a list of objects and a bulksize, and is used like this:

   1: client.IndexBulks(objects, 50);

Numbers
Indexing 1000 objects –  time taken:

  • One by one: 8 minutes, 13 seconds.
  • Bulks of 50: 4 minutes, 29 seconds. 
  • Single large bulk : As expected, the service timed out.

 

Happy indexing!

May 29, 2013

Comments

Marcus Granström
Marcus Granström May 30, 2013 10:25 AM

Very nice post Per Magne.

Thanks for sharing

Frederik Vig
Frederik Vig Oct 31, 2013 03:56 PM

Tip from Henrik Lindström: the more the better
as long as you keep below 50mb per request.

This is when calling the index method.

Henrik Fransas
Henrik Fransas Sep 27, 2018 08:46 AM

Thanks for this!

Please login to comment.
Latest blogs
Improve error handling for visitor groups

In CMS UI 12.15.0 we added several improvements in handling errors for visitor groups. The visitor group list and detailed visitor group page will...

Linh Nguyen | Dec 8, 2022

Optimizely SendGrid SMTP host is deprecated

SendGrid is a services for sending email that is included in Optimizely DXP. Previously smtp.episerver.net was the recommended SMTP server to use,...

Tomas Hensrud Gulla | Dec 4, 2022 | Syndicated blog

Hosting Optimizely CMS 12 on Docker Engine

Since Optimizely CMS can now be deployed as a Docker container, here is a demonstration of building, running and scaling an Optimizely CMS 12 site ...

Stefan Holm Olsen | Dec 4, 2022 | Syndicated blog

How to use CacheTagHelper with content areas in Optimizely CMS 12

I might be going out on a limb here - if you have a better solution, feel very free to share it!  Upgrading your Optimizely web application from .N...

Andreas J | Dec 2, 2022