EPiServer Find: Bulks, please!
Are you using Find and index lots of custom data? Improve performance by Indexing lists of objects, instead of one by one, as shown in the example below.
1: // Not optimal
2: List<MyObject> objects = GetObjectsFromSomeWhere();
3: foreach (var o in objects)
4: {
5: client.Index(o);
6: }
1: // Better! (obviously)
2: List<MyObject> objects = GetObjectsFromSomeWhere();
3: client.Index(objects);
By doing this, you will significantly reduce the number of calls sent to the Find index, thus increase the general performance and decrease time taken to index.
This is fine as long as your list of objects isn’t too large, (depending on object size), but what if you have a list of 10 000 items? Or 100 000 items? Trying to index all of them at once will most likely result in a timeout error from the service. To solve this, you should split up the list and index the objects in bulks. A simple way to do this is to create an extension method, like so:
1: public static void IndexBulks(this IClient client, IEnumerable<object> objects, int bulkSize)
2: {
3: while (objects.Any())
4: {
5: client.Index(objects.Take(bulkSize));
6: objects = objects.Skip(bulkSize);
7: }
8: }
The extension accept a list of objects and a bulksize, and is used like this:
1: client.IndexBulks(objects, 50);
Numbers
Indexing 1000 objects – time taken:
- One by one: 8 minutes, 13 seconds.
- Bulks of 50: 4 minutes, 29 seconds.
- Single large bulk : As expected, the service timed out.
Happy indexing!
Very nice post Per Magne.
Thanks for sharing
Tip from Henrik Lindström: the more the better
as long as you keep below 50mb per request.
This is when calling the index method.
Thanks for this!