Dynamic data store batch saving? Poor performance.

Vote:
 

Maybe I'm reaching here, but I thought I'd try the DDS for storing large amounts of simple POCO:s - and we're talking LARGE amounts, like 100k objects. Doesn't seem to be working out that good, can't even save 10k objects in a reasonable amount of time. Is there a way to batch save objects? Or am I just pushing the limit of the DDS' capacity? I'm leaning towards just serializing everything directly instead..

#53037
Aug 25, 2011 15:27
Vote:
 

Just out of curiosity: Are you doing 10K calls to save objects or do you have an object graph containing that many objects (like an object containing a large list or something)?

I don't know all the inner workings of the DDS but I'm thinking that if you do separate calls it will end up doing separate DB calls, while when saving a large object graph at once the DDS would be "smart" and group objects into fewer inserts/updates?

If you havent tried putting your objects in an object containing a list (as you probably know you can't store a list directly), it would be interesting to know if it affects the performance.

The same question goes for using transactions - could that make performance better (or worse?). Paul describes DDS transactions here: http://world.episerver.com/Blogs/Paul-Smith/Dates1/2011/4/Dynamic-Data-Store--Tips-Tricks-and-Best-Practises/

#53066
Aug 26, 2011 8:13
Vote:
 

Hi Magnus,

Thanks it's a good idea, I'll try it today. I thought about transactions too but if I'm still just saving them one by one I don't think they will have an positive impact on performance (maybe worsen it). But I'll try that as well (I should be using them anyway).

#53067
Aug 26, 2011 8:39
Vote:
 

Unfortunately I rembered incorrectly - I am in fact using an object graph already. 1 object that holds a few properties and a list of 10k+ simple POCO:s. I tried DDS ExecuteTransaction as well, none of these methods gave any increase in performance.

Saving 1 wrapper object that holds a list of 1k objects takes around 11 seconds. 100 objects clocks in at around 1 second.

#53074
Aug 26, 2011 9:59
Vote:
 

FYI I ended up using a wrapper with a byte array for each type containing the serialized objects, then storing the wrapper to the DDS (= lightning fast). Kind of meaningless to use the DDS for this maybe but at least I don't have to write to the db myself.

#53079
Aug 26, 2011 11:18
Vote:
 

Thanks for sharing your results!

#53082
Aug 26, 2011 11:29
This thread is locked and should be used for reference only. Please use the Episerver CMS 7 and earlier versions forum to open new discussions.
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.