Virtual Happy Hour this month, Jun 28, we'll be getting a sneak preview at our soon to launch SaaS CMS!

Try our conversational search powered by Generative AI!

SqlBlobProvider - OutOfMemoryException, EPiServer 9

Vote:
 

Hello,

I have a scheduled job, which tries to save several thousand images in the EPi CMS and I want to use SqlBlobProvider to achive this, becasue of the many benefits, which it provides.

When I use the default EPi provider (EPiServer.Framework.Blobs.FileBlobProvider), the scheduled job completed without any problems - around 4000 images are saved to the local file storage.
But when i switch to the SqlBlogProvider, OutOfMemoryException occurrs and the job fails.

(EPiServer version - 9.12.3)

//The EXCEPTION:
//at System.IO.MemoryStream.set_Capacity(Int32 value)
//at System.IO.MemoryStream.EnsureCapacity(Int32 value)
//at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
//at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
//at System.IO.Stream.CopyTo(Stream destination)
//at EPiCode.SqlBlobProvider.SqlBlob.Write(Stream stream)
//at DACL.SHOne.Jobs.CioViewNodesImporter.Steps.CreateImagesFilesStep.ExecuteCore(ItemsStepContext`2 context)

Do you have any suggestions?

#171564
Edited, Nov 11, 2016 8:08
Vote:
 

Probably because streams are not handled correctly and you are reading every image into memory byte array somewhere. Might be in SqlBlobprovider I guess...

This will fill up heap memory with many images and since byte array of images are large the garbage collector will be less effective. Just guessing mind you :)

#171577
Nov 11, 2016 9:41
Vote:
 

It would seem that you're working with pretty large files in order to get an OOM exception. Is that something you could check?

/Steve

#171588
Nov 11, 2016 13:37
Vote:
 

Run it in batches will probably work as workaround. Copy 100 at a time...

#171590
Nov 11, 2016 13:50
Vote:
 

It's not writing all in once, but each file, in a loop. It could be that the GC does not release it in time, so maybe force GC once a while during import (if this is a one-time thing).

/Steve

#171597
Nov 11, 2016 15:55
Vote:
 

Yes but maybe limit scheduled job to run 100 items at the time in a loop...wait 10 mins to let GC work...run it again to handle 101-200 etc

Ugly but will probably work :)

Can you show the relevant code in scheduled job? Maybe you are keeping references to the objects that keep them from being GCed?

#171598
Edited, Nov 11, 2016 15:58
Vote:
 

You can force the GC, so you don't have to wait for so long :-) Ugly, but sometimes ugly is what it takes!

#171600
Nov 11, 2016 16:31
Vote:
 

Hi guys,

Thank you all for the answers!

Sorry for the late reply.
I was working on another issues, but now I'm back to that.

So, here you can see part of our job code:

  if (item.File == null)
  {
    var overview = item.Detail.Overview;

    var imageFolder = overview.ImagesFolder.ContentLink;
    var imageFile = _contentRepository.GetDefault<ImageFile>(imageFolder);
    var extension = Path.GetExtension(item.Url);

    var resolutions = item.Variants.GroupBy(variant => new { variant.Width, variant.Height }).Select(group => group.Key);
    imageFile.Name = string.Join("_", resolutions.Select(resolution => resolution.Width + "-" + resolution.Height)) + extension;

    try
    {
        using (var client = new WebClient())
        {
            var myUri = new Uri(item.Url, UriKind.Absolute);
            var blob = _blobFactory.CreateBlob(imageFile.BinaryDataContainer, extension);

            using (var stream = client.OpenRead(myUri))
            {
                try
                {
                    //NNumber of times called: 846
                    //The exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll
                    blob.Write(stream);
                }
                catch (OutOfMemoryException ex)
                { }
            }

            imageFile.BinaryData = blob;
            _contentRepository.Save(imageFile, SaveAction.Publish);
            item.File = imageFile;
        }
 }


blob.Write(stream); is the key place

The size of the image files is around 3 MB (maximum).

About the leaving more time for the GC, i tried with Thread.Sleep(1000) for every single image - without success...

The separation in batches will definitely do the job, but It'll cost more time for refactoring. The strange here is that with the default provider (EPiServer.Framework.Blobs.FileBlobProvider) it works fine, and that makes me think that maybe the DB connection is the reason for OOM - less connections to the DB more data in the heap... 

Here you can see some memory measurement - http://prntscr.com/d8dap5
 

#171749
Edited, Nov 17, 2016 10:51
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.