Error when running methods in parallel (using Parallel.ForEach)

Anton Piliev

Vote:

Hello,

We recently migrated the project to the ASP.NET Core. After this, we faced a bug with code that runs several methods in parallel.

We have a product mapper class that gets prepared variant models using this code:

var variants = new ConcurrentDictionary<Variant, VariantDto>();
Parallel.ForEach(
    activeVariants,
    new ParallelOptions
    {
        MaxDegreeOfParallelism =
            Convert.ToInt32(Environment.ProcessorCount * 0.8), // max 80% CPU
    },
    v => variants.TryAdd(v, _variantMapper.Map(v)));

Inside "_variantMapper.Map" we use "PromotionEngine.Run()" method and "CustomerContext.Current.GetContactById(customerId)" method. And when loading the product page, one of these methods throws an error every time. Here are examples of these errors:

CustomerContext.Current.GetContactById(customerId) - throws "Object reference not set to an instance of an object" exception.

Stack trace:

   at Mediachase.BusinessFoundation.Data.Business.BusinessManager.Execute(Request request)
   at Mediachase.BusinessFoundation.Data.Business.BusinessManager.Load(String metaClassName, PrimaryKeyId primaryKeyId)
   at Mediachase.Commerce.Customers.CustomerContext.InnerGetContactById(Guid contactId)
   at Mediachase.Commerce.Customers.CustomerContext.<>c__DisplayClass32_0.<GetContactById>b__0()
   at Mediachase.Commerce.Customers.CustomersCache.<>c__DisplayClass6_0`1.<ReadThrough>g__WrapperAction|0()
   at EPiServer.Framework.Cache.ObjectInstanceCacheExtensions.ReadThrough[T](IObjectInstanceCache cache, String key, Func`1 readValue, Func`2 evictionPolicy)
   at Mediachase.Commerce.Extensions.ObjectInstanceCacheExtensions.ReadThrough[T](IObjectInstanceCache cache, Boolean useCache, String cacheKey, IEnumerable`1 masterKeys, TimeSpan duration, Func`1 load)
   at Mediachase.Commerce.Extensions.ObjectInstanceCacheExtensions.ReadThrough[T](IObjectInstanceCache cache, String cacheKey, IEnumerable`1 masterKeys, TimeSpan duration, Func`1 load)
   at Mediachase.Commerce.Customers.CustomersCache.ReadThrough[T](String key, IEnumerable`1 masterKeys, TimeSpan timeout, Func`1 load)
   at Mediachase.Commerce.Customers.CustomerContext.GetContactById(Guid contactId)
   at Commerce.Carts.Extensions.IOrderRepositoryExtensions.GetCart(IOrderRepository orderRepository, Guid customerId, String cartName) in ...\Extensions\IOrderRepositoryExtensions.cs:line 15
   at Commerce.Carts.CartService.Get(Guid customerId) in ...CartService.cs:line 66
   at Commerce.Carts.CartService.Get() in ...CartService.cs:line 61
   at Commerce.Variants.VariantMapper.GetAllowedQuantity(Variant variant, Boolean& isInCart, Decimal qtyPerBaseUnitOfMeasure) in ...VariantMapper.cs:line 645
   at Commerce.Variants.VariantMapper.Map(Variant variant) in ...VariantMapper.cs:line 358
   at Commerce.Products.ProductMapper.<>c__DisplayClass14_0`1.<Map>b__5() in ...ProductMapper.cs:line 117

PromotionEngine.Run() - throws "Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct" exception.

Stack trace:

   at System.ThrowHelper.ThrowInvalidOperationException_ConcurrentOperationsNotSupported()
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
   at EPiServer.Personalization.VisitorGroups.VisitorGroupRole.IsMatchCached(IPrincipal principal, HttpContext httpContext)
   at EPiServer.Personalization.VisitorGroups.VisitorGroupRole.IsMatch(IPrincipal principal, HttpContext httpContext)
   at EPiServer.Commerce.Marketing.CampaignVisitorGroupFilter.IsInVisitorGroup(Guid visitorGroupId, HttpContext httpContext, String campaignName)
   at EPiServer.Commerce.Marketing.CampaignVisitorGroupFilter.ShouldFilter(ContentReference campaignLink, IDictionary`2 cachedVisitorGroupLookup, PromotionFilterContext filterContext)
   at EPiServer.Commerce.Marketing.CampaignVisitorGroupFilter.Filter(PromotionFilterContext filterContext)
   at EPiServer.Commerce.Marketing.PromotionFilters.Filter(IEnumerable`1 allPromotions, IEnumerable`1 couponCodes, RequestFulfillmentStatus requestedStatus)
   at EPiServer.Commerce.Marketing.PromotionEngine.Run(IOrderGroup orderGroup, PromotionEngineSettings settings)
   at Commerce.Discounts.CustomPromotionEngine.Run(IOrderGroup orderGroup, PromotionEngineSettings settings) in ...CustomPromotionEngine.cs:line 42
   at Commerce.Marketing.IPromotionEngineExtensions.GetDiscountedPrice(IPromotionEngine promotionEngine, VariationContent variant, IMarket market, RequestFulfillmentStatus requestFulfillmentStatus, IPriceValue price) in ...IPromotionEngineExtensions.cs:line 65
   at Commerce.Variants.VariantMapper.GetSaleAndDefaultPrices(Variant variant, ProductRelationshipDto outletParent, IMarket market, Nullable`1 ctoCode, Boolean isAvailableForCurrentCustomer, Boolean procaseEnabled, IReadOnlyList`1& promotions) in ...VariantMapper.cs:line 421
   at Commerce.Variants.VariantMapper.Map(Variant variant) in ...VariantMapper.cs:line 367
   at Commerce.Products.ProductMapper.<>c__DisplayClass14_0`1.<Map>b__5() in ...ProductMapper.cs:line 117

I found this article https://world.optimizely.com/blogs/Johan-Bjornfot/Dates1/2023/8/parallel-tasks-and-backgroundcontext/ and tried to use "IBackgroundContextFactory.Create()" inside the lambda method in "Parallel.ForEach". Here's the updated code:

Parallel.ForEach(
    activeVariants,
    new ParallelOptions
    {
        MaxDegreeOfParallelism =
            Convert.ToInt32(Environment.ProcessorCount * 0.8), // max 80% CPU
    },
    v =>
    {
        using (_backgroundContextFactory.Create())
        {
            variants.TryAdd(v, _variantMapper.Map(v));
        }
    });

Thanks to this code, the number of errors has decreased. Only one type of error occurs: "Object reference not set to an instance of an object" inside the "CustomerContext.Current.GetContactById(Guid contactId)" (it seems that "PromotionEngine.Run()" use this method too). And I only got one of these errors the first time I loaded the page, and the next time I loaded the product page it opened (at least the error didn't occur every time I loaded the page). But if I reload the page in the browser several times right after it loads, I can still catch one of these errors.

I'm not sure what we can do from our code side to get rid of this error. Can you help me how to properly run several methods in parallel so that a this exception doesn't occur?

Thanks!

#311302

Edited, Oct 23, 2023 10:14

Quan Mai

Vote:

The question is why do you need to run these in parallel? Many code is not meant to be thread safe (i.e. to run in parallel)

If you aim to improve performance, maybe try to profile to see where the bottleneck is?

#311305

Oct 23, 2023 11:07

Anton Piliev

Vote:

Hi Quan, thanks for your reply.

We need this because some products may have more than 100 options and mapping them all one by one will take a lot of time, which is not acceptable for our client.

We have complex logic for variant mapping, including calculating the displayed price depending on applied and available discounts, custom calculation of available quantities and other business logic. Yes, perhaps we can review the code in the mapper and gain some time, but in any case, without parallel processing of variants, this process may take much longer.

If you mean that we can't use parallel processing (and therefore methods like "Task.WhenAll") if we want to use "ContentRepository" or methods from "CustomerContext.Current", then of course we will have to look for another solution. But before that, I would like to understand if there is any way to make these methods work.

#311307

Oct 23, 2023 11:56

Anton Piliev

Vote:

Another example of what for we use "Parallel.ForEach" is API that can accept thousands of product data and save them in catalog. We process them in parallel and save them using "IContentRepository.Save" method. This method throws different types of errors: "The transaction operation cannot be performed because there are pending requests working on this transaction" , "'Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct", null reference errors.

#311308

Oct 23, 2023 12:08

Quan Mai

Vote:

You can look at IContentRepository.Publish which can publish multiple content at once. it was added to improve performance, without having to use multiple threads.

#311390

Oct 24, 2023 10:06

Quan Tran - Dec 19, 2023 4:19

Hello Quan, I am facing the same exception and I have already checked the IContentRepository and there are no method can publish multiple contents as you said (we are using episerver.cms.core\12.19.0), Could you please provide more details?

Quan Mai - Dec 19, 2023 7:12

The publish one is for catalog content only, unfortunately https://world.optimizely.com/blogs/Quan-Mai/Dates/2019/10/new-simple-batch-saving-api-for-commerce/

Anton Piliev

Vote:

Thanks, I'll take a look and see if this method helps with saving content. But I still want to understand if I can do something to fix the first issue.

I also want to note that the same code works fine in the previous version. What has changed so that now the same code fails with an error?

#311394

Oct 24, 2023 13:35

Quan Mai

Vote:

it is very difficult to track down what change has "caused" that. as these codes touch a lot of objects, running them in parallel can produce non determistic results

#311395

Oct 24, 2023 15:49