Issues with running tests in parallel

Vote:
 

Hi

It seems that attempting to run otpimizely solution in parallel using aspnetCore.Mvc.Testing and its webApplicationFactory to run component tests results in exceptions:

  Message: 
System.ArgumentException : An item with the same key has already been added. Key: ConfigureContainer

  Stack Trace: 
Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
Dictionary`2.Add(TKey key, TValue value)
TimeMeters.Start(String key)
ModuleNode.Execute(Action a, String key)
ModuleNode.ConfigureContainer(ServiceConfigurationContext context)
InitializationEngine.ConfigureCurrentModules(Boolean final)
InitializationEngine.ConfigureModules(IEnumerable`1 modules)
ServiceCollectionExtensions.AddCmsHost(IServiceCollection services)
ServiceCollectionExtensions.AddCms(IServiceCollection services)
  Message: 
System.InvalidOperationException : Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.

  Stack Trace: 
Dictionary`2.FindValue(TKey key)
Dictionary`2.TryGetValue(TKey key, TValue& value)
AssemblyScanner.HasEPiServerFrameworkDependency(RuntimeLibrary library, IDictionary`2 allLibraries, IDictionary`2 resolvedAssemblies)
<>c__DisplayClass4_0.<GetScannableAssemblies>b__1(RuntimeLibrary library)
WhereArrayIterator`1.MoveNext()
SelectManySingleSelectorIterator`2.MoveNext()
SelectEnumerableIterator`2.MoveNext()
WhereEnumerableIterator`1.ToArray()
Enumerable.ToArray[TSource](IEnumerable`1 source)
AssemblyScanner.GetScannableAssemblies()

It is random which exception i will get but it seems that there is static list of something somewhere inside.

For now walkaround is to run tests sequentially but this is more a quick hack than a solution.

#292394
Edited, Nov 29, 2022 7:39
Vote:
 

How's running tests sequentially a quick hack? I think it's a valid solution. 

#292396
Nov 29, 2022 9:42
Vote:
 

Parallel execution of tests is so baked into test runs that xUnit doesn't even have flag to run tests in series - you walkaround it by putting tests into one collection.

Not being able to run tests in parallel means that your code is probably not thread safe, and it is considered a major red flag.

#292439
Nov 30, 2022 7:21
Vote:
 

OK that's new to me, I have been not using xUnit for quite a long time, and i assume they shifted to parallel execution in later version.

What you are trying to say, I suppose, is that TimeMeters.Start and AssemblyScanner.HasEPiServerFrameworkDependency are not thread safe. they aren't. however if they should be thread safe or not, is quite arguable. they are meant to be used by initialization process so there should be no multi threading involved.

which kind of tests you are trying to run? Why does it need to initialize the framework? 

#292441
Nov 30, 2022 9:59
Vote:
 

As i be live i've written i'm using mvc testkit that runs host:

        [Theory]
        [InlineData("/")]
        public async Task PageReturns200(string url)
        {
            var suite = new HttpTestWorker();
            var response = await suite.HttpClient.GetAsync(url);
            Assert.Equal(System.Net.HttpStatusCode.OK, response.StatusCode);
        }

TestWorker:
        protected override IHost _host { get; }
        private HttpClient _httpClient;
        public HttpClient HttpClient => _httpClient;

        private HttpTestWorker()
        {
            var factory = new CustomWebApplicationFactory<Program>(Register);
            _httpClient = factory.CreateClient();
            _host = factory.GetHost();

            AddDefaultMocks();
        }
        protected void Register(IServiceCollection services)
        {
            services.ConfigureBasicMocks();
            ReplaceOverride(services);
        }
        public static void ConfigureBasicMocks(this IServiceCollection services)
        {
            MockDatabase(services);
            RemoveHostedServices(services);
        }
        private static void MockDatabase(IServiceCollection services)
        {
            var options = services.Where(x => x.ServiceType.Name.Contains("DbContextOptions")).ToList();

            foreach (var service in options)
            {
                services.Remove(service);
            }

            var optionsBuilder = new DbContextOptionsBuilder<KpiDatabaseContext>();
            optionsBuilder.UseInMemoryDatabase("kpiDatabase");

            services.AddSingleton(optionsBuilder.Options);


            var optionsBuilder2 = new DbContextOptionsBuilder<EPiServer.Marketing.Testing.Dal.DatabaseContext>();
            optionsBuilder2.UseInMemoryDatabase("kpiDatabase");

            services.AddSingleton(optionsBuilder2.Options);

            var optionsBuilder3 = new DbContextOptionsBuilder();
            optionsBuilder3.UseInMemoryDatabase("epi");

            services.AddSingleton(optionsBuilder3.Options);
        }
and from mvc test kit:
public class CustomWebApplicationFactory<TProgram> : WebApplicationFactory<TProgram> where TProgram : class
#292519
Edited, Dec 01, 2022 13:15
Vote:
 

also currently overriding dbcontextoptions doesn't seem to work as expected

So why i'm doing initialziation? - because i'm running whole host inmemory.

Why i'm running entire host inmemory? - because epi is a minefield of extension methods with service locators in them and other traps like "content area", that makes unit tests as joyful as prostate exam.

#292520
Edited, Dec 01, 2022 13:16
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.