Azure DevOps and failing VSTest task

Sometime during the summer, our Test task for a customers Azure Devops CI/CD environment stopped working for no apparent reason and as it had been unchanged for more than a year, the problem looked like it would be related to the Azure Devops environment itself. Checking build logs, we could conclude that Microsoft did an update to the default build runner used in Azure DevOps sometime in August or perhaps late July, 2022 (We lack builds from just before so we don't know the exact date) and it included a later version of the VSTest task that had a different behaviour than the previous one. It turned out that our test task included a lot of dlls targeted for different frameworks that were not actual tests that should not have been included but this selection of dlls hadn't changed, it was always the same and it had always picked up a lot more test dlls than it should have done - all with different targets - but since that was never a problem with the previous runner, we didn't pay attention to this. Where the old version would ignore this situation, the newer version of the VSTask returned a fail when it picked up dlls from different framework targets, something it does not support, thus failing the build even though the tests it actually managed to run all passed. The solution to this is to specify a less exclusive search pattern than the default one that you get if you define the test task in your yaml file like this:

- task: VSTest@2
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

(Not specifying wildcard pattern defaults to this search pattern: **\*test*.dll,!**\*TestAdapter.dll,!**\obj\**)

In my build, this would include all these files:

vstest.console.exe "D:\a\1\s\Customer.Web\bin\EPiServer.Marketing.Testing.Core.dll"
"D:\a\1\s\Customer.Web\bin\EPiServer.Marketing.Testing.Dal.dll"
"D:\a\1\s\Customer.Web\bin\EPiServer.Marketing.Testing.Web.dll"
"D:\a\1\s\Tests\Tests.Customer.Framework\bin\Release\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pl\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pl\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pt\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pt\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pt\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\ru\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\ru\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\ru\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\tr\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\tr\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\tr\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hans\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hans\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hans\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hant\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hant\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hant\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\netcoreapp1.0\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\uap10.0\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\net45\Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\net45\Microsoft.VisualStudio.TestPlatform.TestFramework.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\netstandard1.0\Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\netstandard1.0\Microsoft.VisualStudio.TestPlatform.TestFramework.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\uap10.0\Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\uap10.0\Microsoft.VisualStudio.TestPlatform.TestFramework.dll"

After changing the way test assemblies are found like this:

- task: VSTest@2
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
      **\tests.customer*.dll
    searchFolder: '$(System.DefaultWorkingDirectory)'

    
It now includes only the relevant assemblies:

vstest.console.exe "D:\a\1\s\Tests\Tests.Customer.Framework\bin\Release\Tests.Customer.Framework.dll"
"D:\a\1\s\Tests\Tests.Customer.Framework\obj\Release\Tests.Customer.Framework.dll"
"D:\a\1\s\Tests\Tests.Customer.Web\bin\Release\Tests.Customer.Web.dll"
"D:\a\1\s\Tests\Tests.Customer.Web\obj\Release\Tests.Customer.Web.dll"

And with this, the test now passes. Succes! In other words, make sure you have a coherent naming scheme for your tests that is easy to pick up with a wildcard pattern and make sure your test task only picks up these specific dlls. You can see the output of the VSTest task in the pipeline in Azure Devops to detrmine what dlls it's trying to probe for tests.