Johan Antila
Sep 16, 2022
  256
(1 votes)

Azure DevOps and failing VSTest task

Sometime during the summer, our Test task for a customers Azure Devops CI/CD environment stopped working for no apparent reason and as it had been unchanged for more than a year, the problem looked like it would be related to the Azure Devops environment itself. Checking build logs, we could conclude that Microsoft did an update to the default build runner used in Azure DevOps sometime in August or perhaps late July, 2022 (We lack builds from just before so we don't know the exact date) and it included a later version of the VSTest task that had a different behaviour than the previous one. It turned out that our test task included a lot of dlls targeted for different frameworks that were not actual tests that should not have been included but this selection of dlls hadn't changed, it was always the same and it had always picked up a lot more test dlls than it should have done - all with different targets - but since that was never a problem with the previous runner, we didn't pay attention to this. Where the old version would ignore this situation, the newer version of the VSTask returned a fail when it picked up dlls from different framework targets, something it does not support, thus failing the build even though the tests it actually managed to run all passed. The solution to this is to specify a less exclusive search pattern than the default one that you get if you define the test task in your yaml file like this:

- task: VSTest@2
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
(Not specifying wildcard pattern defaults to this search pattern: **\*test*.dll,!**\*TestAdapter.dll,!**\obj\**)

In my build, this would include all these files:

vstest.console.exe "D:\a\1\s\Customer.Web\bin\EPiServer.Marketing.Testing.Core.dll"
"D:\a\1\s\Customer.Web\bin\EPiServer.Marketing.Testing.Dal.dll"
"D:\a\1\s\Customer.Web\bin\EPiServer.Marketing.Testing.Web.dll"
"D:\a\1\s\Tests\Tests.Customer.Framework\bin\Release\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pl\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pl\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pt\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pt\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pt\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\ru\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\ru\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\ru\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\tr\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\tr\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\tr\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hans\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hans\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hans\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hant\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hant\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hant\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\netcoreapp1.0\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\uap10.0\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\net45\Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\net45\Microsoft.VisualStudio.TestPlatform.TestFramework.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\netstandard1.0\Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\netstandard1.0\Microsoft.VisualStudio.TestPlatform.TestFramework.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\uap10.0\Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\uap10.0\Microsoft.VisualStudio.TestPlatform.TestFramework.dll"


After changing the way test assemblies are found like this:

- task: VSTest@2
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
    **\tests.customer*.dll
    searchFolder: '$(System.DefaultWorkingDirectory)'

    
It now includes only the relevant assemblies:

vstest.console.exe "D:\a\1\s\Tests\Tests.Customer.Framework\bin\Release\Tests.Customer.Framework.dll"
"D:\a\1\s\Tests\Tests.Customer.Framework\obj\Release\Tests.Customer.Framework.dll"
"D:\a\1\s\Tests\Tests.Customer.Web\bin\Release\Tests.Customer.Web.dll"
"D:\a\1\s\Tests\Tests.Customer.Web\obj\Release\Tests.Customer.Web.dll"

And with this, the test now passes. Succes! In other words, make sure you have a coherent naming scheme for your tests that is easy to pick up with a wildcard pattern and make sure your test task only picks up these specific dlls. You can see the output of the VSTest task in the pipeline in Azure Devops to detrmine what dlls it's trying to probe for tests.

Sep 16, 2022

Comments

Please login to comment.
Latest blogs
Dynamic Route in ASP.NET Core When MapDynamicControllerRoute Does Not Work

Background Creating one of the add-on for Optimizely I had to deal with challenge to register dynamically route for the API controller. Dynamic rou...

valdis | Sep 25, 2022 | Syndicated blog

404 Error on Static Assets Within an Optimizely plugin

Background With the move to CMS 12 and .NET 5/6, developers are now able to build Plugins and Extensions using Razor Class Libraries (RCL).  These...

Mark Stott | Sep 23, 2022

How to bypass the content creation view in Optimizely

Something that has come up a couple of times in the last few year is feedback from content editors about the editing view that comes up when creati...

Ynze | Sep 23, 2022 | Syndicated blog

Welcome to Optimizely World's New Tech Video Portal

Optimizely, leader in the digital experience realm, has become a wealth of world class SaaS products including Web Experimentation, Full Stack, B2B...

The Developer Marketing Team of Optimizely | Sep 22, 2022