Random MessageBox when executing tests

Question

Random MessageBox when executing tests

Lorenzo Zambelli 170

Hello,

we are running vstest.console in Visual Studio 2022 Enterprise Edition on a Hyper-V Windows 11 24H2 virtual machine, and we randomly get the blocking messagebox you see in the screenshot below, does anyone know the cause of that and how to fix it?

Thank you in advance for your support. image_2025_03_04T12_43_27_197Z

Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-05T09:24:24.74+00:00

Hi @Lorenzo Zambelli ,

The window seems doesn’t provide any error message. If you click the OK button or close it, will the test continue running? Will the tests pass?

Do you use a custom .runsettings file? If so, can you confirm that this issue is not caused by one of your custom configurations in the .runsettings file?

Is this issue replicated on a newly created test project?

Besides, I see many messages say "The test attribute has a null or empty value: Owner". Do you consider fixing this warning and testing again?
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-10T13:11:07.0766667+00:00
Hi @Lorenzo Zambelli ,

Does this issue persist? If you try to run the PowerShell as administrator, will it work?

I suggest you try following to troubleshoot this issue

Try to use vstest.console.exe to run a newly created simple project, see if the same issue appears. It's helpful to confirm if this issue is caused by specific configurations or tests.

Try to test your project on another machine if possible. It's helpful to confirm if this issue is caused by the environment(system).
Lorenzo Zambelli 170 Reputation points

2025-03-11T07:32:14.0366667+00:00

Hello @Tianyu Sun-MSFT ,

thank you for your replies and sorry for my delay.

The window seems doesn’t provide any error message. If you click the OK button or close it, will the test continue running? Will the tests pass?

When I click ok, the tests will continue running, but it won't always fail.

Do you use a custom .runsettings file? If so, can you confirm that this issue is not caused by one of your custom configurations in the .runsettings file?

We don't use a .runsettings file

Is this issue replicated on a newly created test project?

Actually all our tests are new because we migrating to this platform from a previous custom one.

Besides, I see many messages say "The test attribute has a null or empty value: Owner". Do you consider fixing this warning and testing again?

How can we fix that?

Does this issue persist? If you try to run the PowerShell as administrator, will it work?

This issue is persisting, at the moment running PowerShell as administrator the messagebox has not yet appeared, but because it's random I need to run some more tests to confirm that admin role will fix it, but if that's the case how could I run my Azure DevOps pipeline process as administrator?
Lorenzo Zambelli 170 Reputation points

2025-03-11T07:40:03.7933333+00:00

Removed duplicated comment.
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-11T13:58:24.1433333+00:00

Hi @Lorenzo Zambelli,

...but if that's the case how could I run my Azure DevOps pipeline process as administrator?

In that case, I suggest you try to add Azure DevOps tag for further help.

I'm not very sure if there are some settings need to be set there, but in that case, at least the cause is confirmed, and it's related to permissions.

Besides, from your description, I doubt that this window may be a warning window that some files or resources cannot be accessed.

Is there any special permission or rule on your machine? Are you running other processes at the same time?
Lorenzo Zambelli 170 Reputation points

2025-03-11T15:09:05.5066667+00:00

Hello @Tianyu Sun-MSFT ,

after several runs I can confirm that the PowerShell as administrator won't solve the issue.

So our best hope is to fix the warnings you saw in my previous screenshot... Do you have any idea how to fix them?
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-12T10:02:14.85+00:00

Hi @Lorenzo Zambelli ,

I saw that the warning message mentioned test attribute. Did you check the test attribute in your test file? Are there any special or custom settings for the test attribute in your test file?

I also saw that the vstest.console.exe searched and read the YAML file, is there any corresponding setting in the YAML file?

You can try to test with a newly created simple test file(I knew that your tests are also new, maybe try a simple one), I believe that these warnings won’t appear. If so, try to match and test the test attributes, see if one of them triggers the warning.

A simple test for example:
Lorenzo Zambelli 170 Reputation points

2025-03-12T13:24:05.74+00:00

Hello @Tianyu Sun-MSFT ,

thank you for your suggestions.

We fixed the "empty owner" warning, but nothing changed, we still have the MessageBox.

The values from .yaml files are correctly read, otherwise our tests won't run correctly.

I can try a brand new test like you suggested that won't require a .yaml file, but I realized that I forgot to say that we are using C++, not C#... I don't know if this could help you...
Lorenzo Zambelli 170 Reputation points

2025-03-14T09:17:43.1066667+00:00

Hello @Tianyu Sun-MSFT ,

we tried with an empty test as you suggested, and after 100 runs I got no MessageBox.

So, I guess it's something wrong in what we are testing, we'll add some loggers trying to understand the cause, meanwhile I don't know if you know if there's a verbose or debug mode of vstest.console.exe.

Thank you again for your time and support, best regards!
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-14T09:18:29.71+00:00

Hi @Lorenzo Zambelli ,

The same. On my side, either using C# or C++, the unit test runs correctly by using vstest.console.exe.

May I know if you have tried a brand new test, does the same issue appear?
Lorenzo Zambelli 170 Reputation points

2025-03-14T09:22:03.2066667+00:00

Hi @Tianyu Sun-MSFT ,

it seems that the issue happens when we load the module that will execute the test...
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-14T09:39:54.6433333+00:00

Hi @Lorenzo Zambelli ,

meanwhile I don't know if you know if there's a verbose or debug mode of vstest.console.exe.

There's a parameter => /Diag. It records the log of the running process of vstest.console.exe. I would suggest you try it. For example .\vstest.console.exe C:\MyTest.dll /Diag:log.txt.

The log.txt file will be generated and saved to the current directory.
Lorenzo Zambelli 170 Reputation points

2025-03-14T11:45:02.4266667+00:00

alert.txt

Hi ,

I attached the log we got when the MessageBox showed, I hope can help to understand why it happened.
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-14T14:12:30.8233333+00:00
Hi @Lorenzo Zambelli ,

Thanks. I have checked the log. I see many errors about "Microsoft.DiaSymReader.Native.xxxxx.dll", and "Error: System.BadImageFormatException: Failed to load file or assembly".

I doubt the issue is caused by the mismatching of one certain configuration.

If it’s possible to test this test file/project on another machine, maybe we can narrow this issue down. If the same issue appears on another machine, it means you might need to check the test project/file related configurations. If the same issue doesn’t appear on another machine, it means that some system/VS configurations need to be checked/repaired.

I would suggest you check following:

Third-party program like antivirus, if you are using. Try to disable them temporarily and test.

The log also mentions machine.config(C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config) and vstest.console.exe.Config(C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe.Config) file, try to check them, see if some configurations have been modified. About machine.config file, there’s a default machine.config.default file in the same directory, you can try to match their contents.

As you are using C++, consider if some assemblies, tools are loaded incorrectly (just for example, not for native code/tests, but for .NET/.NET Framework tests…).
Lorenzo Zambelli 170 Reputation points

2025-03-14T14:29:23.4666667+00:00

Hi @Tianyu Sun-MSFT ,

thank you for checking the log I sent.

I can reproduce the error in production VMs and also on my laptop, with different versions of Visual Studio, so I agree it's something related to our project configurations.

We got the log on a VM, where there's only Defender running as protection software.

We checked the two machine.config files you mentioned, they have the same content...

I attach our vstest.console.exe.Config.

vstest.console.exe.config.txt

1 answer

Your answer

Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-05T09:24:24.74+00:00

Hi @Lorenzo Zambelli ,

The window seems doesn’t provide any error message. If you click the OK button or close it, will the test continue running? Will the tests pass?

Do you use a custom .runsettings file? If so, can you confirm that this issue is not caused by one of your custom configurations in the .runsettings file?

Is this issue replicated on a newly created test project?

Besides, I see many messages say "The test attribute has a null or empty value: Owner". Do you consider fixing this warning and testing again?
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-10T13:11:07.0766667+00:00

Hi @Lorenzo Zambelli ,

Does this issue persist? If you try to run the PowerShell as administrator, will it work?

I suggest you try following to troubleshoot this issue

Try to use vstest.console.exe to run a newly created simple project, see if the same issue appears. It's helpful to confirm if this issue is caused by specific configurations or tests.

Try to test your project on another machine if possible. It's helpful to confirm if this issue is caused by the environment(system).
Lorenzo Zambelli 170 Reputation points

2025-03-11T07:32:14.0366667+00:00

Hello @Tianyu Sun-MSFT ,

thank you for your replies and sorry for my delay.

The window seems doesn’t provide any error message. If you click the OK button or close it, will the test continue running? Will the tests pass?

When I click ok, the tests will continue running, but it won't always fail.

Do you use a custom .runsettings file? If so, can you confirm that this issue is not caused by one of your custom configurations in the .runsettings file?

We don't use a .runsettings file

Is this issue replicated on a newly created test project?

Actually all our tests are new because we migrating to this platform from a previous custom one.

Besides, I see many messages say "The test attribute has a null or empty value: Owner". Do you consider fixing this warning and testing again?

How can we fix that?

Does this issue persist? If you try to run the PowerShell as administrator, will it work?

This issue is persisting, at the moment running PowerShell as administrator the messagebox has not yet appeared, but because it's random I need to run some more tests to confirm that admin role will fix it, but if that's the case how could I run my Azure DevOps pipeline process as administrator?
Lorenzo Zambelli 170 Reputation points

2025-03-11T07:40:03.7933333+00:00

Removed duplicated comment.
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-11T13:58:24.1433333+00:00

Hi @Lorenzo Zambelli,

...but if that's the case how could I run my Azure DevOps pipeline process as administrator?

In that case, I suggest you try to add Azure DevOps tag for further help.

I'm not very sure if there are some settings need to be set there, but in that case, at least the cause is confirmed, and it's related to permissions.

Besides, from your description, I doubt that this window may be a warning window that some files or resources cannot be accessed.

Is there any special permission or rule on your machine? Are you running other processes at the same time?
Lorenzo Zambelli 170 Reputation points

2025-03-11T15:09:05.5066667+00:00

Hello @Tianyu Sun-MSFT ,

after several runs I can confirm that the PowerShell as administrator won't solve the issue.

So our best hope is to fix the warnings you saw in my previous screenshot... Do you have any idea how to fix them?
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-12T10:02:14.85+00:00

Hi @Lorenzo Zambelli ,

I saw that the warning message mentioned test attribute. Did you check the test attribute in your test file? Are there any special or custom settings for the test attribute in your test file?

I also saw that the vstest.console.exe searched and read the YAML file, is there any corresponding setting in the YAML file?

You can try to test with a newly created simple test file(I knew that your tests are also new, maybe try a simple one), I believe that these warnings won’t appear. If so, try to match and test the test attributes, see if one of them triggers the warning.

A simple test for example:
Lorenzo Zambelli 170 Reputation points

2025-03-12T13:24:05.74+00:00

Hello @Tianyu Sun-MSFT ,

thank you for your suggestions.

We fixed the "empty owner" warning, but nothing changed, we still have the MessageBox.

The values from .yaml files are correctly read, otherwise our tests won't run correctly.

I can try a brand new test like you suggested that won't require a .yaml file, but I realized that I forgot to say that we are using C++, not C#... I don't know if this could help you...
Lorenzo Zambelli 170 Reputation points

2025-03-14T09:17:43.1066667+00:00

Hello @Tianyu Sun-MSFT ,

we tried with an empty test as you suggested, and after 100 runs I got no MessageBox.

So, I guess it's something wrong in what we are testing, we'll add some loggers trying to understand the cause, meanwhile I don't know if you know if there's a verbose or debug mode of vstest.console.exe.

Thank you again for your time and support, best regards!
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-14T09:18:29.71+00:00

Hi @Lorenzo Zambelli ,

The same. On my side, either using C# or C++, the unit test runs correctly by using vstest.console.exe.

May I know if you have tried a brand new test, does the same issue appear?
Lorenzo Zambelli 170 Reputation points

2025-03-14T09:22:03.2066667+00:00

Hi @Tianyu Sun-MSFT ,

it seems that the issue happens when we load the module that will execute the test...
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-14T09:39:54.6433333+00:00

Hi @Lorenzo Zambelli ,

meanwhile I don't know if you know if there's a verbose or debug mode of vstest.console.exe.

There's a parameter => /Diag. It records the log of the running process of vstest.console.exe. I would suggest you try it. For example .\vstest.console.exe C:\MyTest.dll /Diag:log.txt.

The log.txt file will be generated and saved to the current directory.
Lorenzo Zambelli 170 Reputation points

2025-03-14T11:45:02.4266667+00:00

alert.txt

Hi ,

I attached the log we got when the MessageBox showed, I hope can help to understand why it happened.
Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-14T14:12:30.8233333+00:00

Hi @Lorenzo Zambelli ,

Thanks. I have checked the log. I see many errors about "Microsoft.DiaSymReader.Native.xxxxx.dll", and "Error: System.BadImageFormatException: Failed to load file or assembly".

I doubt the issue is caused by the mismatching of one certain configuration.

If it’s possible to test this test file/project on another machine, maybe we can narrow this issue down. If the same issue appears on another machine, it means you might need to check the test project/file related configurations. If the same issue doesn’t appear on another machine, it means that some system/VS configurations need to be checked/repaired.

I would suggest you check following:

Third-party program like antivirus, if you are using. Try to disable them temporarily and test.

The log also mentions machine.config(C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config) and vstest.console.exe.Config(C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe.Config) file, try to check them, see if some configurations have been modified. About machine.config file, there’s a default machine.config.default file in the same directory, you can try to match their contents.

As you are using C++, consider if some assemblies, tools are loaded incorrectly (just for example, not for native code/tests, but for .NET/.NET Framework tests…).
Lorenzo Zambelli 170 Reputation points

2025-03-14T14:29:23.4666667+00:00

Hi @Tianyu Sun-MSFT ,

thank you for checking the log I sent.

I can reproduce the error in production VMs and also on my laptop, with different versions of Visual Studio, so I agree it's something related to our project configurations.

We got the log on a VM, where there's only Defender running as protection software.

We checked the two machine.config files you mentioned, they have the same content...

I attach our vstest.console.exe.Config.

vstest.console.exe.config.txt

Answer 1

Lorenzo Zambelli 170

Hi @Tianyu Sun-MSFT ,

we got it! In a parameter passed by yaml file we passed a non existing directory.. Thank you for your support!

Tianyu Sun-MSFT 34,311 Reputation points Microsoft External Staff

2025-03-19T06:51:39.47+00:00

Great!

Sounds the issue is indeed caused by the configurations in the yaml file. It also explains why this error doesn’t appear without using the yaml file. But we might should confirm this at the very beginning.

Anyway, I’m glad to hear that this issue doesn’t block you anymore. Thank you for sharing the solution and the cause here. I appreciate it.

Have a great day!

Tianyu

Share via

Random MessageBox when executing tests

1 answer

Your answer