Unit test pass in debug mode but fails sometime in release mode

Question

_{Tuesday, December 18, 2012 9:20 AM}

Hi

I use VS2010 and have a number of unit tests in one module. When I run the test in debug mode they all run. When I run the test one by one they all pass. But when i run the tests together from the test list editor they randomly fail in release mode. It seems timing dependent! Sometimes test pass on my local machine and fail on the build server!

The tested code does not have any static variables. Each test load a configuration from an XML file, do a lot of computation and then the results are checked.

Will all tests when started as a group of tests run at the same time?

What are the difference between debug and release mode for the tests?

My code does not have any if-debug settings or conditional compile.

I have even tried to set up a global mutex around the test to avoid any influence from any shared memory or variable I have missed, but that did not change the result.

Any ideas of what to do - wait for VS2012?

Jens Mose Pedersen

All replies (6)

_{Wednesday, December 19, 2012 1:47 PM ✅Answered}

Ohh my god. I am so ashamed. By help of a colleague I just found an unitialized class member variable.

The debug code must have initialized it for me. And that is the reason the tests worked in debug mode. Now tests work.

Sorry for the trouble.

Regards Jens

Jens Mose Pedersen

_{Tuesday, December 18, 2012 9:47 AM}

Deleted

_{Tuesday, December 18, 2012 11:36 AM}

Thank you for some hints. However please clarify a few things for me.

My code uses a C# module with a managed C++ wrapper to read my XML configurations files. My code under test and the test stubs are plain C++.

You mention "without the using(...) pattern" - what do you mean with that?

I don't think the problem is in the C#/C++ managed code, but how can I control the garbage collection?

Another question. "by default, MsTest will run them sequentially unless you have configured it to behave otherwise". Where do you control the configuration of the test sequences?

When I make a new instance of my object under test and the test stubs used for each test and I don't have any static variables, how could one test run influence the other?

Jens Mose Pedersen

_{Tuesday, December 18, 2012 9:42 PM}

Deleted

_{Wednesday, December 19, 2012 9:58 AM}

Hello Jens,

Thank you for your post.

Hope you can provide us more information to narrow down this issue:

1. Can all unit tests run successfully in Release mode with VS2010 on your local machine?
2. Did the random failure in Release mode only on the builder server?
3. What do you specify about the Debug/Release mode? The Configuration of Build pane or the Configuration of Debug pane in the testsettings?

If yes 1 and 2, I am afraid that the issue may be more related to Builder server.

If some tests also failed in Release mode with VS2010 on your local machine, you can provide us some related warning or error messages so that we can further look at this issue.

“Will all tests when started as a group of tests run at the same time?”

I think that the tests will run sequentially rather than at the same time.

If you have other environments, I suggest that you can try the same scenario on another machine to check if the issue is related to your test environment.

If I have misunderstood anything, please feel free to let me know.

Best regards,

Amanda Zhu [MSFT]
MSDN Community Support | Feedback to us
Develop and promote your apps in Windows Store
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

_{Wednesday, December 19, 2012 12:41 PM}

Hello Amanda

Just a few words about my system first. It is a rather large embedded application written mostly in C++. Basically it takes an XML parameter file, reads the parameters and use these parameters for calculation of HW setup and finally transfers the HW setup to the HW. The program consists of a lot of individual control blocks each with an abstract interface so the module easily can be stubbed of during unit tests. It is one of these control blocks I want to unit test. I use the real XML file reader that is made as a managed C++ wrapper around a C# module for reading the XML and marshal this to unmanaged data structures. This works in a lot of unit tests both in debug and release mode. The access to the HW is stubbed of with a special module that simulates the real HW. This also work for a lot of units tests.

Now the module that causes problems during test is kind of special as it uses a lot of memory on the heap, including aligned memory for intrinsic calculations. The calculation includes calculation of up to 1000000 squareroots.

I have 4 specific test runs in my unit test class. When I run each of these test on my local mashine they all pass in debug mode. No matter if I run them one by one or start them together from the Test View or the Test List Editor.

When I run the tests in release mode they also pass when I run them one by one. But when I run them together from the Test View I sometime exerience that one or more of the tests fail.

The way the test fail is typically that some internal assert or limit check inside the unit under test fails. The failure seems to be timing dependent. Right now I have enabled some debug prints during my test an then it does not fail!

The unit under test is a C++ class and is in the test created with a unique pointer. The tested class has no static variables. The other modules and stubs used during the test are also classes created with unique pointers.

I have also tried to run my application with Intels inspector tools looking for memory leaks or use of unallocated memory and found no problems. Unfortunately this tool can not be used during unit tests.

So back to your questions. As it seems like the test is timing dependent when the test are run as a block of tests it sometimes fails on the local machine and sometimes on the build server.

The tests always pass in debug mode, both locally and on the build server.

I am not sure what you mean with configuration of Build pane or debug pane. I compile my unit under test and all other modules to debug or to release. I use "run" unit test for testing in release mode.

I really cannot see how one test can influence on the execution of another when the whole unit and and all the stubs are created from scratch in each test.

Regards Jens

Jens Mose Pedersen