Logs are unavailable when running tests from the Test Explorer

[TheJCAB]

Brief Issue Summary

Hi! The new Test Explorer integration is great, but... when tests fail, it's hard or impossible to get at the output from the failed test.

On my first run, a Terminal window was opened that was showing test output. But all the tests seem to be writing to it in parallel, so all the outputs were garbled.

On subsequent runs, that Terminal window was no longer updated (no output was shown for the newer runs).

On the Output window, it still showed the log of which tests were being launched (not their output). At the end of the run, it used to say "[ctest] Output from these tests are in: /Testing/Temporary/LastTest.log", and clicking on this file would load nothing at all (just three lines, showing nothing more that the entire test run start/stop times). The full test logs then could be found in a different file in the same directory that included date and time, like for instance "LastTest_20230425-2240.log".

That Output window behavior was... before. Now, the "LastTest_20230425-2240.log" only includes the output from the last test run. I think each test that runs is overwriting the previous test's output (from the same run). So the output is being lost.

Please:

• Preserve the output of all the tests that run.
• Provide a way from the Test Explorer to open the log of a particular test. Especially from failed tests.
• Figure out what's happening to the Terminal window:
  • Why it seems to stop receiving output, and...
  • Why the output form all the tests are being mixed together there.

Thanx!

CMake Tools Diagnostics

{
  "os": "win32",
  "vscodeVersion": "1.77.3",
  "cmtVersion": "1.14.30",
  "configurations": [
    {
      "folder": "x:\\TTD4\\CoreTtd.Tests",
      "cmakeVersion": "3.20.5",
      "configured": true,
      "generator": "Ninja",
      "usesPresets": true,
      "compilers": {
        "C": "clang-cl.exe",
        "CXX": "clang-cl.exe"
      }
    }
  ],
  "cpptoolsIntegration": {
    "isReady": true,
    "hasCodeModel": true,
    "activeBuildType": "RelWithDebInfo",
    "buildTypesSeen": [
      "RelWithDebInfo"
    ],
    "requests": [
      "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/TestX87Registers/Main.cpp",
      "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/AVXTest/Source.cpp",
      "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/AVXTest/AVXTest.cpp",
      "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/AVXTest/AVX2Test.cpp",
      "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/TestAVX2Vgather/Source.cpp",
      "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/TestFullCacheLineAccess/TestFullCacheLineAccess.cpp"
    ],
    "responses": [
      {
        "uri": "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/TestX87Registers/Main.cpp",
        "configuration": {
          "includePath": [
            "x:/ttd4/corettd.tests/src/testcommon/inc",
            "x:/ttd4/corettd.tests/build/x64-checked-clang/src/testcommon/geninc"
          ],
          "defines": [
            "DBG",
            "TAEF_NO_STD_FORMAT",
            "TTD_COMPILER_TOOLCHAIN_CLANG",
            "TTD_INTERNAL_TEAM_USE_ONLY",
            "UNDOCKED_WINDOWS_UCRT",
            "UNICODE",
            "WIL_SUPPRESS_PRIVATE_API_USE",
            "_UNICODE"
          ],
          "compilerPath": "c:/program files/microsoft visual studio/2022/preview/vc/tools/llvm/x64/bin/clang-cl.exe",
          "compilerArgs": [],
          "compilerFragments": [
            " --target=x86_64-pc-windows -MT",
            "/EHsc",
            "/Oxs",
            "/Ob2",
            "/Oy-",
            "/Brepro",
            "/Z7",
            "/d2Zi+",
            "/Zl",
            "/Zp8",
            "/GF",
            "/ZH:SHA_256",
            "/FC",
            "/bigobj",
            "/W4",
            "/WX",
            "/GR-",
            "/Gw",
            "/Gy",
            "/wd4324",
            "/wd4499",
            "/D_SILENCE_CLANG_COROUTINE_MESSAGE",
            "-mcx16",
            "-ferror-limit=100",
            "-Wno-assume",
            "-Wno-deprecated-declarations",
            "-Wno-ignored-attributes",
            "-Wno-invalid-offsetof",
            "-Wno-microsoft-include",
            "-Wno-microsoft-cast",
            "-Wno-missing-field-initializers",
            "-Wno-typename-missing",
            "-Wno-unused-variable",
            "-std:c++20"
          ]
        }
      },
      {
        "uri": "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/AVXTest/Source.cpp",
        "configuration": {
          "includePath": [
            "x:/ttd4/corettd.tests/src/testcommon/inc",
            "x:/ttd4/corettd.tests/build/x64-checked-clang/src/testcommon/geninc"
          ],
          "defines": [
            "DBG",
            "TAEF_NO_STD_FORMAT",
            "TTD_COMPILER_TOOLCHAIN_CLANG",
            "TTD_INTERNAL_TEAM_USE_ONLY",
            "UNDOCKED_WINDOWS_UCRT",
            "UNICODE",
            "WIL_SUPPRESS_PRIVATE_API_USE",
            "_UNICODE"
          ],
          "compilerPath": "c:/program files/microsoft visual studio/2022/preview/vc/tools/llvm/x64/bin/clang-cl.exe",
          "compilerArgs": [],
          "compilerFragments": [
            " --target=x86_64-pc-windows -MT",
            "/EHsc",
            "/Oxs",
            "/Ob2",
            "/Oy-",
            "/Brepro",
            "/Z7",
            "/d2Zi+",
            "/Zl",
            "/Zp8",
            "/GF",
            "/ZH:SHA_256",
            "/FC",
            "/bigobj",
            "/W4",
            "/WX",
            "/GR-",
            "/Gw",
            "/Gy",
            "/wd4324",
            "/wd4499",
            "/D_SILENCE_CLANG_COROUTINE_MESSAGE",
            "-mcx16",
            "-ferror-limit=100",
            "-Wno-assume",
            "-Wno-deprecated-declarations",
            "-Wno-ignored-attributes",
            "-Wno-invalid-offsetof",
            "-Wno-microsoft-include",
            "-Wno-microsoft-cast",
            "-Wno-missing-field-initializers",
            "-Wno-typename-missing",
            "-Wno-unused-variable",
            "-std:c++20"
          ]
        }
      },
      {
        "uri": "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/AVXTest/AVXTest.cpp",
        "configuration": {
          "includePath": [
            "x:/ttd4/corettd.tests/src/testcommon/inc",
            "x:/ttd4/corettd.tests/build/x64-checked-clang/src/testcommon/geninc"
          ],
          "defines": [
            "DBG",
            "TAEF_NO_STD_FORMAT",
            "TTD_COMPILER_TOOLCHAIN_CLANG",
            "TTD_INTERNAL_TEAM_USE_ONLY",
            "UNDOCKED_WINDOWS_UCRT",
            "UNICODE",
            "WIL_SUPPRESS_PRIVATE_API_USE",
            "_UNICODE"
          ],
          "compilerPath": "c:/program files/microsoft visual studio/2022/preview/vc/tools/llvm/x64/bin/clang-cl.exe",
          "compilerArgs": [],
          "compilerFragments": [
            " --target=x86_64-pc-windows -MT",
            "/EHsc",
            "/Oxs",
            "/Ob2",
            "/Oy-",
            "/Brepro",
            "/Z7",
            "/d2Zi+",
            "/Zl",
            "/Zp8",
            "/GF",
            "/ZH:SHA_256",
            "/FC",
            "/bigobj",
            "/W4",
            "/WX",
            "/GR-",
            "/Gw",
            "/Gy",
            "/wd4324",
            "/wd4499",
            "/D_SILENCE_CLANG_COROUTINE_MESSAGE",
            "-mcx16",
            "-ferror-limit=100",
            "-Wno-assume",
            "-Wno-deprecated-declarations",
            "-Wno-ignored-attributes",
            "-Wno-invalid-offsetof",
            "-Wno-microsoft-include",
            "-Wno-microsoft-cast",
            "-Wno-missing-field-initializers",
            "-Wno-typename-missing",
            "-Wno-unused-variable",
            "-std:c++20",
            "-arch:AVX"
          ]
        }
      },
      {
        "uri": "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/AVXTest/AVX2Test.cpp",
        "configuration": {
          "includePath": [
            "x:/ttd4/corettd.tests/src/testcommon/inc",
            "x:/ttd4/corettd.tests/build/x64-checked-clang/src/testcommon/geninc"
          ],
          "defines": [
            "DBG",
            "TAEF_NO_STD_FORMAT",
            "TTD_COMPILER_TOOLCHAIN_CLANG",
            "TTD_INTERNAL_TEAM_USE_ONLY",
            "UNDOCKED_WINDOWS_UCRT",
            "UNICODE",
            "WIL_SUPPRESS_PRIVATE_API_USE",
            "_UNICODE"
          ],
          "compilerPath": "c:/program files/microsoft visual studio/2022/preview/vc/tools/llvm/x64/bin/clang-cl.exe",
          "compilerArgs": [],
          "compilerFragments": [
            " --target=x86_64-pc-windows -MT",
            "/EHsc",
            "/Oxs",
            "/Ob2",
            "/Oy-",
            "/Brepro",
            "/Z7",
            "/d2Zi+",
            "/Zl",
            "/Zp8",
            "/GF",
            "/ZH:SHA_256",
            "/FC",
            "/bigobj",
            "/W4",
            "/WX",
            "/GR-",
            "/Gw",
            "/Gy",
            "/wd4324",
            "/wd4499",
            "/D_SILENCE_CLANG_COROUTINE_MESSAGE",
            "-mcx16",
            "-ferror-limit=100",
            "-Wno-assume",
            "-Wno-deprecated-declarations",
            "-Wno-ignored-attributes",
            "-Wno-invalid-offsetof",
            "-Wno-microsoft-include",
            "-Wno-microsoft-cast",
            "-Wno-missing-field-initializers",
            "-Wno-typename-missing",
            "-Wno-unused-variable",
            "-std:c++20",
            "-arch:AVX2"
          ]
        }
      },
      {
        "uri": "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/TestAVX2Vgather/Source.cpp",
        "configuration": {
          "includePath": [
            "x:/ttd4/corettd.tests/src/testcommon/inc",
            "x:/ttd4/corettd.tests/build/x64-checked-clang/src/testcommon/geninc"
          ],
          "defines": [
            "DBG",
            "TAEF_NO_STD_FORMAT",
            "TTD_COMPILER_TOOLCHAIN_CLANG",
            "TTD_INTERNAL_TEAM_USE_ONLY",
            "UNDOCKED_WINDOWS_UCRT",
            "UNICODE",
            "WIL_SUPPRESS_PRIVATE_API_USE",
            "_UNICODE"
          ],
          "compilerPath": "c:/program files/microsoft visual studio/2022/preview/vc/tools/llvm/x64/bin/clang-cl.exe",
          "compilerArgs": [],
          "compilerFragments": [
            " --target=x86_64-pc-windows -MT",
            "-arch:AVX2",
            "/EHsc",
            "/Oxs",
            "/Ob2",
            "/Oy-",
            "/Brepro",
            "/Z7",
            "/d2Zi+",
            "/Zl",
            "/Zp8",
            "/GF",
            "/ZH:SHA_256",
            "/FC",
            "/bigobj",
            "/W4",
            "/WX",
            "/GR-",
            "/Gw",
            "/Gy",
            "/wd4324",
            "/wd4499",
            "/D_SILENCE_CLANG_COROUTINE_MESSAGE",
            "-mcx16",
            "-ferror-limit=100",
            "-Wno-assume",
            "-Wno-deprecated-declarations",
            "-Wno-ignored-attributes",
            "-Wno-invalid-offsetof",
            "-Wno-microsoft-include",
            "-Wno-microsoft-cast",
            "-Wno-missing-field-initializers",
            "-Wno-typename-missing",
            "-Wno-unused-variable",
            "-std:c++20"
          ]
        }
      },
      {
        "uri": "file:///x%3A/TTD4/CoreTtd.Tests/src/TestPrograms/TestFullCacheLineAccess/TestFullCacheLineAccess.cpp",
        "configuration": {
          "includePath": [],
          "defines": [
            "DBG",
            "TTD_COMPILER_TOOLCHAIN_CLANG",
            "TTD_INTERNAL_TEAM_USE_ONLY",
            "UNDOCKED_WINDOWS_UCRT",
            "UNICODE",
            "WIL_SUPPRESS_PRIVATE_API_USE",
            "_UNICODE"
          ],
          "compilerPath": "c:/program files/microsoft visual studio/2022/preview/vc/tools/llvm/x64/bin/clang-cl.exe",
          "compilerArgs": [],
          "compilerFragments": [
            " --target=x86_64-pc-windows -MT",
            "-arch:AVX",
            "/EHsc",
            "/Oxs",
            "/Ob2",
            "/Oy-",
            "/Brepro",
            "/Z7",
            "/d2Zi+",
            "/Zl",
            "/Zp8",
            "/GF",
            "/ZH:SHA_256",
            "/FC",
            "/bigobj",
            "/W4",
            "/WX",
            "/GR-",
            "/Gw",
            "/Gy",
            "/wd4324",
            "/wd4499",
            "/D_SILENCE_CLANG_COROUTINE_MESSAGE",
            "-mcx16",
            "-ferror-limit=100",
            "-Wno-assume",
            "-Wno-deprecated-declarations",
            "-Wno-ignored-attributes",
            "-Wno-invalid-offsetof",
            "-Wno-microsoft-include",
            "-Wno-microsoft-cast",
            "-Wno-missing-field-initializers",
            "-Wno-typename-missing",
            "-Wno-unused-variable",
            "-std:c++20"
          ]
        }
      }
    ],
    "partialMatches": [],
    "targetCount": 40,
    "executablesCount": 32,
    "librariesCount": 8,
    "targets": []
  },
  "settings": [
    {
      "communicationMode": "automatic",
      "useCMakePresets": "auto",
      "configureOnOpen": true
    }
  ]
}

Debug Log

I can provide if needed.

Additional Information

No response

[xisui-MSFT]

It may not be very obvious, but you can use this button to get the test output:

This contains the output of all test runs in the current VS Code session.

[ivikhrev]

xisui-MSFT hi!
For gtest output is messed up:

Could it be fixed?

[xisui-MSFT]

It's a line feed issue. I thought I fixed it. I will take a look later.

[TheJCAB]

xisui-MSFT ah, thanx! That was not entirely obvious. So now we have five different sorts of output:

1. The "output" pane, as we had before. Doesn't show much, just what individual test invocations CTest performs. It used to show a summary at the end, but now it no longer does (because it seems to invoke CTest multiple times, one per test, which makes sense).
2. The test results "terminal" pane (added to the list together with cmd, powershell and whatnot). I don't know how I got this one, the first time I used the Test Explorer. I can't figure out how to get it back now. This one got console output from the test invocations, but the output was lines from all invocations, interleaved with each other.
3. The "test results" pane (not to be confused with the "terminal" one above). This is what you were showing me. Thanx! So... this one allows me to select individual runs, but only shows me the first 1024 characters of output from each, which can make it useless (if the data relevant to the failure was clipped).
4. The test explorer itself. You can see the status of the last test run from each test, but can't do anything there to get the output of a particular run. This is what I was asking about in this issue. It'd be most logical to preserve the complete output from all tests and allow you to find it from here. IMHO. The "test results" pane would then feel useless.
5. The test log files. As I mentioned, at the end of a run, the output was offering "LastTest.log" as the file containing the log. This was incorrect. The log could be found in a file in the same directory, with that name but decorated with date/time. But now, this file only contains the output from one single test of the test run.

Besides having way too many different output options, the end result is much poorer now because data (log output) is lost in the process.

I hope this clarifies my sentiment about this. I love that this "Test Explorer" feature is being done, but now it's time to iron out the kinks, of which there are several.

[TheJCAB]

Well... one mystery solved... I managed to get the "terminal" pane showing test output. It's called "Test Output at ##:##:##" in the list of terminals.

I got that by pressing the "Show result output" icon in the right side of the "test results" pane. This only appears for the root entry of an entire run. It looks identical to the button in the Test Explorer that xisui-MSFT pointed out earlier. Let's see if I can capture it...

Also, as you can see in the image, for some test runs, for absolutely no reason I can think of, the "test results" pane only shows the single line "Expecting one test result". It's... bizarre.

But I digress... I think, at this point.

[TheJCAB]

The issue with the entire test output being "Expecting one test result" likely relates to #3150 . In this case, I think the relevant tests are supposed to be ignored via label (specified in a test preset), which CTest will normally be fine with. But... in this case, cmake-tools is invoking CTest to run just the one test that... doesn't (because of the label).

This error message ends up being confusing, if not misleading.

[triou-harmonicinc]

Hi,
Any news on this ticket ?
I still see the messed output for gtest based tests (actual logs are obfuscated):

Plus, the content are truncated.

Also when selecting a test case, no output is reported.

[v-frankwang]

Thibault (@triou-harmonicinc) Thank you for following up.There is no word yet on this ticket，thank you for your patience.