Introduction
Azure DevOps holds testing in two places that look like they should be one.
Azure Pipelines runs your suite and shows the outcome on a Tests tab: which tests ran, which failed, how long they took. Azure Test Plans holds Test Case work items, organised into suites and plans, which people run by hand and link to requirements.
The natural assumption is that the first fills the second — run the suite, watch the test cases update. It does not. Publishing results creates a test run; a Test Case is a work item, and nothing in the publish task creates one. The two are joined only when someone explicitly associates an automated test with a test case.
This article covers both halves: getting a real Playwright run into the Tests tab without the four documented defaults quietly swallowing it, and getting test cases into Test Plans without destroying the steps that were already there.
What we ran and what we did not. We ran the Playwright suite below against Sauce Demo on Playwright 1.63.0, Node.js 20.20.2 and Chromium 153.0.8010.12, and every line of output and XML here is from that run. We did not run an Azure Pipelines pipeline: we have no organisation to run one in. Every statement about what Azure DevOps does with these files is quoted from Microsoft's own reference documentation and linked at the point it is used. Where we could turn a documented behaviour into something checkable, we built a tool for it and show what that tool actually returned.
The run that produces the file
Two tests against Sauce Demo, a public practice site. The second one expects the wrong wording on purpose, so the run has a failure to publish:
import { expect, test } from "@playwright/test";
const PASSWORD = process.env.SAUCE_PASSWORD ?? "secret_sauce";
test("a standard user can log in", async ({ page }) => {
await page.goto("/");
await page.getByPlaceholder("Username").fill("standard_user");
await page.getByPlaceholder("Password").fill(PASSWORD);
await page.getByRole("button", { name: "Login" }).click();
await expect(page.getByText("Products")).toBeVisible();
});
test("a locked out user is told why", async ({ page }) => {
await page.goto("/");
await page.getByPlaceholder("Username").fill("locked_out_user");
await page.getByPlaceholder("Password").fill(PASSWORD);
await page.getByRole("button", { name: "Login" }).click();
// Deliberately the wrong wording, so this run has one failure to publish.
await expect(page.locator('[data-test="error"]')).toHaveText("Sorry, this user has been banned.");
});The JUnit reporter is what Azure Pipelines reads. Note the file name:
import { defineConfig } from "@playwright/test";
export default defineConfig({
testDir: "./tests",
reporter: [["list"], ["junit", { outputFile: "results/junit.xml" }]],
use: { baseURL: "https://www.saucedemo.com" },
});Running it:
npx playwright test 1 failed
tests\checkout.spec.ts:13:5 › a locked out user is told why ────────────────
1 passed (7.6s)And the file it wrote, trimmed to its shape:
<testsuites id="" name="" tests="2" failures="1" skipped="0" errors="0" time="7.553719">
<testsuite name="checkout.spec.ts" timestamp="2026-09-23T10:56:35.973Z" hostname="" tests="2" failures="1" skipped="0" time="6.292" errors="0">
<testcase name="a standard user can log in" classname="checkout.spec.ts" time="0.776">
</testcase>
<testcase name="a locked out user is told why" classname="checkout.spec.ts" time="5.516">
<failure message="expect(locator).toHaveText(expected) failed" type="expect.toHaveText">
<![CDATA[ Expected: "Sorry, this user has been banned."
Received: "Epic sadface: Sorry, this user has been locked out."]]>
</failure>
</testcase>
</testsuite>
</testsuites>Every field the Tests tab shows comes from an attribute here. Microsoft's result formats mapping table for JUnit says the test result title comes from testcase@name, the test file from testcase@classname, the duration from testcase@time, and the outcome is Failed when a failure or error child exists, Not Executed when skipped exists, and Passed otherwise. That is why test titles are worth writing as sentences: the title in your spec is the title a developer reads in the pipeline three weeks later.
The pipeline that publishes nothing
Here is the step as people usually write it first — the format named, everything else left alone:
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: npm ci
displayName: Install
- script: npx playwright install --with-deps chromium
displayName: Install browsers
- script: npx playwright test
displayName: Run tests
- task: PublishTestResults@2
inputs:
testResultsFormat: 'JUnit'This looks finished. It is not, and the reasons are all documented defaults rather than bugs.
We built a checker for exactly this: paste the pipeline and the results file, and it says what the task will do with them, citing the documentation for each claim. Run against the YAML above and the results/junit.xml from our run, it returned six findings. Two of them are high:
HIGH Results won't publish when tests fail
This step has no condition, so it runs only if everything before it
succeeded. The run that most needs its results, the one where tests
failed, publishes nothing. Add condition: succeededOrFailed().
HIGH The pattern doesn't match your results file
testResultsFiles is **/TEST-*.xml, which doesn't match results/junit.xml.The pattern matches nothing
testResultsFiles has a documented default of **/TEST-*.xml. That pattern comes from Java's Surefire, which writes TEST-com.example.LoginTest.xml. Playwright writes whatever you name it, and almost nobody names it TEST-something.xml.
On its own that would be a visible mistake. It pairs with a second default to become an invisible one: failTaskOnMissingResultsFile also defaults to false. The task finds nothing, publishes nothing, and succeeds. The pipeline is green and the Tests tab is empty, which most people read as "no tests ran" rather than "the pattern was wrong".
The step does not run when it matters
A step with no condition uses the default, which requires everything before it to have succeeded. Your npx playwright test step exits non-zero when tests fail. So on every run with a failure — the only runs whose results anyone urgently wants — the publish step is skipped.
Failing tests do not fail the task
failTaskOnFailedTests defaults to false. The documentation is explicit about what that means: "The default is false, which will simply publish the results from the results file."
So the task succeeds with failures in the file. If nothing else in the job fails the run, the pipeline is green and the failures live only in the Tests tab. Whether you want this on depends on where you want the gate: if your test step already fails the job, the publish task failing too is noise; if you run tests with a step that swallows the exit code, this switch is the only thing standing between a broken build and a green tick.
Each file becomes its own run
mergeTestResults defaults to false, so several result files appear as several separate test runs. The documentation adds a detail worth knowing before you shard a suite across twenty jobs: "results will always be merged into a single run if there are more than 100 result files even if this option is set to false."
The pipeline that reports
Same job, with the four defaults answered:
steps:
- script: npx playwright test
displayName: Run tests
- task: PublishTestResults@2
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: 'results/junit.xml'
mergeTestResults: true
failTaskOnMissingResultsFile: true
testRunTitle: 'Playwright $(Build.SourceBranchName)'failTaskOnFailedTests is deliberately absent. The npx playwright test step already fails the job when a test fails, so turning it on would fail the run twice for one reason. Turn it on if — and only if — your test step cannot fail on its own.
Attachments are worth one line. The JUnit reporter writes markers like [[ATTACHMENT|path]] into system-out, and the task's attachments support section says it reads exactly that pattern from testsuites/testsuite/testcase/system-out. Traces and screenshots reach the test result without another task.
Test Plans is a different system
Nothing above creates a Test Case. Published results are test runs; Test Cases are work items with steps, expected results, an area path and a state. If your team has a suite of manual cases, they get there by hand, through the API, or by bulk import.
The bulk import format is a CSV whose shape surprises people: one row per step, not per case. The case's own fields repeat on every row, and the Test Step column counts up.
For Azure DevOps Services the documentation lists nine required fields: ID, Work Item Type, Title, Test Step, Step Action, Step Expected, Area Path, Assigned To and State. On Azure DevOps Server the same page notes that Area Path is not required, and that the import does not modify it.
Four details decide whether an import works:
IDempty creates,IDfilled updates. Leave the column blank for new cases.Work Item Typemust be exactlyTest Case. Spelling and casing both matter.Statemust be one your process accepts. The documented default for a new case isDesign, in the Proposed state category.Area Pathmust already exist, written asMyProject\MyArea— a backslash, not a slash.
And one rule that has cost people work:
Reimporting a test case with a matching ID replaces all existing test steps with the steps in your file. Missing steps are removed.
Export the whole case before you edit it. A CSV with two of a case's five steps does not update two steps; it leaves the case with two.
Writing the CSV from scenarios you already have
Typing that column layout by hand is where the mistakes come from. Our CSV builder takes Gherkin or plain action -> expected lines and writes the documented columns. Given this scenario outline:
Scenario Outline: Log in to the shop
Given the login page is open
When I sign in as <user>
Then I see <outcome>
Examples:
| user | outcome |
| standard_user | the product list |
| locked_out_user | Epic sadface: Sorry, this user has been locked out. |it returned this, with the area path and owner filled in on the page:
ID,Work Item Type,Title,Test Step,Step Action,Step Expected,Area Path,Assigned To,State
,Test Case,Log in to the shop,1,the login page is open I sign in as @user,I see @outcome,Shop\Checkout,[email protected],DesignTwo things there are worth reading closely.
<user> became @user. Azure Test Plans writes data-driven parameters with an @ prefix, so a Gherkin placeholder maps onto one cleanly. The values do not: parameter values are not among the documented import columns, so the tool hands them over separately and says so:
@user,@outcome
standard_user,the product list
locked_out_user,"Epic sadface: Sorry, this user has been locked out."That second row is quoted because the expected text contains a comma — the documentation's own advice is to enclose values containing commas or line breaks in double quotes, and to save as UTF-8.
Three Gherkin lines became one step. Given/When became the action and Then became the expected result, because the import format has exactly one action and one expected result per step. That is our convention, not Microsoft's: Gherkin does not say which Given belongs to which Then, so anything that splits a scenario into steps is guessing. The tool shows you what it produced so you can fix the guess before importing, which is the point of generating a file rather than an import.
What still is not connected
Publishing a run and importing cases leaves a gap neither fills: the automated test in your repository and the Test Case work item in Test Plans are still two records of the same intention. Linking them is a separate act — associating an automated test with a test case — and until someone does it, a green pipeline says nothing about the state of a test plan.
That is worth knowing before anyone promises a dashboard that shows both. It is also the honest limit of this article: we have not run that association, so we are not describing it.
Checklist
- The publish step has
condition: succeededOrFailed(), or it will skip the runs you care about. testResultsFilesnames the file your runner actually writes, not**/TEST-*.xml.failTaskOnMissingResultsFile: true, so a wrong path fails loudly instead of publishing nothing.mergeTestResults: trueif one logical run writes several files.failTaskOnFailedTestsonly if your test step cannot fail the job by itself.- Test titles read as sentences, because the title is what appears in the Tests tab.
- Before editing an exported test case CSV, export the whole case — a partial reimport deletes the steps you left out.
Work Item Typeis exactlyTest Case,Area Pathalready exists,Stateis one your process allows.
Conclusion
Most of what goes wrong here is not a bug in anyone's pipeline. It is four defaults, each reasonable on its own, that combine into a green run with no results in it: a pattern from a different test framework, a missing file that is not an error, a failing test that is not a failure, and a step that does not run when anything before it failed.
The fix is four lines of YAML. Finding out that you need them usually takes a release where nobody noticed the Tests tab was empty. If you want to check yours without waiting for that, paste your pipeline and your results file into the checker — and if you are moving written scenarios into Test Plans, let the CSV builder write the columns rather than typing them.
Sources and further reading
- PublishTestResults@2 — Publish Test Results v2 task, Microsoft Learn. Input defaults, the JUnit field mapping table, and attachment support. Page dated 1 September 2026; read 23 September 2026.
- Bulk import or export test cases, Microsoft Learn. Required columns, the one-row-per-step layout, and the rule that a reimport replaces all steps. Page dated 10 August 2026; read 23 September 2026.
- Repeat a test with different data, Microsoft Learn. How
@parameterswork in a test case. - Specify conditions, Microsoft Learn. What a step with no condition does.
- File matching patterns, Microsoft Learn. The minimatch rules
testResultsFilesfollows. - Playwright JUnit reporter, Playwright documentation. The reporter that wrote the file above.