[sf-plugins-testkit] TestSession.create() fails with NamedOrgNotFoundError

[j-schreiber]

Summary

When using TestSession.create for NUTs (development of a custom CLI plugin), the test setup occasionally fails. I have not been able to reliably reproduce this, since the same commit may have worked yesterday and fails today.

Steps To Reproduce

See the commit history of this PR: https://github.com/j-schreiber/js-sf-cli-security-audit/pull/24/commits
Due to the nature of the problem, I do not know how to describe how to reproduce it. Here are my thoughts on how this can be reproduced

• Review this github actions run
• Checkout the corresponding commit d141d17 and investigate it

The NUTs fail with this SfError, that originates in AuthInfo.create.

{
  "name": "NamedOrgNotFoundError",
  "message": "No authorization information found for ***.",
  "exitCode": 1,
  "context": "OrgCreateScratch",
  "stack": "NamedOrgNotFoundError: No authorization information found for ***.\n    at SfCommandError.from (file:///opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@salesforce/sf-plugins-core/lib/SfCommandError.js:48:16)\n    at OrgCreateScratch.catch (file:///opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@salesforce/sf-plugins-core/lib/sfCommand.js:332:47)\n    at OrgCreateScratch._run (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@oclif/core/lib/command.js:186:29)\n    at async Config.runCommand (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@oclif/core/lib/config/config.js:456:25)\n    at async run (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@oclif/core/lib/main.js:97:16)\n    at async file:///opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/bin/run.js:15:1",
  "cause": "SfError [NamedOrgNotFoundError]: No authorization information found for ***.\n    at Messages.createError (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@salesforce/plugin-org/node_modules/@salesforce/core/lib/messages.js:448:16)\n    at AuthInfo.loadDecryptedAuthFromConfig (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@salesforce/plugin-org/node_modules/@salesforce/core/lib/org/authInfo.js:772:28)\n    at AuthInfo.initAuthOptions (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@salesforce/plugin-org/node_modules/@salesforce/core/lib/org/authInfo.js:722:53)\n    at async AuthInfo.init (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@salesforce/plugin-org/node_modules/@salesforce/core/lib/org/authInfo.js:693:13)\n    at async AuthInfo.create (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@salesforce/kit/lib/creatable.js:66:9)\n    at async authorizeScratchOrg (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@salesforce/plugin-org/node_modules/@salesforce/core/lib/org/scratchOrgInfoApi.js:177:20)\n    at async scratchOrgCreate (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@salesforce/plugin-org/node_modules/@salesforce/core/lib/org/scratchOrgCreate.js:176:32)\n    at async OrgCreateScratch.run (file:///opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@salesforce/plugin-org/lib/commands/org/create/scratch.js:214:72)\n    at async OrgCreateScratch._run (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@oclif/core/lib/command.js:182:22)\n    at async Config.runCommand (/opt/hostedtoolcache/node/24.13.0/x64/lib/node_modules/@salesforce/cli/node_modules/@oclif/core/lib/config/config.js:456:25) {\n  actions: undefined,\n  exitCode: 1,\n  context: undefined,\n  data: undefined,\n  cause: undefined\n}",
  "warnings": [],
  "code": "NamedOrgNotFoundError",
  "status": 1,
  "commandName": "OrgCreateScratch"
}

• I verified that the DevHub is successfully authenticated - with TESTKIT_AUTH_URL and TESTKIT_JWT_KEY
• I can see in the login history of my CI user, that GitHub Actions successfully authenticates
• I verified on my DevHub, that scratch orgs are successfully created.
• The same overall setup (library versions, env variables) worked 24 hours earlier

From my own investigation, it looks like StateAggregator from salesforce/sfdx-core fails to return the org credentials. The error happens here: loadDecryptedAuthFromConfig

Unfortunately, sf doctor is not available in the CI environment, since I do not control the workflow.

Expected result

When the DevHub credentials are valid, TestSession.create can successfully access the AuthInfo and create a scratch org. It then proceeds to run NUTs.

Actual result

TestSession fails with an SfError that hints towards a problem reading the authinfo from ~./sfdx
The test aborts because before all hook fails and no NUTs are executed.
A test org is created on the DevHub, but the CLI cannot complete the process.

Additional information

Some of the created scratch orgs that were created, even though "createScratchOrg" throws an error

System Information

Github runner, ubuntu-latest together with Salesforce-managed NUTs workflow: https://github.com/salesforcecli/github-workflows/blob/main/.github/workflows/nut.yml

NUT workflow uses the nightly build 2.121.3

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[github-actions]

Hello @j-schreiber 👋 It looks like you didn't include the full Salesforce CLI version information in your issue.
Please provide the output of version --verbose --json for the CLI you're using (sf or sfdx).

A few more things to check:

• Make sure you've provided detailed steps to reproduce your issue.
  • A repository that clearly demonstrates the bug is ideal.
• Make sure you've installed the latest version of Salesforce CLI. (docs)
  • Better yet, try the rc or nightly versions. (docs)
• Try running the doctor command to diagnose common issues.
• Search GitHub for existing related issues.

Thank you!

[j-schreiber]

NUTs use the nightly build of the CLI, which is 2.121.3 at the time of writing.

[j-schreiber]

I re-ran a commit that worked on January, 24th: https://github.com/j-schreiber/js-sf-cli-security-audit/actions/runs/21313959202/job/61445012663.

Today it fails with the same error: NamedOrgNotFound.

This commit used TESTKIT_AUTH_URL (while the other commits in the original PR used JWT). The Auth URL is freshly generated, so it can't be the issue.

This means there has to be something wrong with the nightly build (the other variable would have been the node_modules cache, but it hits the same hash, so I would expect that the dependencies are unchanged.

[WillieRuemmele]

Hey @j-schreiber - thanks for investigating! Just a few questions/suggestions:

1. can you add the env DEBUG=* to enable all debugging during NUTs to see if any new information is exposed
2. Be careful when using standard sfdx-core methods to manipulate the authenticated org, if I remember correctly, testkit moves a few authentication files locally into the generated NUT directory, which could cause issues with standard uses of sfdx-core
3. I see the NUT with 3 retries GHA runs, but I never see them actually try the NUTs three times

my initial hunch is that there's an error when writing the auth files to the non-standard location, and then further along in the NUTs, it throws that error because the files don't exist

---

We run probably thousands of NUTs throughout the day, and don't see this issue. I'd take a look at this plugin and just see how it compares to your setup, similar envs for authing, similar test setup, etc...

[dipakparmar]

We are also getting the same error, it was working till last week and now not, no new changes on our end. except the sf cli update.

[j-schreiber]

I aborted the retry runs because it would consume too many of my active scratch orgs / daily scratch org limits. But the last time I ran them, all 3 retries failed.

I did not use sfdx-core directly - you are right that testkit moves around auth files (method is called "transferAuth" or something like this). I only investigated the failing code because of the stack trace. Testkit essentially decides on the auth strategy and then delegates to sfdx-core to setup auth.

Unfortunately, I cannot really compare my exact setup with plugin-data. You use secrets: inherit, but this is not available to me since we are in different github organisations. I have to explicitly export the secrets. And as far as I can see, that's the only fundamental difference, besides the fact that you use 2d matrix (os and cmds), while I only use 1d matrix (os).

Comparing test setup code for convenience

data.bulk.results.nuts.ts

before(async () => {
    session = await TestSession.create({
      scratchOrgs: [
        {
          config: 'config/project-scratch-def.json',
          setDefault: true,
        },
      ],
      project: { sourceDir: path.join('test', 'test-files', 'data-project') },
      devhubAuthStrategy: 'AUTO',
    });
  });

org.quick-scan.nuts.ts

session = await TestSession.create({
      project: {
        name: 'orgQuickScanNuts',
        sourceDir: testingWorkingDir,
      },
      devhubAuthStrategy: 'AUTO',
      scratchOrgs: [
        {
          alias: scratchOrgAlias,
          config: path.join('config', 'default-scratch-def.json'),
          setDefault: true,
          duration: 1,
        },
      ],
    });

I will try and see, if the issue still occurs (it did earlier today) and will post a log file with all 3 tries.

my initial hunch is that there's an error when writing the auth files to the non-standard location, and then further along in the NUTs, it throws that error because the files don't exist

That sounds very plausible.

[dipakparmar]

According to my test, it works below cli version <= 2.121.1 , facing issue in 1.121.4.

In my issue, not an issue with plugin but sf cli it self.

[j-schreiber]

logs_55471802087.zip

Here's a log file of NUTs on ubuntu-latest that fail three times - deterministically with the same error.

[j-schreiber]

I removed the name property of my project, but it still fails: https://github.com/j-schreiber/js-sf-cli-security-audit/actions/runs/21371802023/job/61518206603

[WillieRuemmele]

@j-schreiber - if you can use a CLI version in the range provided: "it works below cli version <= 2.121.1 , facing issue in 1.121.4." and test to see, that will be a huge help

[j-schreiber]

I am using the NUTs workflow that is managed by Salesforce - as far as I can see, I can't decide the release channel that is used to install the sf CLI. It always uses "nightly".

Of course I could rebuilt the workflow myself, but this is what I tried to avoid tbh :(. Is there a way to control the CLI that is being installed?