[BUG] Adding a PSRuleConvention triggers more tests to run

[ReneRebsdorf]

Description of the issue

Hi,

This bug report is a bit strange, and there are probably other ways to reproduce this.
We were debugging why we are somewhat frequently getting the "bicep.exe is missing" error that other issues have been raised regarding.

As part of that process we added this file:

.ps-rule/Debug.Rule.ps1 which has write-host output of PSRULE_AZURE_BICEP_PATH. This works as expected.

However, when we added that file (even if we didn't reference the convention in the ps-rule.yaml file), the following tests triggered, flagging problems with our IaC:

• Azure.VNET.PrivateSubnet
• Azure.LogAnalytics.Replication (This is marked as Preview, so I am confused as to why that is ran, we do not pass the @pre tag)
• CAF.Tag.ResourceGroup

If we delete the Debug.Rule.ps1 file, the those tests no longer fail (e.g. they are not ran).

our .ps-rule.yaml file (at root of repo) has requires for PSRule.Rules.Azure, and PSRule.Rules.CAF, they also include both of these under include.module

Of note: When calling Assert-PSRule we provide InputPath (repo to test), Option (the yaml file in the repo to test), but also the Path property, which is a path to a library-repo containing rules we have written to ensure compliance. These rules are processed both with, and without the .ps-rule/ folder, so works as expected.

For reference, the .ps-rule/Debug.Rule.ps1 file content:

Export-PSRuleConvention 'DebugBicepCLI' -Initialize {
    Write-Host "PSRULE_AZURE_BICEP_PATH=$($Env:PSRULE_AZURE_BICEP_PATH)"
    if ($null -ne $Env:PSRULE_AZURE_BICEP_PATH) {
        Write-Host "Bicep CLI PATH=$((Get-ChildItem -Path $Env:PSRULE_AZURE_BICEP_PATH | Select-Object -First 1).FullName)"
        Write-Host "Bicep CLI version=$(& $Env:PSRULE_AZURE_BICEP_PATH --version)"
    }
}

How do we ensure the desired tests are ran from third party modules (CAF and Azure), preferably without having to provide a .ps-rule/ folder, as we often do not need it.
I tried to reproduce with adding a .ps-rule/gitkeep file, but that didn't cause the new tests to run, so it appears to be filename or file content dependant.

Error messages

N/A

Reproduction

• Try to run CAF tests without having a .ps-rule/ folder in the repo to test.

Version of PSRule

2.9.0

How are you running PSRule

locally with powershell, reproducible in Azure DevOps pipeline

Additional context

No response

[github-actions]

Thanks for raising your first issue, the team appreciates the time you have taken 😉

[BernieWhite]

@ReneRebsdorf Thanks for reporting this. Let me see if I can get a reproduction working, and get back to you, however we know there is some issues that can be observed when using multiple modules (#1215).

Noting Azure.VNET.PrivateSubnet is also a preview rule. So, rule filtering may not be applying correctly. Unsure on CAF.Tag.ResourceGroup but it could be you wanted to either excluded that rule or have overridden the CAF_ResourceGroupMandatoryTags which is not applying.

We are working on providing on a fix with v3 to provide greater isolation between modules, and that's the last major requirement for us to ship v3.

---

In terms of a workaround, I would suggest trying one of the following:

1. Override the ps-rule.yaml with a rule configuration like the target PSRule for Azure baseline. e.g.

rule:
  tag:
    release: GA

Details are here (https://github.com/Azure/PSRule.Rules.Azure/blob/main/src/PSRule.Rules.Azure/rules/Baseline.Rule.yaml)

2. Consider removing the include.module option and separate the assert-psrule or ADO tasks into separate steps for PSRule.Rules.Azure and PSRule.Rules.CAF.

---

For you other issue, setting the PSRULE_AZURE_BICEP_USE_AZURE_CLI environment variable and using the Azure CLI if it is up to date may be another option. See https://azure.github.io/PSRule.Rules.Azure/setup/setup-bicep/#using-azure-cli.

---

Hope that helps.

[ReneRebsdorf]

Hi @BernieWhite

We do have CAF_ResourceGroupMandatoryTags in our yaml file

Here are the findings from my tests:

1. Add -Module when calling Assert-PSRule, have only 1 module at a time, and remove include.module from ps-rule.yaml, have no rule.tag.release in the yaml file, a convention file in .ps-rule/, and referenced in convention.include: Azure.LogAnalytics.Replication was run which was not expected. CAF tests ran in this test
2. Running the exact same in step 1, but with rule.tag.release: GA gives the expected behaviour for the Azure rules, but causes CAF rules to not run (both as expected, but means the solution is not viable if wanting to use the same file for both CAF and Azure, but maybe if the CAF, but if more tags were added to control this, this could potentially be a viable option. Our local rules as per rule.includeLocal ran despite not having tags, which I am guessing is by design
3. Adding PSRule.Rules.Azure and PSRule.Rules.CAF to include.module and removing the use of the -Module param in Assert-PSRule did not reproduce the error, but as rule.tag.release is specified, we are missing CAF tests, as per step 2.

Based on the above, it appears there is no solution to have non-preview Azure tests, CAF tests, and a convention specified at the same time, or is there something I am missing? :-)

Wrt. PSRULE_AZURE_BICEP_USE_AZURE_CLI: We can use that variable, not a problem. Are you proposing this based on experience of fewer errors, or as a shot in the dark? I will gladly provide feedback based on our findings, although I do think this was attempted previously, but I am not 100% sure.

Lastly, do you have any guesstimates on when v3 will be GA? My team is eagerly waiting. If relevant we can also look into starting to use v3 and can provide feedback to you, but would need some info wrt. how many breaking changes you are still expecting to introduce inside of v3 (as they likely will affect the stability and amount of maintenance time needed).