Timing issue between creation of service principal and delegation on Azure subscription

[CraigWiand]

The open source PowerShell script here is used to create an Azure AD Application associate it with a new service principal via New-AzureRmADServicePrincipal and then delegate that service principals rights on a target Azure subscription via New-AzureRMRoleAssignment. I have been able to reproduce a periodic timing issue when running the script where the delegation fails with a “PrincipalNotFound” http REST response. The error is inconstant and if I subsequently re-issue the same command it will succeed. I can provide the output of the Azure PowerShell cmdlets with the addition of -Debug -Verbose to the commands to return the raw REST output (ping me directly internal to MSFT please)

In the results, when https://graph.windows.net/{tenant_id}/$metadata#directoryObjects/Microsoft.DirectoryServices.ServicePrincipal is queried for the service principal, it locates the object and successfully returns attributes "objectId": "e7e918c7-536c-49c1-ac96-f3f80b89bd54". However, when a subsequent call goes to https://management.azure.com//subscriptions/{subscriptionID}/providers/Microsoft.Authorization/roleAssignments that call fails with “Principal e7e918c7536c49c1ac96f3f80b89bd54 does not exist in the directory”.

It is my belief that the graph endpoint the service principal create call is going to may be load balanced to a different backend directory instance than the ARM call is querying and potentially suffering from directory version skew until the replication of the object completes.

[lucmoco]

Same problem here. I have implemented a (rather ugly) work-around: I try several times until New-AzureRmRoleAssignment succeds:

Function NewAzureRmRoleAssignment($resourceGroupName, $objectId, $roleDefinitionName, $retryCount) {
    $totalRetries = $retryCount
    While ($True) {
        Try {
            $Null = New-AzureRmRoleAssignment -ResourceGroupName $resourceGroupName -ObjectId $objectId -RoleDefinitionName $roleDefinitionName
            Return
        }
        Catch {
            # The principal could not be found. Maybe it was just created.
            If ($retryCount -eq 0) {
                Write-Error "An error occurred: $($_.Exception)`n$($_.ScriptStackTrace)"
                throw "The principal '$objectId' cannot be granted '$roleDefinitionName' role on the resource group '$resourceGroupName'. Please make sure the principal exists and try again later."
            }
            $retryCount--
            Write-Warning "  The principal '$objectId' cannot be granted '$roleDefinitionName' role on the resource group '$resourceGroupName'. Trying again (attempt $($totalRetries - $retryCount)/$totalRetries)"
            Start-Sleep 10
        }
    }
}

In my current environment and script usually the 3rd attempt succeeds (which means I have to wait 20 seconds until the newly created app principal is taken into account).

Of course this behavior brings uncertainty and unnecessary complication to the deployment scripts so seeing it fixed would be a great improvement.

[maddieclayton]

@darshanhs90 Can you take a look at this issue?

CC @cormacpayne

[darshanhs90]

Looks like this has to do with the caching withing the service/ARM

[amoghnatu]

I'm facing this too. Do you think having a sleep statement would help?

[kumammanish]

Any luck in getting this issue resolved ? facing the same issue.

[CraigWiand]

Sadly I do not have a great resolution. I believe the issue has also been seen in the cli utility as reported here and there was a PR here. From quick glance it looks like they used a combination of passing the OID instead of the resolved name and doing a sleep if the specific error was returned.

[amoghnatu]

Hello @maddieclayton / @darshanhs90 , Is there any update or an ETA for this fix? Currently one of our customer is seeing the same problem even after putting a sleep in the code for 30 seconds! Is there any alternate for this?

Thank you.

[syedhassaanahmed]

Running into the same issue using Azure CLI. Any update on this?

[lukasmrtvy]

Same problem with CLI