Skip to content

online-deployment issue "The Managed Inference service creation is taking longer than our normal time" #4088

Closed
Closed
online-deployment issue "The Managed Inference service creation is taking longer than our normal time"#4088
Assignees
RakeshMohanMSFT
Labels
Machine LearningService AttentionThis issue is responsible by Azure service team.customer-reportedIssues that are reported by GitHub users external to the Azure organization.extension/ml

Description

@bhushangholave
Bhushan Gholave (bhushangholave)
opened on Nov 9, 2021

az --version
azure-cli 2.29.0 *

az extension list

[
{
"experimental": false,
"extensionType": "whl",
"name": "ml",
"path": "/home/icertisadmin/.azure/cliextensions/ml",
"preview": true,
"version": "2.0.2"
}
]

Description of issue (in as much detail as possible)

Command:- az ml online-deployment create --name blue --endpoint advanced-ner-endpoint-demo -f deployment.yml --all-traffic
Exception details
Creating or updating online_deployments
Check: endpoint advanced-endpoint exists
The deployment request bhushan-workspace-advanced-ner-endpoint-demo-8612672 was accepted. ARM deployment URI for reference:
https://ms.portal.azure.com/#blade/HubsExtension/DeploymentDetailsBlade/overview/id/%2Fsubscriptions%2F2b044625-9119-453a-8f50-53426430883b%2
Registering model version (ad2f42d3-xxxx-xxxx-be9f-2e20ef135c16 1 ) Done (1s)
Registering code version (c665e43f-xxxx-xxxx-a98d-a84911248e91 1 ) Done (1s)
Registering environment version (32890045-xxxx-48d3-a43c-6f2ec0323d08 1 ) Done (6s)
Creating deployment blue ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Code: ResourceDeploymentFailure
Message: The resource operation completed with terminal provisioning state 'Failed'.
Exception Details: (DeploymentTimedOut) The Managed Inference service creation is taking longer than our normal time.

Activity

  1. ghost added
    needs-triageThis is a new issue that needs to be triaged to the appropriate team.
    on Nov 9, 2021
  2. ghost added
    questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that
    on Nov 9, 2021
  3. ghost added
    customer-reportedIssues that are reported by GitHub users external to the Azure organization.
    on Nov 9, 2021
  4. Yong Zhang (yonzhan)
    added
    extension/ml
    Machine Learning
    on Nov 9, 2021
  5. ghost removed
    needs-triageThis is a new issue that needs to be triaged to the appropriate team.
    on Nov 9, 2021
  6. Yong Zhang (yonzhan)
    added
    Service AttentionThis issue is responsible by Azure service team.
    and removed
    questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that
    on Nov 9, 2021

  8. yonzhan commented on Nov 9, 2021

    @yonzhan
    Yong Zhang (yonzhan)
    on Nov 9, 2021
    Collaborator

    route to service team

  9. diondrapeck commented on Nov 10, 2021

    @diondrapeck
    Diondra (diondrapeck)
    on Nov 10, 2021
    Contributor

    Acknowledged - beginning investigation and will reach out for more information if needed.

  10. vs-li commented on Nov 10, 2021

    @vs-li
    Vivian Li (vs-li)
    on Nov 10, 2021

    Hi Bhushan Gholave (@bhushangholave)! We found an error stating that the operation timed out waiting for an image build to complete. If you check the Azure portal, are you able to find an image build error under operation details?

  11. vizhur commented on Nov 10, 2021

    @vizhur
    vizhur
    on Nov 10, 2021

    AzureML doesn't have access to the image build log because of the privacy. Though it is available for the user from the storage associated with the workspace or from the container registry associated with user workspace (ACR task runs). Timeout issue from the image build usually means poor environment specification, and pip fails to resolve the conflicts in user dependencies with a reasonable time.

    Solution from the user end would be to inspect the dependencies and fix conflicts. Should be easily repro-ed locally by materializing environment from the conda spec.

    Temporary workaround that might work is to pin pip<=20.2.4 to use older resolver that ignores some of the conflicts

  12. bhushangholave commented on Nov 11, 2021

    @bhushangholave
    Bhushan Gholave (bhushangholave)
    on Nov 11, 2021
    Author

    Vivian Li (@vs-li) and vizhur thank you for your comments.
    Looks like some timeout issue to me but it reproducing every time.
    Error details in ARM deployments are

    {
    "status": "Failed",
    "error": {
        "code": "DeploymentTimedOut",
        "message": "The Managed Inference service creation is taking longer than our normal time.",
        "details": [],
        "additionalInfo": []
    }
}

    ARM Failure
    Details

  13. bhushangholave commented on Nov 11, 2021

    @bhushangholave
    Bhushan Gholave (bhushangholave)
    on Nov 11, 2021
    Author

    Vivian Li (@vs-li) In Azure portal Environment tab it is still stuck at installing pip dependencies.
    Same environment is working on our other workspace
    Attaching logs and dependencies file
    after 1HR30 m it timed out
    Installing pip dependencies: ...working...
    Run ID: cf4 timed out after 1h30m0s
    Environment.log
    .
    pipdependencies file.txt

  14. vizhur commented on Nov 12, 2021

    @vizhur
    vizhur
    on Nov 12, 2021

    Bhushan Gholave (@bhushangholave) you have dependencies conflict. Unfortunately conda swallows pip's log and you cannot see pips struggles. if you pip install -r requirements.txt with those deps with the latest pip you should see where pip gets stuck. There are a bunch of tuckets for pip for those kind of issues, feel free to file one too.

    just to unblock yourself you can try to pin pip<=20.2.4 to use old resolver that ignores some of the conflicts

  15. bhushangholave commented on Nov 12, 2021

    @bhushangholave
    Bhushan Gholave (bhushangholave)
    on Nov 12, 2021
    Author

    vizhur you are right, corrected my dependencies file and it worked. But portal or deployment creation needs to throw correct error and logs for debug.

  16. RakeshMohanMSFT commented on Apr 22, 2022

    @RakeshMohanMSFT
    RakeshMohanMSFT (RakeshMohanMSFT)
    on Apr 22, 2022
    Contributor

    Bhushan Gholave (@bhushangholave) Since the work around helped. Closing this thread for now.

  17. RakeshMohanMSFT (RakeshMohanMSFT)
    closed this as completedon Apr 22, 2022
  18. RakeshMohanMSFT (RakeshMohanMSFT)
    self-assigned this
    on Apr 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Machine LearningService AttentionThis issue is responsible by Azure service team.customer-reportedIssues that are reported by GitHub users external to the Azure organization.extension/ml

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions