Repository navigation
Azure Data Factory Extension - Linked Service Creation - not reading JSON file #2872
Description
Functionality Feedback
When I am trying to run a command to create a linked service where the contents of the properties are stored in JSON file, it is not reading them. It is sending a request to the API with a request body which looks like this -
{"properties": {"type": "LinkedService"}}
Here is the debug log for the request:
azure.core.pipeline.policies._universal : Request URL: 'https://management.azure.com/subscriptions/(redacted-subscription)/resourceGroups/(redacted-rg)/providers/Microsoft.DataFactory/factories/(readacted-adf-name)/linkedservices/HyasDataPrep2?api-version=2018-06-01'
azure.core.pipeline.policies._universal : Request method: 'PUT'
azure.core.pipeline.policies._universal : Request headers:
azure.core.pipeline.policies._universal : 'Content-Type': 'application/json'
azure.core.pipeline.policies._universal : 'Accept': 'application/json'
azure.core.pipeline.policies._universal : 'Content-Length': '41'
azure.core.pipeline.policies._universal : 'x-ms-client-request-id': '28f918d4-5135-11eb-b3a4-0242ac110002'
azure.core.pipeline.policies._universal : 'CommandName': 'datafactory linked-service create'
azure.core.pipeline.policies._universal : 'ParameterSetName': '--factory-name --name --properties --resource-group --debug'
azure.core.pipeline.policies._universal : 'User-Agent': 'AZURECLI/2.14.0 (DEB) azsdk-python-datafactorymanagementclient/unknown Python/3.6.10 (Linux-4.19.128-microsoft-standard-x86_64-with-debian-10.6)'
azure.core.pipeline.policies._universal : 'Authorization': '*****'
azure.core.pipeline.policies._universal : Request body:
azure.core.pipeline.policies._universal : {"properties": {"type": "LinkedService"}}
The az-cli command is -
az datafactory linked-service create --factory-name "redacted-factory-name" --name "linked-service" --properties @./"linked-services/somelinkedservice.json" --resource-group "redacted-rg" --debug
Design Feedback
The JSON payload for the properties is very different what is expressed in ADF Authoring tool, when you look at the JSON declaration on the service.
For example:
- JSON Payload for a Storage account in Portal/Azure Data Factory Authoring tool
{
"name": "AzureDataLakeStorage1",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"annotations": [],
"type": "AzureBlobFS",
"typeProperties": {
"url": "https://astoreageaccountincloud.dfs.core.windows.net",
"encryptedCredential": "superdupersecretkey"
}
}
}- JSON Payload required by
az datafactory linked-service createcommand
{
"type": "AzureStorage",
"typeProperties": {
"connectionString" :{
"type": "SecureString",
"value":"DefaultEndpointsProtocol=https;AccountName=examplestorageaccount;AccountKey=<storage key>"
}
}
}
Unlike the az datafactory pipeline create command, which seems to accept the same JSON as it is in the pipeline code in the Authoring tool.
I think, for ease of developer experience and consistency, the JSON payload should be same as the one in the portal/authoring tool. . The one in az datafactory pipeline create is the better model. I am hoping the API is consistent as well.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.
- ID: 13386c48-1ac8-9452-fcb3-263829e9982c
- Version Independent ID: 1060fc99-cdb7-9c06-2374-f3127fa5e3f5
- Content: az datafactory linked-service
- Content Source: latest/docs-ref-autogen/ext/datafactory/datafactory/linked-service.yml
- GitHub Login: Robert Outlaw (@rloutlaw)
- Microsoft Alias: routlaw
Activity
yonzhan commentedon Jan 8, 2021
datafactory
yungezz commentedon Jan 8, 2021
thanks for feedback. Feng Zhou (@fengzhou-msft) could you pls have a look? thanks
fengzhou-msft commentedon Mar 30, 2021
Shashank Banerjea (@codepossible) for the functionality issue, which platform do you use and how do you install
azure-cli? I cannot reproduce it on windows and WSL.For the design feedback, I think you are comapring 2 differnet types:
AzureBlobFSandAzureStorage.fengzhou-msft commentedon Apr 2, 2021
For the functionality issue, I can reproduce it with a type that is not supported by the extension yet, for instance:
AzureDatabricksDeltaLakeand there should be a warning message with--debug:Then the request body becomes:
{"properties": {"type": "LinkedService"}}We will work on regenerating the extension to support new types.