Skip to content

Add check_interval and max_attempts as parameter of DynamoDBToS3Operator - #34972

Merged
ferruzzi merged 1 commit into
apache:mainfrom
aws-mwaa:vincbeck/dynamo
Oct 16, 2023
Merged

Add check_interval and max_attempts as parameter of DynamoDBToS3Operator#34972
ferruzzi merged 1 commit into
apache:mainfrom
aws-mwaa:vincbeck/dynamo

Conversation

@vincbeck

Copy link
Copy Markdown
Contributor

Add check_interval and max_attempts as parameter of DynamoDBToS3Operator to override default values if needed.

I modified the system test example_dynamodb_to_s3.py as well to increase max_attempts because sometimes the waiter times out while the export is still in progress. That's the reason for the last failure in this system test.



Traceback (most recent call last):
--
51 | File "/opt/airflow/airflow/models/taskinstance.py", line 432, in _execute_task
52 | result = execute_callable(context=context, **execute_callable_kwargs)
53 | File "/opt/airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py", line 143, in execute
54 | self._export_table_to_point_in_time()
55 | File "/opt/airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py", line 167, in _export_table_to_point_in_time
56 | waiter.wait(ExportArn=export_arn)
57 | File "/usr/local/lib/python3.8/site-packages/botocore/waiter.py", line 55, in wait
58 | Waiter.wait(self, **kwargs)
59 | File "/usr/local/lib/python3.8/site-packages/botocore/waiter.py", line 388, in wait
60 | raise WaiterError(
61 | botocore.exceptions.WaiterError: Waiter export_table failed: Max attempts exceeded. Previously accepted state: For expression "ExportDescription.ExportStatus" we matched expected path: "IN_PROGRESS"

cc @ferruzzi @o-nikolas


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@ferruzzi
ferruzzi merged commit b119646 into apache:main Oct 16, 2023
@ferruzzi
ferruzzi deleted the vincbeck/dynamo branch October 16, 2023 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants