DynamoDBToS3Operator - Add a feature to export the table to a point in time. by utkarsharma2 · Pull Request #31142 · apache/airflow

utkarsharma2 · 2023-05-09T05:53:58Z

DynamoDBToS3Operator - Add a feature to export the table to a point in time.
Fixed - Now fetching the TableARN from client.describe_table(TableName=self.dynamodb_table_name) and added file_size in the example.

closes: #28830

Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>

utkarsharma2 · 2023-05-09T05:57:35Z

airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py

+         state at this point in time.
+        """
+        client = self.hook.conn.meta.client
+        table_description = client.describe_table(TableName=self.dynamodb_table_name)


Fixed TableARN issue

utkarsharma2 · 2023-05-09T05:57:51Z

tests/system/providers/amazon/aws/example_dynamodb_to_s3.py

+    backup_db_to_point_in_time = DynamoDBToS3Operator(
+        task_id="backup_db_to_point_in_time",
+        dynamodb_table_name=table_name,
+        file_size=1000,


Added required param.

utkarsharma2 · 2023-05-09T06:21:32Z

Have tested this locally with the below dag and it's working for me

from datetime import datetime

from airflow import DAG
from airflow.providers.amazon.aws.transfers.dynamodb_to_s3 import DynamoDBToS3Operator

with DAG(
    dag_id='example_export_dynamodb',
    schedule_interval=None,
    start_date=datetime(2021, 1, 1),
    tags=['example'],
    catchup=False,
) as dag:
    dynamodb_to_s3_operator = DynamoDBToS3Operator(
        task_id="dynamodb_to_s3",
        dynamodb_table_name="test",
        s3_bucket_name="tmp9",
        file_size=4000,
        export_time=datetime.now(),
        aws_conn_id="aws_default",
        s3_key_prefix="test1"
    )

ferruzzi

Thanks for not only taking on the fixes, but getting the done so quick. LGTM (again 🤦)

ferruzzi · 2023-05-09T14:16:24Z

airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py

+        client = self.hook.conn.meta.client
+        table_description = client.describe_table(TableName=self.dynamodb_table_name)
+        response = client.export_table_to_point_in_time(
+            TableArn=table_description.get("Table", {}).get("TableArn"),


I saw your question in the other PR before I saw you already submitted this one. Glad you came tot his solution anyway :) As I mentioned over there, another possible solution would have been to use cached_property to get and store the ARN, but I don't think there is much difference in this case in the end. 👍

+1 on this one :) Thanks for your reactivity!

Thanks for a quick review. I'll be more mindful going forward with respect to system tests. 👍

o-nikolas

Thanks for the quick turnaround!

ferruzzi · 2023-05-10T16:27:54Z

@utkarsharma2 System tests still failing with the mesage



Traceback (most recent call last):
--
File "/opt/airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py", line 142, in execute
self._export_table_to_point_in_time()
File "/opt/airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py", line 158, in _export_table_to_point_in_time
ExportFormat=self.export_format,
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 960, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the ExportTableToPointInTime operation: Invalid Request: 1 validation error detected: Value '' at 's3Prefix' failed to satisfy constraint: Member must have length greater than or equal to 3

ferruzzi · 2023-05-10T16:36:17Z

I'm at Open source Summit so I can't look too closely right now, but I wonder if the prefix has to start with s3: or s3/ or something like that?

ferruzzi · 2023-05-10T20:02:45Z

Or if a blank prefix is viable, maybe try using airflow.helpers.prune_dict() to drop the None value

utkarsharma2 · 2023-05-11T03:47:03Z

@ferruzzi I was able to reproduce the issue on local and there was a missing key s3_key_prefix which I have added and the format for export_time was not correct both of the issues have been fixed in PR - #31197 and I have tested and verified below code on my local.

backup_db_to_point_in_time = DynamoDBToS3Operator(
        task_id="backup_db_to_point_in_time",
        dynamodb_table_name=table_name,
        file_size=1000,
        s3_bucket_name=bucket_name,
        export_time=datetime.now(),
        s3_key_prefix=f"{S3_KEY_PREFIX}-3-",
    )

utkarsharma2 and others added 27 commits April 25, 2023 18:18

Add feature to export table to a point in time

19f7ef9

Fix static check

3f81d30

Moved the hook to a cached_property

4015ff6

Remove client_token param

7c09852

Add boto3 waiter

ccf6bae

Add test for waiters

ea08c54

Add tests for operator

897da47

Add document for DynamoDBToS3Operator point in time export

5635d28

Add param to docstring

079e908

Remove

a435b2c

Add check for future dates

8d40409

Change waiter API

60226a3

Update custom waiter config

9aab299

Add client to get_waiter

f3a99aa

Merge branch 'main' into DynamoDBToS3Operator

0066ead

Add foot note for custom waiters

d8248eb

Update airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py

224b8f3

Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>

Merge branch 'main' into DynamoDBToS3Operator

52ce310

Merge branch 'main' into DynamoDBToS3Operator

ab194a1

Remove unwanted constant

677032b

Remove comma to have a one liner code

0d5b3f8

Remove unwanted code

aadd23f

Fix testcase

ed620c1

Merge branch 'main' into DynamoDBToS3Operator

fccfe9c

Fix unit test cases

ee863e8

Merge branch 'main' into DynamoDBToS3Operator

f4b9df9

Get tableARN from client.describe_table

5ddaf58

utkarsharma2 requested review from eladkal and o-nikolas as code owners May 9, 2023 05:53

boring-cyborg bot added the area:providers label May 9, 2023

boring-cyborg bot added area:system-tests kind:documentation provider:amazon AWS/Amazon - related issues labels May 9, 2023

utkarsharma2 commented May 9, 2023

View reviewed changes

utkarsharma2 mentioned this pull request May 9, 2023

DynamoDBToS3Operator - Add feature to export table to a point in time #30501

Merged

utkarsharma2 changed the title ~~Dynamo db to s3 operator~~ DynamoDBToS3Operator - Add a feature to export the table to a point in time. May 9, 2023

Merge branch 'main' into DynamoDBToS3Operator

6f6296e

ferruzzi approved these changes May 9, 2023

View reviewed changes

vincbeck approved these changes May 9, 2023

View reviewed changes

o-nikolas approved these changes May 9, 2023

View reviewed changes

o-nikolas merged commit cd3fa33 into apache:main May 9, 2023

eladkal mentioned this pull request May 16, 2023

Status of testing Providers that were prepared on May 19, 2023 #31322

Closed

80 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

DynamoDBToS3Operator - Add a feature to export the table to a point in time.#31142

DynamoDBToS3Operator - Add a feature to export the table to a point in time.#31142
o-nikolas merged 28 commits intoapache:mainfrom
utkarsharma2:DynamoDBToS3Operator

utkarsharma2 commented May 9, 2023 •

edited

Loading

Uh oh!

utkarsharma2 May 9, 2023

Uh oh!

utkarsharma2 May 9, 2023

Uh oh!

utkarsharma2 commented May 9, 2023

Uh oh!

ferruzzi left a comment

Uh oh!

ferruzzi May 9, 2023

Uh oh!

vincbeck May 9, 2023

Uh oh!

utkarsharma2 May 10, 2023

Uh oh!

o-nikolas left a comment

Uh oh!

ferruzzi commented May 10, 2023

Uh oh!

ferruzzi commented May 10, 2023

Uh oh!

ferruzzi commented May 10, 2023

Uh oh!

utkarsharma2 commented May 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

Conversation

utkarsharma2 commented May 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

utkarsharma2 May 9, 2023

Choose a reason for hiding this comment

Uh oh!

utkarsharma2 May 9, 2023

Choose a reason for hiding this comment

Uh oh!

utkarsharma2 commented May 9, 2023

Uh oh!

ferruzzi left a comment

Choose a reason for hiding this comment

Uh oh!

ferruzzi May 9, 2023

Choose a reason for hiding this comment

Uh oh!

vincbeck May 9, 2023

Choose a reason for hiding this comment

Uh oh!

utkarsharma2 May 10, 2023

Choose a reason for hiding this comment

Uh oh!

o-nikolas left a comment

Choose a reason for hiding this comment

Uh oh!

ferruzzi commented May 10, 2023

Uh oh!

ferruzzi commented May 10, 2023

Uh oh!

ferruzzi commented May 10, 2023

Uh oh!

utkarsharma2 commented May 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

utkarsharma2 commented May 9, 2023 •

edited

Loading