Apache Airflow version

2.6.0

What happened

When marking an EMR Serverless job as failed, the job continues to run.

What you think should happen instead

The job should be cancelled. Looking at the EMR Serverless Operator, I don't see an on_kill method, so assuming we just need to add that.

I'm not sure how to handle the EmrServerlessCreateApplicationOperator operator, though - if the workflow has a corresponding EmrServerlessDeleteApplicationOperator, we'd probably want to delete the application if the job is cancelled.

How to reproduce

• Start an EMR Serverless DAG
• Mark the job as failed in the Airflow UI
• See that the EMR Serverless job continues to run

Operating System

n/a

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.0.0

Deployment

Amazon (AWS) MWAA

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct