AWS Batch Executor

[syedahsn]

Overview

This PR introduces the AWS Batch Executor. This Executor can be configured to run Airflow tasks using AWS Batch. It is based on an initial contribution from @aelzeiny.

From the README:

This is an Airflow executor powered by Amazon Batch. Each task scheduled by Airflow is run inside a
separate container, scheduled by Batch. Some benefits of an executor like this include:

1. Scalability and Lower Costs: AWS Batch allows the ability to dynamically provision the resources needed to execute tasks.
   Depending on the resources allocated, AWS Batch can autoscale up or down based on the workload,
   ensuring efficient resource utilization and reducing costs.
2. Job Queues and Priority: AWS Batch provides the concept of job queues, allowing
   the ability to prioritize and manage the execution of tasks. This ensures that when multiple
   tasks are scheduled simultaneously, they are executed in the desired order of priority.
3. Flexibility: AWS Batch supports Fargate (ECS), EC2 and EKS compute environments. This range of
   compute environments, as well as the ability to finely define the resources allocated to
   the compute environments gives a lot of flexibility to users in choosing the most suitable
   execution environment for their workloads.
4. Rapid Task Execution: By maintaining an active worker within AWS Batch, tasks submitted to
   the service can be executed swiftly. With a ready-to-go worker, there's minimal startup delay,
   ensuring tasks commence immediately upon submission. This feature is particularly advantageous
   for time-sensitive workloads or applications requiring near-real-time processing,
   enhancing overall workflow efficiency and responsiveness.

This PR comes with most features now included in the ECS Executor, but we will continue to update both executors as we work to improve them.

Review Notes

Similar to the ECS Executor, this PR comes as a fully functional feature, which is great for testing, but it does mean that the PR is large. When reviewing, it isn't necessary to go through every single line. Instead, it would be more beneficial to have read through the documentation, and glance at how the Executor is implemented.

There are a lot of similarities in the code between the Batch Executor and the ECS Executor - this is by design. As we continue to refine the process of writing custom Executors, we are converging on an optimal framework to write efficient, reliable, and fault-tolerant custom Executors.

Testing

There is extensive unit testing which has near 100% line coverage in most cases:

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[Taragolis]

Minor things that I've found

[syedahsn]

@Lee-W @eladkal I'd like to hear your thoughts on this feature. If you have some time to take a look, I'd really appreciate it!

[o-nikolas]

I'm halfway through, but need to go for lunch, so I'm "committing" the first batch (no pun intended) of comments :)

Looking great so far!

[Taragolis]

Looking good to me, if it take in account that executor has an experimental status

[o-nikolas]

Some minor comments on old threads.

Can you add one last change to the docs to list this as an experimental executor (as we currently do for ECS here).

[syedahsn]

Sorry I missed those! Added them now, thanks!