Skip to content

RFC: Lambda Powertools for Python v2 #1459

New issue
New issue
Closed
#1625
Closed
RFC: Lambda Powertools for Python v2#1459
#1625
Assignees
rubenfonseca
Labels
RFCv2

Description

@rubenfonseca
rubenfonseca
opened on Aug 18, 2022

Is this related to an existing feature request or issue?

No response

Which AWS Lambda Powertools utility does this relate to?

Powertools itself

Summary

The Python 3.6 Lambda Runtime was deprecated as of Aug 17th 2022. This follows Python 3.6 End-Of-Life (EOL) reached on December 23, 2021. This means it's not possible to create or update Lambda functions using that runtime.

We should take this opportunity to release a new major version of Powertools, and decide what breaking changes to include.

Use case

Customers need to upgrade their Lambda Functions to one of the supported Python runtimes (at the moment of writing, Python 3.7-3.9).

We could use this major upgrade to introduce some breaking changes on Lambda Powertools that will simplify the code and/or improve support for existing features.

Proposal

We strive to make minimal breaking changes due to timing constraints with Python 3.6 deprecation. In ideal scenarios, we would provide ample time, nightly builds, a Beta for a major version, including a linter to help detect and upgrade from breaking changes.

This means, V2 will focus primarily on the following themes:

  • Drop Python 3.6. Follows alignment with Lambda runtime deprecation policy. It also allows us to upgrade development dependencies and documentation niceties we couldn't before due to prolonged Python 3.6 EOL life in Lambda.
  • Make all dependencies optional to optimize package size. AWS SDK makes up for over 90% of our package size. We also heard from customers that they want to use additional observability providers, thus wanting X-Ray SDK to be optional.
    • Lambda Layer will include all optional dependencies excluding those already available at Lambda runtime to further optimize cold start. This also means dropping SAR Extras as it becomes redundant.
  • Remove deprecated features. Specifically sqs_batch_processor and PartialSQSProcessor in favour of BatchProcessor launched 11 months ago that improved accuracy, security, and speed due to the new Lambda integration.
  • Improve correctness. Event Handler (API Gateway) doesn't support multi-header and cookies by default as it requires a breaking change. Similarly, Idempotency and Tracer doesn't use fully qualified names, impacting ABC or Classes that use the exact same method name - changing it would impact billions of transactions in production.

Quick summary

Item Issue/PR Status Code change required
Write What's new for v2
Update upgrade guide #1623 ✅
Remove POWERTOOLS_EVENT_HANDLER_DEBUG env var #1620 ✅
Drop support for python 3.6 #1339 ✅
Remove the old batch processing legacy implementation (sqs_batch_processor, PartialSQSProcessor) #1462 ✅ Yes
Event Handler REST - multi-value Headers by default, and cookies support #1455 ✅ tests only
Use fully qualified names for idempotency #1330 ✅
Use fully qualified names for tracer subsegments #1454 ✅
Update AppConfig API in Parameters/Feature flags due to GetConfiguration deprecation #1506 ✅
Deprecate SAR Extras #1543 ✅
Make all runtime dependencies optional #1164 ✅
Update docs on required dependencies (validation, parser) #1573 ✅
Event Handler REST v1 supports trailing slash route by default #1609 ✅
Replace AttributeValue in DynamoDBStreamEvent with deserialized Python values #1619 ✅ Yes
Replace email-validator dependency with str in Parser SES Model #1608 ✅

Launch plan

  • un-comment "deploy-prod" from publish_v2_layer.yml
  • update docs to include SAR v2 and Layer v2
  • review Upgrade Guide
  • review What's New
  • remove V2 banner from docs
  • remove V2 admonition from docs

Out of scope

We should make the breaking release as small as possible. For this we should not include in v2:

Potential challenges

We need to decide what to do with the Lambda Powertools Layer. One option would be to release a new "V2" layer.

UPDATE: We're launching a separate V2 layer and ARM support along with it!

Dependencies and Integrations

No response

Alternative solutions

No response

Acknowledgment

Activity

  1. rubenfonseca
    added
    RFC
    triagePending triage from maintainers
    on Aug 18, 2022

  2. boring-cyborg commented on Aug 18, 2022

    @boring-cyborg
    boring-cyborgBot
    on Aug 18, 2022

    Thanks for opening your first issue here! We'll come back to you as soon as we can.

  3. heitorlessa commented on Aug 18, 2022

    @heitorlessa
    heitorlessa
    on Aug 18, 2022
    Contributor

    Thank you @rubenfonseca! Let's make Event Handler Headers part of v2 as this will impact our ability to support cookies and multi-value headers without a maintenance hit -- #1455

    We should also add that we're going to make minimal breaking changes due to not having ample time to discuss and run experiments with customers - we shall leave those to 3.0 (modularization, potentially making pydantic the default, etc.)

  4. rubenfonseca commented on Aug 18, 2022

    @rubenfonseca
    rubenfonseca
    on Aug 18, 2022
    ContributorAuthor

    @heitorlessa thanks, updated! Do you know if an existing issue about removing the first batch processing implementation?

  5. heitorlessa commented on Aug 18, 2022

    @heitorlessa
    heitorlessa
    on Aug 18, 2022
    Contributor

    Nope, haven't created yet. We'd need to make a PR to add a deprecation warning too.

    https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/aws_lambda_powertools/utilities/batch/sqs.py#L24

  6. heitorlessa
    changed the title [-]RFC: Lambda Powertools Python v2[/-] [+]RFC: Lambda Powertools for Python v2[/+] on Aug 19, 2022
  7. heitorlessa
    removed
    triagePending triage from maintainers
    on Aug 19, 2022
  8. heitorlessa
    self-assigned this
    on Aug 19, 2022
  9. github-actions
    added
    pending-releaseFix or implementation already in dev waiting to be released
    on Aug 19, 2022
  10. heitorlessa
    removed
    pending-releaseFix or implementation already in dev waiting to be released
    on Aug 22, 2022
  11. heitorlessa
    pinned this issue on Aug 25, 2022

  12. rubenfonseca commented on Aug 25, 2022

    @rubenfonseca
    rubenfonseca
    on Aug 25, 2022
    ContributorAuthor

    We started creating an "Upgrade guide" with detailed upgrade steps for each breaking change, together with before/after examples. This will help everyone go faster through each major version changes.

  13. heitorlessa commented on Aug 26, 2022

    @heitorlessa
    heitorlessa
    on Aug 26, 2022
    Contributor

    I'd like to propose we make all dependencies optional until we modularise in V3.

    Although we managed to reduce >90% of AWS SDK, keeping AWS X-Ray SDK as a main dependency brings the full AWS SDK - this won't benefit customers using pip install.

    This would be a boost for customers not using X-Ray, those with sensitive startup time, and reduce build/deployment times.

    Compressed

     11M	lambda_powertools_v1.zip
464K	lambda_powertools_v2.zip

    Uncompressed

     75M	/tmp/lambda_powertools_v1
1.8M	/tmp/lambda_powertools_v2

  14. heitorlessa commented on Aug 29, 2022

    @heitorlessa
    heitorlessa
    on Aug 29, 2022
    Contributor

    Early days but we started documenting the breaking changes in v2: https://awslabs.github.io/aws-lambda-powertools-python/v2/upgrade/

    @rubenfonseca one thing we missed is adding another bullet point on multiValueHeaders for REST v1 payload

  15. 7 remaining items

  16. peterschutt commented on Oct 19, 2022

    @peterschutt
    peterschutt
    on Oct 19, 2022
    Contributor

    @mew1033 @peterschutt after V2 is launched, let's bring typing_extensions as a runtime dependency to start working on strict typing for the entire code base. We've made significant efforts in reducing package size (>95%), and bringing typing_extensions outweigh the effort of various workarounds to make typing work for multiple Python versions.

    Sounds great! I'll touch base with you after the v2 launch, congrats!

  17. bryantbiggs commented on Oct 19, 2022

    @bryantbiggs
    bryantbiggs
    on Oct 19, 2022

    ya'll are awesome - great work @heitorlessa / @rubenfonseca 🎉

  18. github-actions
    added
    pending-releaseFix or implementation already in dev waiting to be released
    on Oct 19, 2022
  19. heitorlessa
    removed
    pending-releaseFix or implementation already in dev waiting to be released
    on Oct 20, 2022
  20. github-actions
    added
    pending-releaseFix or implementation already in dev waiting to be released
    on Oct 20, 2022
  21. rubenfonseca
    linked a pull request that will close this issuechore: merge v2 branch #1625on Oct 20, 2022
  22. rubenfonseca
    closed this as completedin #1625on Oct 20, 2022
  23. heitorlessa
    removed
    pending-releaseFix or implementation already in dev waiting to be released
    on Oct 20, 2022
  24. heitorlessa
    reopened this on Oct 20, 2022
  25. aws-powertools
    deleted a comment from github-actions on Oct 20, 2022

  26. heitorlessa commented on Oct 20, 2022

    @heitorlessa
    heitorlessa
    on Oct 20, 2022
    Contributor

    All Ready! We're gonna publish a pre-release to test all workflows, and then we start writing the official Release Notes for 2.0.

    Finally! 🎉

  27. rubenfonseca commented on Oct 24, 2022

    @rubenfonseca
    rubenfonseca
    on Oct 24, 2022
    ContributorAuthor

    We just released AWS Lambda Powertools Python v2! 🎉🎉🎉🎉

    Don't forget to check our Upgrade Guide too :)

    Thank you everyone for your help getting this our of the door! 🙏 Onwards and upwards! 🎉

  28. rubenfonseca
    closed this as completedon Oct 24, 2022

  29. github-actions commented on Oct 24, 2022

    @github-actions
    github-actionsBot
    on Oct 24, 2022
    Contributor

    ⚠️COMMENT VISIBILITY WARNING⚠️

    This issue is now closed. Please be mindful that future comments are hard for our team to see.

    If you need more assistance, please either tag a team member or open a new issue that references this one.

    If you wish to keep having a conversation with other community members under this issue feel free to do so.

  30. heitorlessa
    unpinned this issue on Oct 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

RFCv2

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions