CloudFormation: Store resource scaffolding in 'generated' directory. #13534

peter-smith-phd · 2025-12-16T21:07:15Z

Motivation

Generate CloudFormation resource scaffolding into a service's resource_providers/generated directory to separate auto-generated code from hand-written code. Resource provider classes should inherit from their auto-generated base class, permitting automatic updates of the base class and associated data types, without overwriting any hand-written code.

Changes

CloudFormation scaffolding script has been updated to:
- Introduce new generated/<resource>_base.py file that contains auto-generated type definitions, as well as abstract CRUD methods. These can be re-generated at any time, without overwriting hand-written code.
- Also generate the existing *_plugin.py and *.schema.json files into the generated subdirectory.
- Generate a default <resource>.py file that inherits types/methods from <resource>_base.py. This file is expected to be modified by hand and contain the full provider implementation.
Update the exist AWS::SQS::QueuePolicy provider to demonstrate that this new split works correctly (other resource providers for AWS::SQS will be updated in a later PR).

Tests

Tested by:

Migrating the existing AWS::SQS::QueuePolicy to use this new structure (included in this PR)
Testing with a Pro resource type, and visually inspecting the generated files.

Test Results - Preflight, Unit

23 046 tests ±0 21 201 ✅ ±0 6m 4s ⏱️ -14s
1 suites ±0 1 845 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 806a1d5. ± Comparison against base commit 6269d34.

♻️ This comment has been updated with latest results.

github-actions · 2025-12-16T21:32:43Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 2m 59s ⏱️ -1s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 806a1d5. ± Comparison against base commit 6269d34.

♻️ This comment has been updated with latest results.

github-actions · 2025-12-16T21:48:53Z

Test Results - Alternative Providers

584 tests - 887 325 ✅ - 564 18m 21s ⏱️ - 14m 54s
1 suites - 4 259 💤 - 323
1 files - 4 0 ❌ ± 0

Results for commit 806a1d5. ± Comparison against base commit 6269d34.

This pull request removes 887 tests.

tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_basic_operations_multiple_protocols[json]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_basic_operations_multiple_protocols[query]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_basic_operations_multiple_protocols[smithy-rpc-v2-cbor]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_exception_serializing_with_no_shape_in_spec[json]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_exception_serializing_with_no_shape_in_spec[query]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_exception_serializing_with_no_shape_in_spec[smithy-rpc-v2-cbor]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_multi_protocol_client_fixture[json]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_multi_protocol_client_fixture[query]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_multi_protocol_client_fixture[smithy-rpc-v2-cbor]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudwatch ‑ test_alarm_lambda_target
…

♻️ This comment has been updated with latest results.

github-actions · 2025-12-16T22:01:07Z

Test Results (amd64) - Integration, Bootstrap

5 files ± 0 5 suites ±0 2h 5m 25s ⏱️ - 28m 50s
4 134 tests - 1 428 3 829 ✅ - 1 173 305 💤 - 255 0 ❌ ±0
4 140 runs - 1 428 3 829 ✅ - 1 173 311 💤 - 255 0 ❌ ±0

Results for commit 806a1d5. ± Comparison against base commit 6269d34.

This pull request removes 1428 tests.

tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

♻️ This comment has been updated with latest results.

github-actions · 2025-12-16T22:10:17Z

LocalStack Community integration with Pro

2 files ± 0 2 suites ±0 1h 33m 5s ⏱️ - 20m 59s
4 110 tests - 1 045 3 801 ✅ - 957 309 💤 - 88 0 ❌ ±0
4 112 runs - 1 045 3 801 ✅ - 957 311 💤 - 88 0 ❌ ±0

Results for commit 806a1d5. ± Comparison against base commit 6269d34.

This pull request removes 1045 tests.

tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

♻️ This comment has been updated with latest results.

simonrw

I am hugely on board with this change, thank you! I have raised an issue about hiding the docstrings which I think will help developers build a conforming implementation (until we enforce an engine -> resource provider contract at least). I'd like to open a discussion about that.

I tried this out while implementing AWS::RDS::DBProxyEndpoint for a feature request. Thank you for making this change backwards compatible! It means that we can migrate the resource providers gradually! I really appreciate this.

I found that the files needed formatting. Can we maybe run make format-modified (or similar) after rendering the templates? That would make the initial experience much nicer.

simonrw · 2025-12-17T10:34:19Z

...lstack-core/localstack/services/cloudformation/scaffolding/templates/provider_template.py.j2

        self,
        request: ResourceRequest[{{ resource }}Properties],
    ) -> ProgressEvent[{{ resource }}Properties]:
-        """


issue: I don't like removing these docstrings to the base model. I realise that their content is autogenerated, and will get out of sync with the generated code but the docstrings are a great reminder of what

properties the model needs to set (writeOnlyProperties)

potential operations the method may need to invoke (through required iam actions)

I can see the docstring if I hover the method in PyCharm, but for people who have not written many resource providers it's not intuitive that this docstring is present. I bet the wouldn't look in the generated subdir anyway. To that end, relying on the schema.json file is also not a good solution.

I was hoping to start a discussion here rather than just implementing my suggestion. I don't 100% think this is a good approach because it introduces an element of the autogenerated values into the manually updated base class. I don't know if you have any better suggestions?

I'm fine with this suggestion, which is why I went ahead and implemented it. These files will only be auto-generated once, and then manually edited every other time. Given that service teams will be writing the resource providers, and they're not going to be experts at writing them, having extra clues will be helpful to get them started.

Honestly though, I think a lot of these templates instructions (and templated test cases) could be re-evaluated, given the behaviour of the new CloudFormation engine, but I was going to think about this in a later PR. For example, I'm not yet clear on the whole permissions model. Should these be evaluated/checked by the engine, rather than the providers, which means we don't really need them in the scaffolding comments?

For me they serve as documentation and guidance for the implementor first and foremost. I also agree with your last point in that I don't think the engine should check permissions. I don't think anything should check permissions, it should be apparent from the failed operation that an IAM action is missing (though perhaps we can be helpful with this information when presenting it to the user given we have access to the required permissions).

I think you are right, the best place for these comments is in the manually edited files. If they go stale it's not the end of the world as they are guidance anyway. Thanks for the discussion!

...k-core/localstack/services/cloudformation/scaffolding/templates/provider_base_template.py.j2

peter-smith-phd · 2025-12-17T21:05:24Z

Thanks @simonrw , I've made your suggested changes.

simonrw

Thanks for making these changes! This looks like a huge improvement to the way we manage resource providers and scaffold them in the future!

localstack-core/localstack/services/cloudformation/scaffolding/__main__.py

simonrw · 2026-01-05T12:45:04Z

...lstack-core/localstack/services/cloudformation/scaffolding/templates/provider_template.py.j2

        self,
        request: ResourceRequest[{{ resource }}Properties],
    ) -> ProgressEvent[{{ resource }}Properties]:
-        """


For me they serve as documentation and guidance for the implementor first and foremost. I also agree with your last point in that I don't think the engine should check permissions. I don't think anything should check permissions, it should be apparent from the failed operation that an IAM action is missing (though perhaps we can be helpful with this information when presenting it to the user given we have access to the required permissions).

I think you are right, the best place for these comments is in the manually edited files. If they go stale it's not the end of the world as they are guidance anyway. Thanks for the discussion!

…o separate auto-generated code from human-written code.

peter-smith-phd requested review from baermat, dominikschubert, gregfurman, pinzon, simonrw and thrau as code owners December 16, 2025 21:07

peter-smith-phd added semver: patch Non-breaking changes which can be included in patch releases docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes aws:cloudformation AWS CloudFormation labels Dec 16, 2025

peter-smith-phd force-pushed the sqs/make-cfn-gold branch from 0713e38 to d8931e2 Compare December 16, 2025 21:13

simonrw requested changes Dec 17, 2025

View reviewed changes

simonrw reviewed Dec 17, 2025

View reviewed changes

...k-core/localstack/services/cloudformation/scaffolding/templates/provider_base_template.py.j2 Show resolved Hide resolved

thrau removed their request for review December 18, 2025 14:08

simonrw self-requested a review December 22, 2025 11:34

simonrw approved these changes Jan 5, 2026

View reviewed changes

peter-smith-phd added 2 commits January 6, 2026 04:12

CloudFormation: Store resource scaffolding in 'generated' directory t…

7a28cfa

…o separate auto-generated code from human-written code.

Updates, as recommend in PR review.

806a1d5

peter-smith-phd force-pushed the sqs/make-cfn-gold branch from 570b146 to 806a1d5 Compare January 5, 2026 15:14

peter-smith-phd merged commit a9a6799 into main Jan 5, 2026
43 checks passed

peter-smith-phd deleted the sqs/make-cfn-gold branch January 5, 2026 16:14

gregfurman mentioned this pull request Jan 6, 2026

fix(pyproject): Make inclusion of additional cfn schemas more permissive #13586

Merged

Uh oh!

CloudFormation: Store resource scaffolding in 'generated' directory. #13534

CloudFormation: Store resource scaffolding in 'generated' directory. #13534

Uh oh!

Conversation

peter-smith-phd commented Dec 16, 2025

Motivation

Changes

Tests

Related

Uh oh!

github-actions bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results - Preflight, Unit

Uh oh!

github-actions bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results - Alternative Providers

Uh oh!

github-actions bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Integration, Bootstrap

Uh oh!

github-actions bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

simonrw left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simonrw Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

simonrw Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

peter-smith-phd Jan 1, 2026

Choose a reason for hiding this comment

Uh oh!

simonrw Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

peter-smith-phd commented Dec 17, 2025

Uh oh!

simonrw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

simonrw Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Dec 16, 2025 •

edited

Loading

github-actions bot commented Dec 16, 2025 •

edited

Loading

github-actions bot commented Dec 16, 2025 •

edited

Loading

github-actions bot commented Dec 16, 2025 •

edited

Loading

github-actions bot commented Dec 16, 2025 •

edited

Loading

simonrw left a comment •

edited

Loading