Skip to content

bug: Cloutwatch PutMetricData fails with AWS Java SDK 2.40.10 #13538

New issue
New issue
Closed
#13541
Closed
bug: Cloutwatch PutMetricData fails with AWS Java SDK 2.40.10#13538
#13541
Assignees
bentsku
Labels
area: asfstatus: in progressCurrently being worked ontype: bugBug report
@kaklakariada

Description

@kaklakariada
kaklakariada
opened on Dec 17, 2025

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Cloudwatch PutMetricData request fails with the latest AWS Java SDK 2.40.10.

AWS client exception:

software.amazon.awssdk.services.cloudwatch.model.CloudWatchException: exception while calling cloudwatch with unknown operation: An unknown error occurred when trying to parse the request. (Service: CloudWatch, Status Code: 500, Request ID: 5741b6fc-15f1-4d18-a74c-55a7ec26286b) (SDK Attempt Count: 4)

	at software.amazon.awssdk.services.cloudwatch.model.CloudWatchException$BuilderImpl.build(CloudWatchException.java:113)
	at software.amazon.awssdk.services.cloudwatch.model.CloudWatchException$BuilderImpl.build(CloudWatchException.java:61)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
	at software.amazon.awssdk.services.cloudwatch.DefaultCloudWatchClient.putMetricData(DefaultCloudWatchClient.java:4048)
	at com.exasol.cloudwatch.CloudWatchPointWriter.putPoints(CloudWatchPointWriter.java:51)
	at com.exasol.cloudwatch.CloudWatchPointWriter.putPointsInChunks(CloudWatchPointWriter.java:42)
	at com.exasol.cloudwatch.CloudWatchAdapter.runSynchronization(CloudWatchAdapter.java:88)
	at com.exasol.cloudwatch.CloudWatchAdapter.handleRequest(CloudWatchAdapter.java:69)
	at com.exasol.cloudwatch.CloudWatchAdapterIT.runAdapter(CloudWatchAdapterIT.java:208)
	at com.exasol.cloudwatch.CloudWatchAdapterIT.runAdapter(CloudWatchAdapterIT.java:212)
	at com.exasol.cloudwatch.CloudWatchAdapterIT.testIncrementalReport(CloudWatchAdapterIT.java:167)
	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: exception while calling cloudwatch with unknown operation: An unknown error occurred when trying to parse the request. (Service: CloudWatch, Status Code: 500, Request ID: 9c48b196-42cd-4167-9d83-98188637037d)
	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: exception while calling cloudwatch with unknown operation: An unknown error occurred when trying to parse the request. (Service: CloudWatch, Status Code: 500, Request ID: 47ef6ae1-cc12-4870-b4a4-7163089476c8)
	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: exception while calling cloudwatch with unknown operation: An unknown error occurred when trying to parse the request. (Service: CloudWatch, Status Code: 500, Request ID: 0163b2d9-fd98-4f48-9aa4-3196ad57de85)

Localstack debug log (DEBUG=1):

localstack.aws.protocol.parser.UnknownParserError: An unknown error occurred when trying to parse the request.
2025-12-17T09:58:26.059  INFO --- [et.reactor-0] localstack.request.http    : POST /service/GraniteServiceVersion20100801/operation/PutMetricData => 500
2025-12-17T09:58:26.414 ERROR --- [et.reactor-1] l.aws.handlers.logging     : exception during call chain
Traceback (most recent call last):
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 187, in wrapper
    return func(*args, **kwargs)
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1313, in _do_parse
    self._parse_payload(request, shape, parsed, uri_params)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1363, in _parse_payload
    original_parsed = self._initial_body_parse(request)
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1388, in _initial_body_parse
    return self.parse_data_item(body_contents_stream)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1032, in parse_data_item
    return method(stream, additional_info)
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1107, in _parse_type_map
    self._parse_type_key_value_pair(stream, items)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1112, in _parse_type_key_value_pair
    value = self.parse_data_item(stream)
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1032, in parse_data_item
    return method(stream, additional_info)
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1089, in _parse_type_array
    return [self.parse_data_item(stream) for _ in range(length)]
            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1032, in parse_data_item
    return method(stream, additional_info)
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1107, in _parse_type_map
    self._parse_type_key_value_pair(stream, items)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1112, in _parse_type_key_value_pair
    value = self.parse_data_item(stream)
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1032, in parse_data_item
    return method(stream, additional_info)
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1122, in _parse_type_tag
    return self._parse_type_datetime(value)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1128, in _parse_type_datetime
    return self._convert_str_to_timestamp(str(value))
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 353, in _convert_str_to_timestamp
    final_value = converter(value)
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 362, in _timestamp_unixtimestamp
    return datetime.datetime.fromtimestamp(int(timestamp_string), tz=datetime.UTC)
                                           ~~~^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: '1765965323.442'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/rolo/gateway/chain.py", line 166, in handle
    handler(self, self.context, response)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/localstack-core/localstack/aws/handlers/service.py", line 68, in __call__
    return self.parse_and_enrich(context)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/opt/code/localstack/localstack-core/localstack/aws/handlers/service.py", line 72, in parse_and_enrich
    operation, instance = parser.parse(context.request)
                          ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 187, in wrapper
    return func(*args, **kwargs)
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 1299, in parse
    final_parsed = self._do_parse(request, shape, uri_params)
  File "/opt/code/localstack/localstack-core/localstack/aws/protocol/parser.py", line 191, in wrapper
    raise UnknownParserError(
        "An unknown error occurred when trying to parse the request."
    ) from e
localstack.aws.protocol.parser.UnknownParserError: An unknown error occurred when trying to parse the request.
2025-12-17T09:58:26.417  INFO --- [et.reactor-1] localstack.request.http    : POST /service/GraniteServiceVersion20100801/operation/PutMetricData => 500

Expected Behavior

The PutMetricData request suceeds.

How are you starting LocalStack?

Custom (please describe below)

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

Using Java testcontainer library 2.0.3, see full sources

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

Java client code (full source):

        final PutMetricDataRequest putRequest = PutMetricDataRequest.builder().namespace(CLOUDWATCH_NAMESPACE)
                .metricData(points.collect(Collectors.toList())).build();
        cloudwatch.putMetricData(putRequest);

Environment

- OS: macOS 15.7.3
- LocalStack:
  LocalStack version: 4.12.0
  LocalStack Docker image sha: localstack/localstack@sha256:0df3a97da57de03a588c05d9b8f390f15c7033fc7c4512f94619d344bc3cd317
  LocalStack build date: 2025-12-11T13:10:58.457586274Z
  LocalStack build git hash: LOCALSTACK_BUILD_GIT_HASH=b570b3ee2

Anything else?

Request succeeds with AWS Client version 2.40.5 but fails with 2.40.6.

AWS client wire log shows actual payload of a failed request:

2025-12-17 11:17:41.841 [FINE   ] http-outgoing-2 >> "POST /service/GraniteServiceVersion20100801/operation/PutMetricData HTTP/1.1[\r][\n]" 
2025-12-17 11:17:41.842 [FINE   ] http-outgoing-2 >> "Host: 127.0.0.1:32804[\r][\n]" 
2025-12-17 11:17:41.842 [FINE   ] http-outgoing-2 >> "Accept: application/cbor[\r][\n]" 
2025-12-17 11:17:41.842 [FINE   ] http-outgoing-2 >> "amz-sdk-invocation-id: 417c758e-70b1-db2d-1d3f-e332ecdd996b[\r][\n]" 
2025-12-17 11:17:41.842 [FINE   ] http-outgoing-2 >> "amz-sdk-request: attempt=4; max=4[\r][\n]" 
2025-12-17 11:17:41.842 [FINE   ] http-outgoing-2 >> "Authorization: AWS4-HMAC-SHA256 Credential=test/20251217/us-east-1/monitoring/aws4_request, SignedHeaders=accept;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;smithy-protocol;x-amz-content-sha256;x-amz-date;x-amzn-query-mode, Signature=77e038dfc4a22cc615622fec68a1854b19c94d6d41cd546f3e90b0cbeb4d9fb9[\r][\n]" 
2025-12-17 11:17:41.842 [FINE   ] http-outgoing-2 >> "Content-Type: application/cbor[\r][\n]" 
2025-12-17 11:17:41.842 [FINE   ] http-outgoing-2 >> "smithy-protocol: rpc-v2-cbor[\r][\n]" 
2025-12-17 11:17:41.842 [FINE   ] http-outgoing-2 >> "User-Agent: aws-sdk-java/2.40.7 md/io#sync md/http#Apache ua/2.1 api/CloudWatch#2.40.x os/Mac_OS_X#15.7.3 lang/java#21.0.1 md/OpenJDK_64-Bit_Server_VM#21.0.1+12-LTS md/vendor#Eclipse_Adoptium md/en_DE m/D,M,N,e[\r][\n]" 
2025-12-17 11:17:41.842 [FINE   ] http-outgoing-2 >> "x-amz-content-sha256: 92ff4547493e23842bfa1952ffea1559df0603109d1d7386c0a6154d5753ea79[\r][\n]" 
2025-12-17 11:17:41.843 [FINE   ] http-outgoing-2 >> "X-Amz-Date: 20251217T101741Z[\r][\n]" 
2025-12-17 11:17:41.843 [FINE   ] http-outgoing-2 >> "x-amzn-query-mode: true[\r][\n]" 
2025-12-17 11:17:41.843 [FINE   ] http-outgoing-2 >> "Content-Length: 177[\r][\n]" 
2025-12-17 11:17:41.843 [FINE   ] http-outgoing-2 >> "Connection: Keep-Alive[\r][\n]" 
2025-12-17 11:17:41.843 [FINE   ] http-outgoing-2 >> "[\r][\n]" 
2025-12-17 11:17:41.843 [FINE   ] http-outgoing-2 >> "[0xbf]iNamespacefExasoljMetricData[0x81][0xbf]jMetricNameeUSERSjDimensions[0x82][0xbf]dNamejDeploymenteValueuTEST-1765966658420-86[0xff][0xbf]dNamelCluster NameeValuedMAIN[0xff]iTimestamp[0xc1][0xfb]A[0xda]P[0xa0][0xa3][0xc7][\n]" 
2025-12-17 11:17:41.843 [FINE   ] http-outgoing-2 >> "=eValue[0x1]dUniteCount[0xff][0xff]" 
2025-12-17 11:17:41.859 [FINE   ] http-outgoing-2 << "HTTP/1.1 500 INTERNAL SERVER ERROR[\r][\n]" 
2025-12-17 11:17:41.860 [FINE   ] http-outgoing-2 << "Server: TwistedWeb/24.3.0[\r][\n]" 
2025-12-17 11:17:41.860 [FINE   ] http-outgoing-2 << "Date: Wed, 17 Dec 2025 10:17:42 GMT[\r][\n]" 
2025-12-17 11:17:41.860 [FINE   ] http-outgoing-2 << "Content-Type: application/cbor[\r][\n]" 
2025-12-17 11:17:41.860 [FINE   ] http-outgoing-2 << "Content-Length: 160[\r][\n]" 
2025-12-17 11:17:41.860 [FINE   ] http-outgoing-2 << "x-amzn-query-error: InternalError;Receiver[\r][\n]" 
2025-12-17 11:17:41.861 [FINE   ] http-outgoing-2 << "x-amzn-RequestId: e6b0cd1f-ba99-433f-85c0-1b8002351648[\r][\n]" 
2025-12-17 11:17:41.861 [FINE   ] http-outgoing-2 << "Smithy-Protocol: rpc-v2-cbor[\r][\n]" 
2025-12-17 11:17:41.861 [FINE   ] http-outgoing-2 << "x-localstack: true[\r][\n]" 
2025-12-17 11:17:41.861 [FINE   ] http-outgoing-2 << "[\r][\n]" 
2025-12-17 11:17:41.861 [FINE   ] http-outgoing-2 << "[0xbf]f__typevCommonServiceExceptiongmessagexvexception while calling cloudwatch with unknown operation: An unknown error occurred when trying to parse the request.[0xff]"

AWS client log of successful request:

2025-12-17 11:23:18.169 [FINE   ] http-outgoing-4 >> "POST / HTTP/1.1[\r][\n]" 
2025-12-17 11:23:18.170 [FINE   ] http-outgoing-4 >> "Host: 127.0.0.1:32804[\r][\n]" 
2025-12-17 11:23:18.170 [FINE   ] http-outgoing-4 >> "amz-sdk-invocation-id: afdfbd19-7112-c560-8806-ccaefc083cfd[\r][\n]" 
2025-12-17 11:23:18.170 [FINE   ] http-outgoing-4 >> "amz-sdk-request: attempt=1; max=4[\r][\n]" 
2025-12-17 11:23:18.170 [FINE   ] http-outgoing-4 >> "Authorization: AWS4-HMAC-SHA256 Credential=test/20251217/us-east-1/monitoring/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=a6d0da9ef456e5723b6b0cc1764796655a171ceed00765d2c16f25fb598b5d5b[\r][\n]" 
2025-12-17 11:23:18.170 [FINE   ] http-outgoing-4 >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]" 
2025-12-17 11:23:18.171 [FINE   ] http-outgoing-4 >> "User-Agent: aws-sdk-java/2.40.5 md/io#sync md/http#Apache ua/2.1 api/CloudWatch#2.40.x os/Mac_OS_X#15.7.3 lang/java#21.0.1 md/OpenJDK_64-Bit_Server_VM#21.0.1+12-LTS md/vendor#Eclipse_Adoptium md/en_DE m/D,N,e[\r][\n]" 
2025-12-17 11:23:18.171 [FINE   ] http-outgoing-4 >> "x-amz-content-sha256: dc590d0b889b114018de959aad968159b520be5ecb4d2755ed68cf96ed56ea7a[\r][\n]" 
2025-12-17 11:23:18.171 [FINE   ] http-outgoing-4 >> "X-Amz-Date: 20251217T102318Z[\r][\n]" 
2025-12-17 11:23:18.171 [FINE   ] http-outgoing-4 >> "Content-Length: 447[\r][\n]" 
2025-12-17 11:23:18.171 [FINE   ] http-outgoing-4 >> "Connection: Keep-Alive[\r][\n]" 
2025-12-17 11:23:18.172 [FINE   ] http-outgoing-4 >> "[\r][\n]" 
2025-12-17 11:23:18.172 [FINE   ] http-outgoing-4 >> "Action=PutMetricData&Version=2010-08-01&Namespace=Exasol&MetricData.member.1.MetricName=USERS&MetricData.member.1.Dimensions.member.1.Name=Deployment&MetricData.member.1.Dimensions.member.1.Value=TEST-1765966995506-771&MetricData.member.1.Dimensions.member.2.Name=Cluster+Name&MetricData.member.1.Dimensions.member.2.Value=MAIN&MetricData.member.1.Timestamp=2025-12-17T10%3A21%3A15.845Z&MetricData.member.1.Value=1.0&MetricData.member.1.Unit=Count" 
2025-12-17 11:23:18.188 [FINE   ] http-outgoing-4 << "HTTP/1.1 200 OK[\r][\n]" 
2025-12-17 11:23:18.188 [FINE   ] http-outgoing-4 << "Server: TwistedWeb/24.3.0[\r][\n]" 
2025-12-17 11:23:18.189 [FINE   ] http-outgoing-4 << "Date: Wed, 17 Dec 2025 10:23:18 GMT[\r][\n]" 
2025-12-17 11:23:18.189 [FINE   ] http-outgoing-4 << "Content-Type: text/xml[\r][\n]" 
2025-12-17 11:23:18.189 [FINE   ] http-outgoing-4 << "Content-Length: 238[\r][\n]" 
2025-12-17 11:23:18.189 [FINE   ] http-outgoing-4 << "x-amzn-RequestId: 02ab3375-a166-401d-a567-6210f164ef0f[\r][\n]" 
2025-12-17 11:23:18.189 [FINE   ] http-outgoing-4 << "x-localstack: true[\r][\n]" 
2025-12-17 11:23:18.189 [FINE   ] http-outgoing-4 << "[\r][\n]" 
2025-12-17 11:23:18.189 [FINE   ] http-outgoing-4 << "<?xml version='1.0' encoding='utf-8'?>[\n]" 
2025-12-17 11:23:18.190 [FINE   ] http-outgoing-4 << "<PutMetricDataResponse xmlns="http://monitoring.amazonaws.com/doc/2010-08-01/"><ResponseMetadata><RequestId>02ab3375-a166-401d-a567-6210f164ef0f</RequestId></ResponseMetadata></PutMetricDataResponse>"

Metadata

Metadata

Assignees

Labels

area: asfstatus: in progressCurrently being worked ontype: bugBug report

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions