Implement AWS S3 authentication with an explicitly provided IAM role; implement OAuth for GCS. by alexey-milovidov · Pull Request #84011 · ClickHouse/ClickHouse

alexey-milovidov · 2025-07-18T19:48:52Z

Changelog category (leave one):

Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Implement AWS S3 authentication with an explicitly provided IAM role. Implement OAuth for GCS. These features were recently only available in ClickHouse Cloud and are now open-sourced. Synchronize some interfaces such as serialization of the connection parameters for object storages.

clickhouse-gh · 2025-07-18T19:49:17Z

Workflow [PR], commit [6cbba69]

dcheckoway · 2025-07-24T02:18:43Z

src/IO/S3/Credentials.cpp

+{
+    if (!sts_endpoint_override.empty())
+    {
+        endpoint.SetURL(sts_endpoint_override);


NOTE: This ends up doing a POST to this URL without the Action parameter. You're only appending ?Action=AssumeRole&Version=2011-06-15 below on the non-overridden endpoint.

I think it's important that those query params get used regardless of override vs. not. The params aren't technically part of the "endpoint" (which should just be the base URL imho).

…isks

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250729) * Fix build due to ClickHouse/ClickHouse#76802 * Fix build due to ClickHouse/ClickHouse#81837 * Fix build due to ClickHouse/ClickHouse#84011 * Fix gtest due to ClickHouse/ClickHouse#83599 --------- Co-authored-by: kyligence-git <gluten@kyligence.io> Co-authored-by: Chang chen <chenchang@apache.com>

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

CheSema · 2025-08-01T15:23:05Z

src/Disks/ObjectStorages/DiskObjectStorageTransaction.cpp

-                    tx->object_storage.removeObjectsIfExist(tx->metadata_storage.getStorageObjects(path));
+                if (!tx->object_storage.isPlain())
+                {
+                    auto maybe_blobs = tx->metadata_transaction->tryGetBlobsFromTransactionIfExists(path);


This is quite important place.
Could you please explain this changes.

I just picked up the change from the private repository.

I do not see this in private.
Check this
https://github.com/ClickHouse/clickhouse-private/commit/ab882425ac04fa96f814a7db2a21ffb970f6c10d#r163325404

Let me explain a bit, I hope it will help you

When this code is run, metadata_transaction is empty. metadata_transaction is build inside commit after disk operations are executed.
maybe_blobs is always nullopt here.

I was not right here.
This only confirms how difficult the place is.

Use case:

open disk tx write A (blob_1) commit disk tx open disk tx write A (blob_2) write A (blob_3) commit disk tc

Before your changes: blob_2 is left orphan
After your changes: blob_1 is left orphan

Both are not correct.

this code should not be changes in such big PR.

this code should not be change without proper review and discussion.

changes in this code should be tested

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

Fix rebase issue: - 20250806 ClickHouse#84854 - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

Implement AWS S3 authentication with an explicitly provided IAM role; implement OAuth for GCS.

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da) (cherry picked from commit 91a244c)

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da) (cherry picked from commit 91a244c) (cherry picked from commit bf02481)

Synchronize Disks, IO, Backups

ab88242

clickhouse-gh bot added the pr-improvement Pull request with some product improvements label Jul 18, 2025

alexey-milovidov added 3 commits July 18, 2025 22:08

Fix build

1b15d75

Merge branch 'master' into sync-disks

7a481f9

Fix build

9c79f87

pamarcos mentioned this pull request Jul 22, 2025

Add extra_credentials to AzureBlobStorage to authenticate with client_id and tenant_id #84235

Merged

1 task

dcheckoway reviewed Jul 24, 2025

View reviewed changes

alexey-milovidov added 10 commits July 26, 2025 02:41

Merge branch 'master' into sync-disks

82421b1

Merge branch 'master' into sync-disks

3668ddd

Merge branch 'master' of github.com:ClickHouse/ClickHouse into sync-d…

1551d57

…isks

Merge with private

0416de7

Sync with private

1423598

Sync with private

f5d0261

Merge branch 'master' into sync-disks

c9ae652

Fix whitespace

970c3eb

Merge branch 'master' into sync-disks

b8e310b

Sync with private

6cbba69

alexey-milovidov self-assigned this Jul 28, 2025

alexey-milovidov merged commit 51f0b57 into master Jul 28, 2025
120 of 125 checks passed

alexey-milovidov deleted the sync-disks branch July 28, 2025 08:05

robot-clickhouse-ci-1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jul 28, 2025

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Jul 29, 2025

support assume role (bug fix included) (#449)

3bfc945

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

baibaichen pushed a commit to Kyligence/gluten that referenced this pull request Jul 29, 2025

Fix build due to ClickHouse/ClickHouse#84011

28ff184

baibaichen pushed a commit to Kyligence/gluten that referenced this pull request Jul 29, 2025

Fix build due to ClickHouse/ClickHouse#84011

3470a65

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Jul 29, 2025

support assume role (bug fix included) (#449)

0c6d991

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Jul 30, 2025

support assume role (bug fix included) (#449)

181d59f

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Jul 31, 2025

support assume role (bug fix included) (#449)

8c469a8

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

CheSema reviewed Aug 1, 2025

View reviewed changes

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 1, 2025

support assume role (bug fix included) (#449)

a2e74b7

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 2, 2025

support assume role (bug fix included) (#449)

07dcdd9

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 3, 2025

support assume role (bug fix included) (#449)

da8292a

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 7, 2025

support assume role (bug fix included) (#449)

6465fb4

Fix rebase issue: - 20250806 ClickHouse#84854 - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 8, 2025

support assume role (bug fix included) (#449)

21c6722

Fix rebase issue: - 20250806 ClickHouse#84854 - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 11, 2025

support assume role (bug fix included) (#449)

6223c33

Fix rebase issue: - 20250806 ClickHouse#84854 - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 12, 2025

support assume role (bug fix included) (#449)

70543ce

Fix rebase issue: - 20250806 ClickHouse#84854 - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 13, 2025

support assume role (bug fix included) (#449)

636b107

Fix rebase issue: - 20250806 ClickHouse#84854 - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 14, 2025

support assume role (bug fix included) (#449)

60ea0fc

Fix rebase issue: - 20250806 ClickHouse#84854 - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 15, 2025

support assume role (bug fix included) (#449)

2747304

Fix rebase issue: - 20250806 ClickHouse#84854 - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 16, 2025

support assume role (bug fix included) (#449)

ec611d3

Fix rebase issue: - 20250806 ClickHouse#84854 - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 18, 2025

support assume role (bug fix included) (#449)

386e563

Fix rebase issue: - 20250806 ClickHouse#84854 - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Aug 29, 2025

Merge pull request ClickHouse#84011 from ClickHouse/sync-disks

8fd0d42

Implement AWS S3 authentication with an explicitly provided IAM role; implement OAuth for GCS.

zvonand mentioned this pull request Aug 29, 2025

Antalya 25.6.5 backport of 84011: Implement AWS S3 authentication with an explicitly provided IAM role; implement OAuth for GCS. Altinity/ClickHouse#986

Merged

13 tasks

zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Aug 31, 2025

Merge pull request ClickHouse#84011 from ClickHouse/sync-disks

c09470d

Implement AWS S3 authentication with an explicitly provided IAM role; implement OAuth for GCS.

zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Sep 2, 2025

Merge pull request ClickHouse#84011 from ClickHouse/sync-disks

31d09b7

Implement AWS S3 authentication with an explicitly provided IAM role; implement OAuth for GCS.

lgbo-ustc pushed a commit to bigo-sg/ClickHouse that referenced this pull request Sep 17, 2025

support assume role (bug fix included) (ClickHouse#449)

91a244c

Fix rebase issue: - 20250729 ClickHouse#84011 - 20250318 ClickHouse#77257 (cherry picked from commit e7061da)

svb-alt mentioned this pull request Feb 2, 2026

Project Antalya Roadmap 2025 - Real-Time Data Lakes Altinity/ClickHouse#804

Closed

31 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement AWS S3 authentication with an explicitly provided IAM role; implement OAuth for GCS.#84011

Implement AWS S3 authentication with an explicitly provided IAM role; implement OAuth for GCS.#84011
alexey-milovidov merged 14 commits intomasterfrom
sync-disks

alexey-milovidov commented Jul 18, 2025

Uh oh!

clickhouse-gh bot commented Jul 18, 2025 •

edited

Loading

Uh oh!

dcheckoway Jul 24, 2025

Uh oh!

Uh oh!

CheSema Aug 1, 2025

Uh oh!

alexey-milovidov Aug 1, 2025

Uh oh!

CheSema Aug 2, 2025

Uh oh!

CheSema Aug 4, 2025

Uh oh!

CheSema Aug 5, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

alexey-milovidov commented Jul 18, 2025

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Uh oh!

clickhouse-gh bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dcheckoway Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CheSema Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

CheSema Aug 2, 2025

Choose a reason for hiding this comment

Uh oh!

CheSema Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

CheSema Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

clickhouse-gh bot commented Jul 18, 2025 •

edited

Loading

CheSema Aug 5, 2025 •

edited

Loading