Skip to content

fixes for cloud fetch - part un#356

Merged
andrefurlan-db merged 2 commits intodatabricks:2.9.4from
andrefurlan-db:ES-1011531-v2
Feb 16, 2024
Merged

fixes for cloud fetch - part un#356
andrefurlan-db merged 2 commits intodatabricks:2.9.4from
andrefurlan-db:ES-1011531-v2

Conversation

@andrefurlan-db
Copy link
Contributor

@andrefurlan-db andrefurlan-db commented Feb 16, 2024

This is a first set of fixes for cloud fetch.

  • Throw when failed to download file
  • Retry properly while downloading file
  • Add a bunch of debug logs
  • Added URL redacting
  • Prevent thread issues

TODO:

  • check rows context to avoid duplicates
  • http connection pools for cloud storage, proxies, etc.

Backported to version 2

backport to version 2

Signed-off-by: Andre Furlan <andre.furlan@databricks.com>
@andrefurlan-db andrefurlan-db marked this pull request as ready for review February 16, 2024 02:23
@andrefurlan-db andrefurlan-db requested review from benc-db and jackyhu-db and removed request for arikfr February 16, 2024 02:25
@andrefurlan-db andrefurlan-db changed the title fixes for cloud fetch fixes for cloud fetch - part un Feb 16, 2024
Signed-off-by: Andre Furlan <andre.furlan@databricks.com>
@andrefurlan-db andrefurlan-db merged commit 6a348ec into databricks:2.9.4 Feb 16, 2024
andrefurlan-db added a commit to andrefurlan-db/databricks-sql-python that referenced this pull request Feb 21, 2024
* fixes for cloud fetch

Signed-off-by: Andre Furlan <andre.furlan@databricks.com>
---------

Signed-off-by: Andre Furlan <andre.furlan@databricks.com>
Co-authored-by: Raymond Cypher <raymond.cypher@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Comments