Skip to content

docs(samples): Update BigQuery Storage Arrow samples batching logic #14961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Linchin merged 20 commits into from
Jan 8, 2026
Merged

docs(samples): Update BigQuery Storage Arrow samples batching logic #14961

Linchin merged 20 commits into from
Jan 8, 2026

Conversation

@AlexZMLyu
Copy link
Contributor

@AlexZMLyu AlexZMLyu commented Dec 11, 2025

Bases batching on size rather than row count to avoid exceeding an internal 10MB limit. Also removes an obsolete assertion in the test.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@AlexZMLyu
Fix: Update BigQuery Storage Arrow samples batching logic
a975186 
Bases batching on size rather than row count to avoid exceeding an
internal 10MB limit. Also removes an obsolete assertion in the test.
@AlexZMLyu AlexZMLyu requested a review from a team as a code owner December 11, 2025 05:43
@gemini-code-assist
Copy link
Contributor

gemini-code-assist bot commented Dec 11, 2025

Summary of Changes

Hello @AlexZMLyu, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the data batching mechanism within the BigQuery Storage Arrow sample. The primary goal is to prevent AppendRowsRequest payloads from exceeding BigQuery's internal 10MB size limit by implementing a size-aware batching strategy. This change improves the robustness and reliability of data ingestion when using PyArrow with BigQuery Storage.

Highlights

  • Batching Logic Update: The generate_write_requests function in the BigQuery Storage Arrow sample has been updated to base batching on the serialized size of PyArrow record batches rather than row count. This ensures that individual AppendRowsRequest payloads do not exceed the 10MB internal limit, using a conservative 7MB soft limit.
  • Improved Error Handling: A new check has been added to raise a ValueError if a single PyArrow batch containing just one row is larger than the maximum allowed request size (7MB), preventing potential issues with oversized individual records.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist bot reviewed Dec 11, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the batching logic in the BigQuery Storage Arrow sample to be based on byte size instead of row count. This is a great improvement as it prevents requests from exceeding the 10MB size limit. The new implementation is more robust. I've suggested a small refactoring to reduce code duplication, which will improve maintainability.

@AlexZMLyu
Fix: Update verify_result in pyarrow sample
2239db8 
- Changes query to SELECT DISTINCT int64_col to count unique rows.
- Asserts the count is exactly TABLE_LENGTH, removing the allowance for extra rows from potential retries.
@AlexZMLyu
Fix: Update PyArrow serialization in append_rows_with_arrow.py
b510cec
@parthea parthea changed the title Fix: Update BigQuery Storage Arrow samples batching logic docs(samples): Update BigQuery Storage Arrow samples batching logic Dec 17, 2025
@parthea parthea requested a review from GaoleMeng December 17, 2025 16:16
@AlexZMLyu
feat: Measure request generation time in pyarrow sample
f581b33
@AlexZMLyu
Fix: Improve PyArrow batching and serialization in BigQuery Storage s…
610f38e 
…ample

- Updates batching logic to use serialized size to avoid exceeding API limits.
- Ensures all rows in the PyArrow table are serialized for the request.
- Includes enhancements for measuring serialized row sizes.
@AlexZMLyu
Refactor: Use generator for request creation and block on send
501e993 
- Changed `generate_write_requests` to be a generator, yielding requests
  instead of returning a list.
- Made `stream.send()` calls blocking by calling `future.result()` immediately,
  ensuring requests are sent sequentially.
@AlexZMLyu
samples: reformat append_rows_with_arrow.py
ec49ec5
@AlexZMLyu
Copy link
Contributor Author

AlexZMLyu commented Jan 2, 2026

The test failures are in other PRs as well. https://github.com/googleapis/google-cloud-python/actions/runs/20482816437/job/58859390604

It is unrelated to my changes.

@AlexZMLyu
Copy link
Contributor Author

AlexZMLyu commented Jan 6, 2026

The new batching logic looks good.

Test Results:

  • 1 batch: 0.4230 seconds
  • 10 batches: 0.4489 seconds
  • 100 batches: 0.4369 seconds
  • 1,000 batches: 0.5522 seconds
  • 10,000 batches: 1.1524 seconds
  • 100,000 batches: 7.7520 seconds
  • 1,000,000 batches: 76.1911 seconds

Summary: 7 passed in 117.18s.

batch number increased 1000 times -> batching time increased only 100 times

GaoleMeng
GaoleMeng previously approved these changes Jan 6, 2026
View reviewed changes
GaoleMeng
GaoleMeng previously approved these changes Jan 7, 2026
View reviewed changes
@GaoleMeng GaoleMeng removed the request for review from agrawal-siddharth January 7, 2026 03:00
@AlexZMLyu AlexZMLyu closed this Jan 8, 2026
@AlexZMLyu AlexZMLyu reopened this Jan 8, 2026
@Linchin Linchin merged commit 45d4cee into googleapis:main Jan 8, 2026
45 checks passed
@cloud-sdk-librarian-robot cloud-sdk-librarian-robot mentioned this pull request Feb 12, 2026
Merged
codyoss pushed a commit that referenced this pull request Feb 12, 2026
@cloud-sdk-librarian-robot
chore: librarian release pull request: 20260212T081040Z (#15564)
0e4866e 
PR created by the Librarian CLI to initialize a release. Merging this PR
will auto trigger a release.

Librarian Version: v0.0.0-20260210205451-43b900a12d48
Language Image:
us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:f5426423676c75008c2135037e7b98f78cbb99f78b3c46fe043b6897be92d836
<details><summary>google-cloud-artifact-registry: v1.20.0</summary>

##
[v1.20.0](google-cloud-artifact-registry-v1.19.0...google-cloud-artifact-registry-v1.20.0)
(2026-02-12)

### Features

* add DIRSUM_SHA256 hash type (PiperOrigin-RevId: 865552557)
([5371e8e](5371e8e9))

</details>


<details><summary>google-cloud-bigquery-storage: v2.36.1</summary>

##
[v2.36.1](google-cloud-bigquery-storage-v2.36.0...google-cloud-bigquery-storage-v2.36.1)
(2026-02-12)

### Bug Fixes

* fix mypy (#15531)
([f2536ae](f2536ae3))

### Documentation

* Update BigQuery Storage Arrow samples batching logic (#14961)
([45d4cee](45d4cee9))

</details>


<details><summary>google-cloud-confidentialcomputing: v0.8.0</summary>

##
[v0.8.0](google-cloud-confidentialcomputing-v0.7.0...google-cloud-confidentialcomputing-v0.8.0)
(2026-02-12)

### Features

* add Nvidia Attestation proto message with its relevant fields and
ConfidentialGke options (PiperOrigin-RevId: 866189208)
([5371e8e](5371e8e9))

</details>


<details><summary>google-cloud-databasecenter: v0.5.0</summary>

##
[v0.5.0](google-cloud-databasecenter-v0.4.0...google-cloud-databasecenter-v0.5.0)
(2026-02-12)

### Features

* Adding Method QueryIssues of Database Center API v1beta
(PiperOrigin-RevId: 868397264)
([5371e8e](5371e8e9))

### Documentation

* A comment for field `order_by` in message
`.google.cloud.databasecenter.v1beta.AggregateFleetRequest` is changed
(PiperOrigin-RevId: 868397264)
([5371e8e](5371e8e9))

* A comment for field `order_by` in message
`.google.cloud.databasecenter.v1beta.QueryDatabaseResourceGroupsRequest`
is changed (PiperOrigin-RevId: 868397264)
([5371e8e](5371e8e9))

* A comment for message `AggregateFleetResponse` is changed
(PiperOrigin-RevId: 868397264)
([5371e8e](5371e8e9))

* A comment for field `filter` in message
`.google.cloud.databasecenter.v1beta.AggregateFleetRequest` is changed
(PiperOrigin-RevId: 868397264)
([5371e8e](5371e8e9))

* A comment for field `filter` in message
`.google.cloud.databasecenter.v1beta.AggregateIssueStatsRequest` is
changed (PiperOrigin-RevId: 868397264)
([5371e8e](5371e8e9))

* A comment for field `filter` in message
`.google.cloud.databasecenter.v1beta.QueryDatabaseResourceGroupsRequest`
is changed (PiperOrigin-RevId: 868397264)
([5371e8e](5371e8e9))

* A comment for field `maintenance_version` in message
`.google.cloud.databasecenter.v1beta.MaintenanceInfo` is changed
(PiperOrigin-RevId: 868397264)
([5371e8e](5371e8e9))

</details>


<details><summary>google-cloud-dataform: v0.9.0</summary>

##
[v0.9.0](google-cloud-dataform-v0.8.0...google-cloud-dataform-v0.9.0)
(2026-02-12)

### Features

* Update GCP Client Libraries in v1beta1 to support Folders,
TeamFolders, and other relevant APIs The v1beta1 API now includes
support for Folders and TeamFolders, allowing users to organize
repositories and files hierarchically and manage access controls. New
Features: - Added TeamFolder resource and methods: CreateTeamFolder,
GetTeamFolder, UpdateTeamFolder, DeleteTeamFolder
QueryTeamFolderContents to list folder contents. SearchTeamFolders to
search for TeamFolders. - Added Folder resource and methods:
CreateFolder, GetFolder, UpdateFolder, DeleteFolder QueryFolderContents
to list folder contents. Added MoveFolder to move Folders between
TeamFolders, other Folders, or the user root folder. - Added
MoveRepository to move Repositories between TeamFolders, Folders, or the
user root folder. - Added QueryUserRootContents to list contents of a
user&amp;#39;s root folder. Repository resource now includes
containing_folder and team_folder_name fields to indicate its location
within the folder hierarchy. - IAM methods (GetIamPolicy, SetIamPolicy,
TestIamPermissions) now support Folder and TeamFolder resources for
access control management
([5371e8e](5371e8e9))

### Documentation

* A comment for field `force` in message
`.google.cloud.dataform.v1beta1.DeleteRepositoryRequest` is changed
PiperOrigin-RevId: 868182714
([5371e8e](5371e8e9))

</details>


<details><summary>google-cloud-dialogflow: v2.46.0</summary>

##
[v2.46.0](google-cloud-dialogflow-v2.45.0...google-cloud-dialogflow-v2.46.0)
(2026-02-12)

### Features

* added a new field
StreamingAnalyzeContentRequest.output_multiple_utterances
(PiperOrigin-RevId: 862959083)
([a6b40b3](a6b40b31))

</details>


<details><summary>google-cloud-discoveryengine: v0.17.0</summary>

##
[v0.17.0](google-cloud-discoveryengine-v0.16.0...google-cloud-discoveryengine-v0.17.0)
(2026-02-12)

### Features

* add CrowdingSpec to SearchRequest to set crowding settings
(PiperOrigin-RevId: 868345232)
([b483e92](b483e921))

* add Natural Language Query Understanding fields and messages to
DiscoveryEngine request, response, and datastore messages
(PiperOrigin-RevId: 868345232)
([b483e92](b483e921))

* add output-only field SemanticState to SearchResponse.
(PiperOrigin-RevId: 868345232)
([b483e92](b483e921))

### Documentation

* other misc documentation updates Clients can specify Natural Language
Query Understanding-related fields, as well as CrowdingSpec. These
features are available in the V1 APIs. (PiperOrigin-RevId: 868345232)
([b483e92](b483e921))

* update documentation for search and data store NLQ features
(PiperOrigin-RevId: 868345232)
([b483e92](b483e921))

</details>


<details><summary>google-cloud-documentai: v3.10.0</summary>

##
[v3.10.0](google-cloud-documentai-v3.9.0...google-cloud-documentai-v3.10.0)
(2026-02-12)

### Features

* A new field `document_prompt` is added to message
`.google.cloud.documentai.v1beta3.DocumentSchema` (PiperOrigin-RevId:
866382481)
([5371e8e](5371e8e9))

* A new field `document_type` is added to message
`.google.cloud.documentai.v1beta3.ImportDocumentsRequest`
(PiperOrigin-RevId: 866382481)
([5371e8e](5371e8e9))

* A new field `revisions` is added to message
`.google.cloud.documentai.v1beta3.Evaluation` (PiperOrigin-RevId:
866382481)
([5371e8e](5371e8e9))

* The method `ReviewDocument` in
`.google.cloud.documentai.v1beta3.DocumentProcessorService` is
deprecated (PiperOrigin-RevId: 866382481)
([5371e8e](5371e8e9))

* A new field `enable_table_split` is added to message
`.google.cloud.documentai.v1beta3.OcrConfig.LayoutParsingParams`
(PiperOrigin-RevId: 866382481)
([5371e8e](5371e8e9))

* Added new messages `Documents` and `RawDocuments` for inline document
input (PiperOrigin-RevId: 866382481)
([5371e8e](5371e8e9))

* The field `skip_human_review` in messages
`.google.cloud.documentai.v1beta3.ProcessRequest` and
`.google.cloud.documentai.v1beta3.BatchProcessRequest` is deprecated
(PiperOrigin-RevId: 866382481)
([5371e8e](5371e8e9))

### Bug Fixes

* Removed the `SpannerIndexingConfig` message and the
`spanner_indexing_config` field from
`.google.cloud.documentai.v1beta3.Dataset` BREAKING CHANGE: The
`SpannerIndexingConfig` message and the `spanner_indexing_config` field
within the `Dataset` message have been removed. Client code referencing
these will need to stop referencing these in case of an error
(PiperOrigin-RevId: 866382481)
([5371e8e](5371e8e9))

### Documentation

* Updated comments for various fields and messages (PiperOrigin-RevId:
866382481)
([5371e8e](5371e8e9))

</details>


<details><summary>google-cloud-gke-hub: v1.21.0</summary>

##
[v1.21.0](google-cloud-gke-hub-v1.20.0...google-cloud-gke-hub-v1.21.0)
(2026-02-12)

### Features

* Add GKE Hub teams scope and RBAC GAPIC (PiperOrigin-RevId: 865111649)
([99a1284](99a1284f))

### Documentation

* A comment for field `cancel_requested` in message
`.google.cloud.gkehub.v1.OperationMetadata` is changed
(PiperOrigin-RevId: 865111649)
([99a1284](99a1284f))

* A comment for field `state` in message
`.google.cloud.gkehub.v1.CommonFeatureState` is changed
(PiperOrigin-RevId: 865111649)
([99a1284](99a1284f))

* A comment for enum value `ENABLING` in enum `State` is changed
(PiperOrigin-RevId: 865111649)
([99a1284](99a1284f))

* A comment for message `CommonFeatureState` is changed
(PiperOrigin-RevId: 865111649)
([99a1284](99a1284f))

* A comment for field `state` in message
`.google.cloud.gkehub.v1.Feature` is changed (PiperOrigin-RevId:
865111649)
([99a1284](99a1284f))

* A comment for field `membership_states` in message
`.google.cloud.gkehub.v1.Feature` is changed (PiperOrigin-RevId:
865111649)
([99a1284](99a1284f))

* A comment for enum value `DISABLING` in enum `State` is changed
(PiperOrigin-RevId: 865111649)
([99a1284](99a1284f))

* A comment for field `request_id` in message
`.google.cloud.gkehub.v1.UpdateFeatureRequest` is changed
(PiperOrigin-RevId: 865111649)
([99a1284](99a1284f))

* A comment for field `spec` in message
`.google.cloud.gkehub.v1.Feature` is changed (PiperOrigin-RevId:
865111649)
([99a1284](99a1284f))

* A comment for message `Feature` is changed (PiperOrigin-RevId:
865111649)
([99a1284](99a1284f))

* A comment for field `request_id` in message
`.google.cloud.gkehub.v1.CreateFeatureRequest` is changed
(PiperOrigin-RevId: 865111649)
([99a1284](99a1284f))

* A comment for enum value `ACTIVE` in enum `State` is changed
(PiperOrigin-RevId: 865111649)
([99a1284](99a1284f))

* A comment for field `membership_specs` in message
`.google.cloud.gkehub.v1.Feature` is changed (PiperOrigin-RevId:
865111649)
([99a1284](99a1284f))

* A comment for message `FeatureResourceState` is changed
(PiperOrigin-RevId: 865111649)
([99a1284](99a1284f))

* A comment for field `labels` in message
`.google.cloud.gkehub.v1.Feature` is changed (PiperOrigin-RevId:
865111649)
([99a1284](99a1284f))

</details>


<details><summary>google-cloud-storage-control: v1.10.0</summary>

##
[v1.10.0](google-cloud-storage-control-v1.9.0...google-cloud-storage-control-v1.10.0)
(2026-02-12)

### Features

* add a DeleteFolderRecursive API definition (PiperOrigin-RevId:
866471251)
([5371e8e](5371e8e9))

</details>


<details><summary>google-cloud-visionai: v0.4.0</summary>

##
[v0.4.0](google-cloud-visionai-v0.3.0...google-cloud-visionai-v0.4.0)
(2026-02-12)

### Bug Fixes

* An existing default host `visionai.googleapis.com` is changed to
`warehouse-visionai.googleapis.com` in service `Warehouse`
(PiperOrigin-RevId: 866154811)
([5371e8e](5371e8e9))

### Documentation

* A comment for field `relevance` in message
`.google.cloud.visionai.v1.SearchResultItem` is changed
(PiperOrigin-RevId: 866154811)
([5371e8e](5371e8e9))

* A comment for field `page_size` in message
`.google.cloud.visionai.v1.SearchIndexEndpointRequest` is changed
(PiperOrigin-RevId: 866154811)
([5371e8e](5371e8e9))

</details>


<details><summary>google-maps-solar: v0.4.0</summary>

##
[v0.4.0](google-maps-solar-v0.3.0...google-maps-solar-v0.4.0)
(2026-02-12)

### Documentation

* Add satellite imagery to `rgb_url` description (PiperOrigin-RevId:
866492024)
([5371e8e](5371e8e9))

* Add pre-GA details for Experiments enum (PiperOrigin-RevId: 866492024)
([5371e8e](5371e8e9))

</details>


<details><summary>Bulk Changes</summary>

* chore: librarian update image pull request: 20260205T214646Z (#15532)
([3654fe7](3654fe76))
Libraries:
google-cloud-artifact-registry,google-cloud-bigquery-storage,google-cloud-confidentialcomputing,google-cloud-databasecenter,google-cloud-dataform,google-cloud-dialogflow,google-cloud-discoveryengine,google-cloud-documentai,google-cloud-gke-hub,google-cloud-storage-control,google-cloud-visionai,google-maps-solar
* chore: sort repo metadata by key (#15529)
([7da7768](7da7768c))
Libraries:
google-cloud-artifact-registry,google-cloud-bigquery-storage,google-cloud-confidentialcomputing,google-cloud-databasecenter,google-cloud-dataform,google-cloud-dialogflow,google-cloud-discoveryengine,google-cloud-documentai,google-cloud-gke-hub,google-cloud-storage-control,google-cloud-visionai,google-maps-solar
* chore: librarian update image pull request: 20260123T001116Z (#15481)
([871cdee](871cdeeb))
Libraries:
google-cloud-artifact-registry,google-cloud-bigquery-storage,google-cloud-confidentialcomputing,google-cloud-databasecenter,google-cloud-dataform,google-cloud-dialogflow,google-cloud-discoveryengine,google-cloud-gke-hub,google-cloud-storage-control,google-cloud-visionai,google-maps-solar
</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@GaoleMeng GaoleMeng GaoleMeng approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@GaoleMeng GaoleMeng

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@AlexZMLyu @GaoleMeng @Linchin