The tap-bigquery Singer tap pulls data from the Google BigQuery data warehouse that can then be sent to a destination using a Singer target.

Alternative variants

Multiple variants of tap-bigquery are available. This document describes the default anelendata variant, which is recommended for new users.

Alternative variants are:

Standalone usage

Install the package using pip:

pip install tap-bigquery

For additional instructions, refer to the README in the repository.

Usage with Meltano

Install Meltano, create your Meltano project, and add the tap to your project as an extractor:

meltano add extractor tap-bigquery

For additional instructions, refer to the Meltano-specific documentation for tap-bigquery.

Settings

streams

Array holding objects describing streams (tables) to extract, with name, table, columns, datetime_key, and filters keys. See docs for details.

credentials_path

This is not a real setting, so it should not appear in the config.json JSON schema. The taret will read the GOOGLE_APPLICATION_CREDENTIALS environment variable which should be a fully qualified path to client_secrets.json for your service account.

start_datetime

Determines how much historical data will be extracted. Please be aware that the larger the time period and amount of data, the longer the initial extraction can be expected to take.

end_datetime

Date up to when historical data will be extracted.

limit

Limits the number of records returned in each stream, applied as a limit in the query.

start_always_inclusive

When replicating incrementally, disable to only select records whose datetime_key is greater than the maximum value replicated in the last run, by excluding records whose timestamps match exactly. This could cause records to be missed that were created after the last run finished, but during the same second and with the same timestamp.

Looking for help?

If you're having trouble getting tap-bigquery to work by itself or with Meltano, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page?

This page is generated from a YAML file that you can contribute changes to! It is also validated against a JSON Schema used for taps and targets.