Google BigQuery

Reading time: 0 minute(s) (0 words)

Google BigQuery is a serverless data warehouse that facilitates scalable analysis over petabytes of data. It is a Platform as a Service that supports querying using ANSI SQL. The BigQuery integration with Nobl9 empowers users to turn their big data into valuable business insights.

Google BigQuery parameters and supported features in Nobl9

General support:: Release channel: Stable, Beta; Connection method: Agent, Direct; Replay and SLI Analyzer: Not supported; Event logs: Supported; Query checker: Not supported; Query parameters retrieval: Supported; Timestamp cache persistence: Supported
Query parameters:: Query interval: 1 min; Query delay: 0; Jitter: 15 sec; Timeout: 30 sec
Agent details and minimum required versions for supported features:: Plugin name: n9bigquery; Query delay environment variable: BIGQUERY_QUERY_DELAY; Query parameters retrieval: 0.73.2; Timestamp cache persistence: 0.65.0

Authentication

Big Query authentication requires the user’s credentials to be entered in Nobl9. Users can retrieve their authentication credentials from the Google Cloud Platform as the Service account key file. For all necessary details on how to get the Service account key file, refer to the Getting Started with Authentication | BigQuery documentation.

For the direct connection, pass the contents of the downloaded Service account key file on the Nobl9 Web. This activates direct integration with the Big Query APIs to retrieve data, leveraging the SaaS-to-SaaS infrastructure in Nobl9.

Agent connection for the user to be granted with BigQuery permissions. The minimal set of permissions required for the BigQuery agent connection is:

bigquery.datasets.get
bigquery.jobs.create
bigquery.jobs.list
bigquery.models.getData
bigquery.models.getMetadata
bigquery.tables.getData

note

Nobl9 agent can use Workload Identity in GCP (Google Cloud Platform) in GKE (Google Kubernetes Engine). For more information, refer to the Deploying BigQuery agent section.

Adding Google BigQuery as a data source

To ensure data transmission between Nobl9 and Google BigQuery, it may be necessary to list Nobl9 IP addresses as trusted.

💻ip allowlist

IP addresses to include in your allowlist for secure access:

If you're using app.nobl9.com instance:

18.159.114.21
18.158.132.186
3.64.154.26

If you're using us1.nobl9.com instance:

34.121.54.120
34.123.193.191
34.134.71.10
35.192.105.150
35.225.248.37
35.226.78.175
104.198.44.161

You can add the BigQuery data source using the direct or agent connection methods.

Direct connection method

Direct connection to BigQuery requires users to enter their credentials which Nobl9 stores safely.

Nobl9 Web

To set up this type of connection:

Navigate to Integrations > Sources.
Click .
Click the required Source button.
Choose Direct.

Select one of the following Release Channels:
- The stable channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of a beta release. Use it to avoid crashes and other limitations.
- The beta channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.
Upload Service Account Key File to authenticate with Google Cloud.
The file needs to be in JSON format. Refer to the Authentication section for more details.

Select a Project.
Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses the default project.
Enter a Display Name.
You can enter a user-friendly name with spaces in this field.
Enter a Name.
The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example, my-project-1). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result.
Enter a Description.
Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
- The default value in BigQuery integration for Query delay is 0 seconds.
info
Changing the Query delay may affect your SLI data. For more details, check the Query delay documentation.
Click Add Data Source.

sloctl

Create a YAML definition to set up a direct connection with Google BigQuery. For this, refer to the following example:

YAML definition for the direct connection method
apiVersion: n9/v1alpha
kind: Direct
metadata:
  name: big-query
  displayName: BigQuery Direct
  project: default
spec:
  description: Example BigQuery Direct
  releaseChannel: stable
  bigQuery:
    serviceAccountKey: "{\n  \"type\": \"service_account\",\n  \"project_id\": \"prod-app\",\n  \"private_key_id\": \"669180ba44964eddba9e2f6623721381\",\n  \"private_key\": \"-----BEGIN PRIVATE KEY-----\\nSECRET_KEY_GOES_HERE\\n-----END PRIVATE KEY-----\\n\",\n  \"client_email\": \"nobl9@nobl9.iam.gserviceaccount.com\",\n  \"client_id\": \"eddba9e2f66237213812\",\n  \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n  \"token_uri\": \"https://oauth2.googleapis.com/token\",\n  \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n  \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/nobl9%40nobl9.iam.gserviceaccount.com\"\n}"
  queryDelay:
    value: 1
    unit: Second

Field	Type	Description
`queryDelay.unit` mandatory	enum	Specifies the unit for the query delay. Possible values: `Second` \| `Minute`. • Check query delay documentation for default unit of query delay for each source.
`queryDelay.value` mandatory	numeric	Specifies the value for the query delay. • Must be a number less than 1440 minutes (24 hours). • Check query delay documentation for default unit of query delay for each source.
`logCollectionEnabled` optional	boolean	Optional. Defaults to `false`. Set to `true` if you'd like your direct to collect event logs. Contact us to activate it.
`releaseChannel` mandatory	enum	Specifies the release channel. Accepted values: `beta` \| `stable`.
Source-specific fields
`bigQuery.serviceAccountKey` mandatory	string	You must embed the Service Account Key File content to authenticate with Google Cloud. See authentication for more details

Apply your YAML definition using the sloctl apply command.

Agent connection method

Nobl9 Web

Follow the instructions below to create your BigQuery agent connection.

Navigate to Integrations > Sources.
Click .
Click the required Source button.
Choose Agent.

Select one of the following Release Channels:
- The stable channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of a beta release. Use it to avoid crashes and other limitations.
- The beta channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.

Select a Project.
Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses the default project.
Enter a Display Name.
You can enter a user-friendly name with spaces in this field.
Enter a Name.
The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example, my-project-1). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result.
Enter a Description.
Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
- The default value in BigQuery integration for Query delay is 0 seconds.
info
Changing the Query delay may affect your SLI data. For more details, check the Query delay documentation.
Click Add Data Source.

Deploy your agent in a Kubernetes cluster or Docker container.

sloctl

Create a YAML definition to set up an agent connection with Google BigQuery. For this, refer to the following example:

YAML definition for the agent connection method
apiVersion: n9/v1alpha
kind: Agent
metadata:
  name: big-query
  displayName: BigQuery Agent
  project: default
spec:
  description: Example BigQuery Agent
  releaseChannel: stable
  bigQuery: {}
  queryDelay:
    value: 1
    unit: Second

Field	Type	Description
`queryDelay.unit` mandatory	enum	Specifies the unit for the query delay. Possible values: `Second` \| `Minute`. • Check query delay documentation for default unit of query delay for each source.
`queryDelay.value` mandatory	numeric	Specifies the value for the query delay. • Must be a number less than 1440 minutes (24 hours). • Check query delay documentation for default unit of query delay for each source.
`releaseChannel` mandatory	enum	Specifies the release channel. Accepted values: `beta` \| `stable`.

Apply your YAML definition using the sloctl apply command.
Deploy your agent in a Kubernetes cluster or Docker container.

Google BigQuery API rate limits

The following rate limits apply to the BigQuery API:

Query jobs. See Quotas and Limits reference.
Point density. Point density greater than 1000 data points per minute leads to errors. To address this, add point aggregation to your query.

Google BigQuery cost optimization

BigQuery pricing is based on bytes read by the query.

Since BigQuery queries must contain the where clause with date between filter, as required by the Nobl9 agent, your can use partitioning on the date_col column to reduce the number of bytes read. For example,

WHERE
{date_col} BETWEEN
DATETIME(@n9date_from)
AND DATETIME(@n9date_to)

Learn more about storage and query costs estimation and partitioned tables.

Useful links

For a more in-depth look, consult additional resources:

Getting StartedBigQuery docs

QueriesBigQuery docs

Creating and Managing ProjectsBigQuery docs

Quotas and LimitsBigQuery docs

Getting started with authenticationGoogle Cloud documentation

Agent metricsAgent

Creating SLOs via TerraformTerraform

Creating via TerraformTerraform

Authentication​

Adding Google BigQuery as a data source​

Direct connection method​

Nobl9 Web​

sloctl​

Agent connection method​

Nobl9 Web​

sloctl​

Google BigQuery API rate limits​

Useful links​

Authentication

Adding Google BigQuery as a data source

Direct connection method

Nobl9 Web

sloctl

Agent connection method

Nobl9 Web

sloctl

Google BigQuery API rate limits

Useful links