Google BigQuery
Nobl9's integration with Google BigQuery enables SLO measurement using data stored in BigQuery tables. This integration allows organizations to define service level objectives based on metrics queried directly from BigQuery datasets and get valuable business insights.
Google BigQuery parameters and supported features in Nobl9
- General support:
- Release channel: Stable, Beta
- Connection method: Agent, Direct
- Replay and SLI Analyzer: Not supported
- Event logs: Supported
- Query checker: Not supported
- Query parameters retrieval: Supported
- Timestamp cache persistence: Supported
- Query parameters:
- Query interval: 1 min
- Query delay: 0 sec
- Jitter: 15 sec
- Timeout: 30 sec
- Agent details and minimum required versions for supported features:
- Plugin name: n9bigquery
- Query delay environment variable: BIGQUERY_QUERY_DELAY
- Query parameters retrieval: 0.73.2
- Timestamp cache persistence: 0.65.0
Authentication
Nobl9 supports two authentication methods for BigQuery integration:
- For the direct connection method:
- Obtain a
Service account key
from Google Cloud Platform. - Provide the key when adding Google BigQuery with the direct connection method in Nobl9.
- Nobl9 establishes direct connection with BigQuery APIs.
- Obtain a
For detailed instructions on creating service account keys, refer to the Getting Started with Authentication.
- For the agent connection method, make sure your agent is granted with the following IAM permissions:
bigquery.datasets.get
bigquery.jobs.create
bigquery.jobs.list
bigquery.models.getData
bigquery.models.getMetadata
bigquery.tables.getData
For deployments in Google Kubernetes Engine (GKE), the Nobl9 agent supports Workload Identity authentication.
Read more about Nobl9 agent deployment.
When using Workload Identity in GCP, ensure your GKE cluster has Workload Identity enabled and properly configured.
Adding Google BigQuery as a data source
To ensure data transmission between Nobl9 and Google BigQuery, it may be necessary to list Nobl9 IP addresses as trusted.
app.nobl9.com
instance:- 18.159.114.21
- 18.158.132.186
- 3.64.154.26
us1.nobl9.com
instance:- 34.121.54.120
- 34.123.193.191
- 34.134.71.10
- 35.192.105.150
- 35.225.248.37
- 35.226.78.175
- 104.198.44.161
You can add the BigQuery data source using the direct or agent connection methods.
Direct connection method
Direct connection to BigQuery requires users to enter their credentials which Nobl9 stores safely.
Nobl9 Web
To set up this type of connection:
- Navigate to Integrations > Sources.
- Click
.
- Click the required Source button.
- Choose Direct.
-
Select one of the following Release Channels:
- The
stable
channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of abeta
release. Use it to avoid crashes and other limitations. - The
beta
channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.
- The
-
Upload Service Account Key File to authenticate with Google Cloud.
The file needs to be in JSON format. Refer to the Authentication section for more details.
- Select a Project.
Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses thedefault
project. - Enter a Display Name.
You can enter a user-friendly name with spaces in this field. - Enter a Name.
The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example,my-project-1
). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result. - Enter a Description.
Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it. - Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
- The default value in BigQuery integration for Query delay is
0 seconds
.
infoChanging the Query delay may affect your SLI data. For more details, check the Query delay documentation. - The default value in BigQuery integration for Query delay is
- Click Add Data Source
sloctl
- Create a YAML definition to set up a direct connection with Google BigQuery. For this, refer to the following example:
apiVersion: n9/v1alpha
kind: Direct
metadata:
name: big-query
displayName: BigQuery Direct
project: default
spec:
description: Example BigQuery Direct
releaseChannel: stable
bigQuery:
serviceAccountKey: "{\n \"type\": \"service_account\",\n \"project_id\": \"prod-app\",\n \"private_key_id\": \"669180ba44964eddba9e2f6623721381\",\n \"private_key\": \"-----BEGIN PRIVATE KEY-----\\nSECRET_KEY_GOES_HERE\\n-----END PRIVATE KEY-----\\n\",\n \"client_email\": \"nobl9@nobl9.iam.gserviceaccount.com\",\n \"client_id\": \"eddba9e2f66237213812\",\n \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n \"token_uri\": \"https://oauth2.googleapis.com/token\",\n \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/nobl9%40nobl9.iam.gserviceaccount.com\"\n}"
queryDelay:
value: 1
unit: Second
Field | Type | Description |
---|---|---|
queryDelay.unit mandatory | enum | Specifies the unit for the query delay. Possible values: Second | Minute . • Check query delay documentation for default unit of query delay for each source. |
queryDelay.value mandatory | numeric | Specifies the value for the query delay. • Must be a number less than 1440 minutes (24 hours). • Check query delay documentation for default unit of query delay for each source. |
logCollectionEnabled optional | boolean | Optional. Defaults to false . Set to true if you'd like your direct to collect event logs. Contact us to activate it. |
releaseChannel mandatory | enum | Specifies the release channel. Accepted values: beta | stable . |
Source-specific fields | ||
bigQuery.serviceAccountKey mandatory | string | Provide your service account key to authenticate with Google Cloud. Read more about authentication with Google BigQuery |
- Apply your YAML definition using the
sloctl apply
command.
Agent connection method
Nobl9 Web
Follow the instructions below to create your BigQuery agent connection.
- Navigate to Integrations > Sources.
- Click
.
- Click the required Source button.
- Choose Agent.
-
Select one of the following Release Channels:
- The
stable
channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of abeta
release. Use it to avoid crashes and other limitations. - The
beta
channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.
- The
- Select a Project.
Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses thedefault
project. - Enter a Display Name.
You can enter a user-friendly name with spaces in this field. - Enter a Name.
The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example,my-project-1
). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result. - Enter a Description.
Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it. - Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
- The default value in BigQuery integration for Query delay is
0 seconds
.
infoChanging the Query delay may affect your SLI data. For more details, check the Query delay documentation. - The default value in BigQuery integration for Query delay is
- Click Add Data Source
- Deploy your agent in a Kubernetes cluster or Docker container.
sloctl
- Create a YAML definition to set up an agent connection with Google BigQuery. For this, refer to the following example:
apiVersion: n9/v1alpha
kind: Agent
metadata:
name: big-query
displayName: BigQuery Agent
project: default
spec:
description: Example BigQuery Agent
releaseChannel: stable
bigQuery: {}
queryDelay:
value: 1
unit: Second
Field | Type | Description |
---|---|---|
queryDelay.unit mandatory | enum | Specifies the unit for the query delay. Possible values: Second | Minute . • Check query delay documentation for default unit of query delay for each source. |
queryDelay.value mandatory | numeric | Specifies the value for the query delay. • Must be a number less than 1440 minutes (24 hours). • Check query delay documentation for default unit of query delay for each source. |
releaseChannel mandatory | enum | Specifies the release channel. Accepted values: beta | stable . |
- Apply your YAML definition using the
sloctl apply
command. - Deploy your agent in a Kubernetes cluster or Docker container.
Google BigQuery API rate limits
The following rate limits apply to the BigQuery API:
-
Query jobs. See Quotas and Limits reference.
-
Point density. Point density greater than 1000 data points per minute leads to errors. To address this, add point aggregation to your query.
BigQuery pricing is based on bytes read by the query.
Since BigQuery queries must contain the where
clause with date between
filter,
as required by the Nobl9 agent, your can use partitioning on the date_col
column to reduce the number of bytes read.
For example,
WHERE
{date_col} BETWEEN
DATETIME(@n9date_from)
AND DATETIME(@n9date_to)
Learn more about storage and query costs estimation and partitioned tables.