Skip to main content

Amazon Redshift

Amazon Redshift is a managed scalable database warehouse where Nobl9 users can store their metrics information. Nobl9 allows retrieving metrics data from Redshift, enabling customers to use standard SQL statements that require two specific return values - a value and a timestamp.

Authentication

Nobl9 supports authenticating with Amazon Redshift’s Data API using the AWS Secrets Manager. To connect to Redshift through Direct and Agent configuration, you must create a secret and ensure the secret is tagged with the RedshiftDataFullAccess permission.

tip

When running the Agent you will also be asked to provide the ARN for the secret.

For more information on Redshift secrets, refer to Using the Amazon Redshift Data API | Amazon Redshift Documentation.

Authentication - Scope of Support

  • Nobl9 does not support SQL connections.
  • Nobl9 does not support temporary credentials.
  • For authenticating with the Data API, Nobl9 only supports authentication via AWS Secrets Manager stored secret.

Adding Amazon Redshift as a Data Source in the UI

To add Amazon Redshift as a data source in Nobl9 using the Agent or Direct connection method, follow these steps:

  1. Navigate to Integrations > Sources.

  2. Click the button.
  3. Select the Amazon Redshift icon.

  4. Choose a configuration method (Direct or Agent), then configure the source as described below.

Amazon Redshift Direct

Direct Configuration in the UI

Direct connection to Amazon Redshift requires users to enter their credentials which Nobl9 stores safely. To set up this type of connection:

  1. Enter your AWS Secret ARN (mandatory).
    The secret must be tagged with RedshiftDataFullAccess permission. For more information, see Data API | Amazon Redshift Documentation

  2. Add AWS Access Key ID and AWS Secret Access Key (mandatory).
    The Access Keys are user security credentials that are used to make programmatic calls to Amazon Web Services. Both AWS Access Key ID and AWS Secret Access Key are created as a pair. Your AWS Access Key ID and AWS Secret Access Key are encrypted before being stored on the Nobl9 server. You will also need Redshift AmazonRedshiftDataFullAccess API permissions.

  3. Select a Project (mandatory).
    Using the Project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, a default value appears.

  4. Enter a Display name (optional).
    You can enter a friendly name with spaces in this field.

  5. Enter a Name (mandatory).
    The name is mandatory and can only contain lowercase, alphanumeric characters and dashes (for example, my-project-name). This field is populated automatically when you enter a display name, but you can edit the result.

  6. Enter a Description (optional).
    Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.

  7. Click the Add Data Source button.

Amazon Redshift Agent

Agent Configuration in the UI

Follow the instructions below to create your Amazon Redshift Agent connection. Refer to the section above for the description of the fields.

  1. Select a Project (mandatory).

  2. Enter a Display name (optional).

  3. Enter a Name (mandatory).

  4. Enter a Description (optional).

  5. Click the Add Data Source button.

Agent Using CLI - YAML

The YAML for setting up an Agent connection to Amazon Redshift looks like this:

apiVersion: n9/v1alpha
kind: Agent
metadata:
name: redshift-agent
displayName: Redshift Agent # optional
project: default
spec:
description: Agent settings for redshift datasource # optional
sourceOf:
- Metrics
redshift: {}
warning

You can deploy only one Agent in one YAML file by using the sloctl apply command.

Deploying Amazon Redshift Agent

When you add the data source, Nobl9 automatically generates a Kubernetes configuration and a Docker command line for you to use to deploy the Agent. Both of these are available in the web UI, under the Agent Configuration section. Be sure to swap in your credentials (e.g., replace <AWS_SECRET_ARN>, <AWS_ACCESS_KEY_ID>, and <AWS_SECRET_ACCESS_KEY> with your organization key).

If you use Kubernetes, you can apply the supplied YAML config file to a Kubernetes cluster to deploy the Agent. It will look something like this:

# DISCLAIMER: This deployment description contains only the fields necessary for the purpose of this demo.
# It is not a ready-to-apply k8s deployment description, and the client_id and client_secret are only exemplary values.

apiVersion: v1
kind: Secret
metadata:
name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
namespace: default
type: Opaque
stringData:
aws_access_key_id: <AWS_ACCESS_KEY_ID>
aws_secret_access_key: <AWS_SECRET_ACCESS_KEY>
aws_secret_arn: <AWS_SECRET_ARN>
client_id: client_id
client_secret: client_secret
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
namespace: default
spec:
replicas: 1
selector:
matchLabels:
nobl9-agent-name: my-amazon-redshift
nobl9-agent-project: default
nobl9-agent-organization: nobl9-dev
template:
metadata:
labels:
nobl9-agent-name: my-amazon-redshift
nobl9-agent-project: default
nobl9-agent-organization: nobl9-dev
spec:
containers:
- name: agent-container
image: nobl9/agent:latest
resources:
requests:
memory: "350Mi"
cpu: "0.1"
env:
- name: N9_CLIENT_ID
valueFrom:
secretKeyRef:
key: client_id
name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: aws_access_key_id
name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: aws_secret_access_key
name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
- name: AWS_SECRET_ARN
valueFrom:
secretKeyRef:
key: aws_secret_arn
name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
- name: N9_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: client_secret
name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
# The N9_METRICS_PORT is a variable specifying the port to which the /metrics and /health endpoints are exposed.
# The 9090 is the default value and can be changed.
# If you don’t want the metrics to be exposed, comment out or delete the N9_METRICS_PORT variable.
- name: N9_METRICS_PORT
value: "9090"

Creating SLOs with Amazon Redshift

Creating SLOs in the UI

Follow the instructions below to create your SLOs with Amazon Redshift in the UI:

  1. Navigate to Service Level Objectives.

  2. Click the button.
  3. In step 1 of the SLO wizard, select the Service the SLO will be associated with.

  4. In step 2, select Redshift as the Data Source for your SLO.

    1. Enter a Region (mandatory).
      Use one of the regional endpoints that are listed here.

    2. Enter a Cluster ID (mandatory).
      It is an identifier of your Amazon Redshift cluster that is a part of your Redshift secret. For more details on Redshift, secrets go here. For example redshift-cluster-1.

    3. Enter a Database name (mandatory).
      It is the name of your Amazon Redshift database. For example dev.

    4. Select a metric type. You can choose between two types of metrics:

      1. A Threshold Metric is a single time series evaluated against a threshold.

      2. A Ratio Metric allows you to enter two-time series to compare (for example, a count of good requests and total requests).

    note

    For the Ratio Metric, you can choose the Data Count Method:

    • For the Non-incremental method, we expect it to be the components of the sum.
    • For the Incremental method, we expect the value of a metric to be the current sum of some numerator.

    For more information, refer to the SLO Calculations Guide.

    1. Enter a Query or Good query and Total query:

      1. Query example for the Threshold metric (Raw metric):
        Query: SELECT value as n9_value, timestamp as n9_time FROM httpstatuscodes WHERE timestamp BETWEEN :n9date_from AND :n9date_to

      2. Query example for the Ratio metric (Count metric):
        Good query:SELECT value as n9value, timestamp as n9date FROM httpstatuscodes WHERE timestamp BETWEEN :n9date_from AND :n9date_to
        Total query: SELECT value as n9value, timestamp as n9date FROM sinusoid WHERE timestamp BETWEEN :n9date_from AND :n9date_to

  5. In step 3, define a Time Window for the SLO.

  6. In step 4, specify the Error Budget Calculation Method and your Objective(s).

  7. In step 5, add a Name, Description, and other details about your SLO. You can also select Alert Policies and Labels on this screen.

  8. When you’re done, click Create SLO.

SLOs using Amazon Redshift - YAML samples

Here’s an example of Amazon Redshift using a rawMetric (Threshold Metric)

apiVersion: n9/v1alpha
kind: SLO
metadata:
name: redshift-raw
displayName: Redshift Raw SLO
project: redshift
spec:
description: Redshift SLO Description
service: redshift
indicator:
metricSource:
name: redshift
project: redshift
budgetingMethod: Occurrences
objectives:
- target: 0.8
value: 0.8
op: lte
rawMetric:
query:
redshift:
region: eu-central-1
clusterId: n9-dev-tooling-redshift
databaseName: dev
query: SELECT value as n9value, timestamp as n9date FROM sinusoid WHERE
timestamp BETWEEN :n9date_from AND :n9date_to
displayName: average
- target: 0.9
value: 0.9
op: lte
rawMetric:
query:
redshift:
region: eu-central-1
clusterId: n9-dev-tooling-redshift
databaseName: dev
query: SELECT value as n9value, timestamp as n9date FROM sinusoid WHERE
timestamp BETWEEN :n9date_from AND :n9date_to
displayName: so-so
timeWindows:
- calendar:
startTime: "2020-11-14 11:00:00"
timeZone: Etc/UTC
count: 1
isRolling: false
unit: Day

The Amazon Redshift SLO requires the following fields:

n9date and n9value:

Amazon Redshift SQL query needs to return two values - n9date and n9value:

  • n9date is the timestamp for the data.

  • n9value is a float containing the actual metric.

    Note that Amazon RedShift accepts these values in the following format:

    • :n9date_from

    • :n9date_to

    This allows users to enter virtually any query.

Querying the Amazon Redshift Server

To call the AWS Redshift Data API, Nobl9 runs the aws redshift-data execute-statement command that is executed once per minute. Nobl9 queries for data from the previous minute.

Amazon Redshift API Rate Limits

The following rate limits apply to the Amazon Redshift API:

  • The maximum query result size is 100 MB. If a call returns more than 100 MB of response data, the call is ended.

  • The maximum retention time for query results is 24 hours.

  • The maximum query statement size is 100 KB.

  • The Data API is available to query single-node and multiple-node clusters of the following node types:

    • dc2.large
    • dc2.8xlarge
    • ds2.xlarge
    • ds2.8xlarge
    • ra3.xlplus
    • ra3.4xlarge
    • ra3.16xlarge
  • The cluster must be in a virtual private cloud (VPC) based on the Amazon VPC service.

Data API | Amazon Redshift Documentation

Regional Endpoints | Amazon Redshift Documentation

Agent Metrics | Nobl9 Documentation

Creating SLOs via Terraform | Nobl9 Terraform Documentation

Creating Agents via Terraform | Nobl9 Terraform Documentation