Skip to main content

Amazon Redshift

Reading time: 0 minute(s) (0 words)

Amazon Redshift is a managed scalable database warehouse where Nobl9 users can store their metrics information. Nobl9 allows retrieving metrics data from Redshift, enabling customers to use standard SQL statements that require two specific return values - a value and a timestamp.

Authentication​

Nobl9 supports authenticating with Amazon Redshift’s Data API using the AWS Secrets Manager. To connect to Redshift through Direct and Agent configuration, you must create a secret and ensure the secret is tagged with the RedshiftDataFullAccess permission.

tip

When running the Agent you will also be asked to provide the ARN for the secret.

For more information on Redshift secrets, refer to Using the Amazon Redshift Data API | Amazon Redshift Documentation.

Authentication - Scope of Support​

  • Nobl9 does not support SQL connections.
  • Nobl9 does not support temporary credentials.
  • For authenticating with the Data API, Nobl9 only supports authentication via AWS Secrets Manager stored secret.

Adding Amazon Redshift as a Data Source in the UI​

To add Amazon Redshift as a data source in Nobl9 using the Agent or Direct connection method, follow these steps:

  1. Navigate to Integrations > Sources.

  2. Click the button.

  3. Select the Amazon Redshift icon.

  4. Choose a configuration method (Direct or Agent), then configure the source as described below.

Amazon Redshift Direct​

Direct Configuration in the UI​

Direct connection to Amazon Redshift requires users to enter their credentials which Nobl9 stores safely. To set up this type of connection:

  1. Enter your AWS Secret ARN (mandatory).
    The secret must be tagged with RedshiftDataFullAccess permission. For more information, see Data API | Amazon Redshift Documentation

  2. Add AWS Access Key ID and AWS Secret Access Key (mandatory).
    The Access Keys are user security credentials that are used to make programmatic calls to Amazon Web Services. Both AWS Access Key ID and AWS Secret Access Key are created as a pair. Your AWS Access Key ID and AWS Secret Access Key are encrypted before being stored on the Nobl9 server. You will also need Redshift AmazonRedshiftDataFullAccess API permissions.

  3. Select a Project (mandatory).
    Using the Project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, a default value appears.

  4. Enter a Display name (optional).
    You can enter a friendly name with spaces in this field.

  5. Enter a Name (mandatory).
    The name is mandatory and can only contain lowercase, alphanumeric characters and dashes (for example, my-project-name). This field is populated automatically when you enter a display name, but you can edit the result.

  6. Enter a Description (optional).
    Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.

  7. Click the Add Data Source button.

Direct using CLI - YAML​

The YAML for setting up a Direct connection to Amazon Redshift looks like this:

apiVersion: n9/v1alpha
kind: Direct
metadata:
  name: redshift-direct
  displayName: Redshift Direct
  project: redshift-direct
spec:
  redshift:
    accessKeyID: "" #secret
    secretAccessKey: "" #secret
    secretARN: "" #secret
  sourceOf:
    - Metrics

Amazon Redshift Agent​

Agent Configuration in the UI​

Follow the instructions below to create your Amazon Redshift Agent connection. Refer to the section above for the description of the fields.

  1. Select a Project (mandatory).

  2. Enter a Display name (optional).

  3. Enter a Name (mandatory).

  4. Enter a Description (optional).

  5. Click the Add Data Source button.

Agent Using CLI - YAML​

The YAML for setting up an Agent connection to Amazon Redshift looks like this:

apiVersion: n9/v1alpha
kind: Agent
metadata:
  name: redshift-agent
  displayName: Redshift Agent # optional
  project: default
spec:
  description: Agent settings for redshift datasource # optional
  sourceOf:
    - Metrics
  redshift: {}
warning

You can deploy only one Agent in one YAML file by using the sloctl apply command.

Deploying Amazon Redshift Agent​

When you add the data source, Nobl9 automatically generates a Kubernetes configuration and a Docker command line for you to use to deploy the Agent. Both of these are available in the web UI, under the Agent Configuration section. Be sure to swap in your credentials (e.g., replace <AWS_SECRET_ARN>, <AWS_ACCESS_KEY_ID>, and <AWS_SECRET_ACCESS_KEY> with your organization key).

If you use Kubernetes, you can apply the supplied YAML config file to a Kubernetes cluster to deploy the Agent. It will look something like this:

# DISCLAIMER: This deployment description contains only the fields necessary for the purpose of this demo.
# It is not a ready-to-apply k8s deployment description, and the client_id and client_secret are only exemplary values.

apiVersion: v1
kind: Secret
metadata:
  name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
  namespace: default
type: Opaque
stringData:
  aws_access_key_id: <AWS_ACCESS_KEY_ID>
  aws_secret_access_key: <AWS_SECRET_ACCESS_KEY>
  aws_secret_arn: <AWS_SECRET_ARN>
  client_id: client_id
  client_secret: client_secret
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      nobl9-agent-name: my-amazon-redshift
      nobl9-agent-project: default
      nobl9-agent-organization: nobl9-dev
  template:
    metadata:
      labels:
        nobl9-agent-name: my-amazon-redshift
        nobl9-agent-project: default
        nobl9-agent-organization: nobl9-dev
    spec:
      containers:
        - name: agent-container
          image: nobl9/agent:latest
          resources:
            requests:
              memory: "350Mi"
              cpu: "0.1"
          env:
            - name: N9_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  key: client_id
                  name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: aws_access_key_id
                  name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: aws_secret_access_key
                  name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
            - name: AWS_SECRET_ARN
              valueFrom:
                secretKeyRef:
                  key: aws_secret_arn
                  name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
            - name: N9_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  key: client_secret
                  name: nobl9-agent-nobl9-dev-default-my-amazon-redshift
            # The N9_METRICS_PORT is a variable specifying the port to which the /metrics and /health endpoints are exposed.
            # The 9090 is the default value and can be changed.
            # If you don’t want the metrics to be exposed, comment out or delete the N9_METRICS_PORT variable.
            - name: N9_METRICS_PORT
              value: "9090"

Creating SLOs with Amazon Redshift​

Creating SLOs in the UI​

Follow the instructions below to create your SLOs with Amazon Redshift in the UI:

  1. Navigate to Service Level Objectives.

  2. Click the button.

  3. In step 1 of the SLO wizard, select the Service the SLO will be associated with.

  4. In step 2, select Redshift as the Data Source for your SLO.

    1. Enter a Region (mandatory).
      Use one of the regional endpoints that are listed here.

    2. Enter a Cluster ID (mandatory).
      It is an identifier of your Amazon Redshift cluster that is a part of your Redshift secret. For more details on Redshift, secrets go here. For example redshift-cluster-1.

    3. Enter a Database name (mandatory).
      It is the name of your Amazon Redshift database. For example dev.

    4. Select a metric type. You can choose between two types of metrics:

      1. A Threshold Metric is a single time series evaluated against a threshold.

      2. A Ratio Metric allows you to enter two-time series to compare (for example, a count of good requests and total requests).

    note

    For the Ratio Metric, you can choose the Data Count Method:

    • For the Non-incremental method, we expect it to be the components of the sum.
    • For the Incremental method, we expect the value of a metric to be the current sum of some numerator.

    For more information, refer to the SLO Calculations Guide.

    1. Enter a Query or Good query and Total query:

      1. Query example for the Threshold metric (Raw metric):
        Query: SELECT value as n9_value, timestamp as n9_time FROM httpstatuscodes WHERE timestamp BETWEEN :n9date_from AND :n9date_to

      2. Query example for the Ratio metric (Count metric):
        Good query:SELECT value as n9value, timestamp as n9date FROM httpstatuscodes WHERE timestamp BETWEEN :n9date_from AND :n9date_to
        Total query: SELECT value as n9value, timestamp as n9date FROM sinusoid WHERE timestamp BETWEEN :n9date_from AND :n9date_to

  5. In step 3, define a Time Window for the SLO.

  6. In step 4, specify the Error Budget Calculation Method and your Objective(s).

  7. In step 5, add a Name, Description, and other details about your SLO. You can also select Alert Policies and Labels on this screen.

  8. When you’re done, click Create SLO.

SLOs using Amazon Redshift - YAML samples​

Here’s an example of Amazon Redshift using a rawMetric (Threshold Metric)

apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: redshift-raw
  displayName: Redshift Raw SLO
  project: redshift
spec:
  description: Redshift SLO Description
  service: redshift
  indicator:
    metricSource:
      name: redshift
      project: redshift
  budgetingMethod: Occurrences
  objectives:
  - target: 0.8
    value: 0.8
    op: lte
    rawMetric:
      query:
        redshift:
         region: eu-central-1
         clusterId: n9-dev-tooling-redshift
         databaseName: dev
         query: SELECT value as n9value, timestamp as n9date FROM sinusoid WHERE
              timestamp BETWEEN :n9date_from AND :n9date_to
    displayName: average
  - target: 0.9
    value: 0.9
    op: lte
    rawMetric:
      query:
        redshift:
         region: eu-central-1
         clusterId: n9-dev-tooling-redshift
         databaseName: dev
         query: SELECT value as n9value, timestamp as n9date FROM sinusoid WHERE
              timestamp BETWEEN :n9date_from AND :n9date_to
    displayName: so-so
  timeWindows:
  - calendar:
      startTime: "2020-11-14 11:00:00"
      timeZone: Etc/UTC
    count: 1
    isRolling: false
    unit: Day

The Amazon Redshift SLO requires the following fields:

n9date and n9value:

Amazon Redshift SQL query needs to return two values - n9date and n9value:

  • n9date is the timestamp for the data.

  • n9value is a float containing the actual metric.

    Note that Amazon RedShift accepts these values in the following format:

    • :n9date_from

    • :n9date_to

    This allows users to enter virtually any query.

Querying the Amazon Redshift Server​

To call the AWS Redshift Data API, Nobl9 runs the aws redshift-data execute-statement command that is executed once per minute. Nobl9 queries for data from the previous minute.

Amazon Redshift API Rate Limits​

The following rate limits apply to the Amazon Redshift API:

  • The maximum query result size is 100 MB. If a call returns more than 100 MB of response data, the call is ended.

  • The maximum retention time for query results is 24 hours.

  • The maximum query statement size is 100 KB.

  • The Data API is available to query single-node and multiple-node clusters of the following node types:

    • dc2.large
    • dc2.8xlarge
    • ds2.xlarge
    • ds2.8xlarge
    • ra3.xlplus
    • ra3.4xlarge
    • ra3.16xlarge

  • The cluster must be in a virtual private cloud (VPC) based on the Amazon VPC service.

Data API | Amazon Redshift Documentation

Regional Endpoints | Amazon Redshift Documentation

Agent Metrics | Nobl9 Documentation

Creating SLOs via Terraform | Nobl9 Terraform Documentation

Creating Agents via Terraform | Nobl9 Terraform Documentation

Last updated on
Submit Feedback