Skip to main content

Amazon Managed Service for Prometheus authentication

Reading time: 0 minute(s) (0 words)

This document is a description of different ways users can authenticate metrics queries sent by the Nobl9 agent to the Amazon Managed Service for Prometheus.

Basic concepts​

The following is a high-level overview of the basic AWS IAM concepts:

  • IAM stands for AWS Identity and Access Management. It is a system that ensures fine-grained permissions management in AWS.

  • ARN stands for Amazon Resource Name. It is a string with a specified format (: signs to separate Its different parts) that uniquely identifies a resource in AWS. For example:

    arn:aws:iam::123123123123:role/amp-query-role

  • IAM Policy is an object in AWS. When it is associated with an identity or resource, it defines its permissions. You can create your own policy or use AWS managed policies and describe them using JSON.

    The following is an example of the AWS managed policy that allows users to query Amazon Managed Service for Prometheus. The policy is called AmazonPrometheusQueryAccess:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "aps:GetLabels",
                    "aps:GetMetricMetadata",
                    "aps:GetSeries",
                    "aps:QueryMetrics"
                ],
                "Effect": "Allow",
                "Resource": "*"
            }
        ]
    }

    The aps:QueryMetrics allows the agent to query AWS for Metrics (where aps stands for Amazon Prometheus Service).

You can find a complete list of permissions along with associated actions here.

  • IAM Identity can either be a user, user group, or role. You can attach IAM Policies to IAM Identities to define what a specific identity is permitted to do.

  • IAM User is an entity you can create. It represents a person or an application that interacts with AWS through permission policies. IAM User consists of a name and static credentials, which can be (most commonly):

    • Password is a standard static credential for a user.

    • Access keys are common for applications and consist ofAccess key and Secret access key.

  • IAM Role is similar to a user with several key differences:

    • It is not associated with one person/application. It can be assumed by any user or application that requires it.

    • It does not have static credentials. Instead, it provides you with temporary security credentials for your role session when you assume a role.

tip

A role can be assumed by another role. This is called role chaining.

Nobl9 agent signing process​

Nobl9 agent uses Golang AWS SDK (v1), which provides a Signature V4 signing process for signing requests to Amazon Managed Service for Prometheus.

Authentication paths​

The paths listed below do not cover all the authentication paths an application may choose from when accessing Amazon Managed Service for Prometheus. The purpose of this document is to list the most common ways that the Nobl9 agent supports.

The following diagram outlines all possible authentication paths for Amazon Prometheus and their copendencies (click to enlarge):

AWS Authentication paths used by Nobl9 agent
Image 1: AWS Authentication paths used by the Nobl9 agent
note

The primary environment we expect the agent to run in is a k8s cluster.

Dedicated user with static credentials​

In the scenario outlined below, we create a dedicated user and generate access keys for this user. You can do this through the Security credentials tab on the user view, where you can Create access key. We will be using the generated access keys to sign our request to query metrics.

tip

Once it is generated, store your secret key in a safe location as you will not be able to access it afterward.

When created, you must expose the Access key to the agent. In the following example, we use the env variables configuration:

export AWS_ACCESS_KEY_ID="ASIA!XAE6VILQ54NYAOV"
export AWS_SECRET_ACCESS_KEY="Q16Vep/WU1JHCLo1oMIYOoGAB6j2MFtjZPqm0ac"
note

This is a preferred method of exposing the AWS security credentials to the agent. For details on other possible ways, go to this page.

If you are running the agent on k8s cluster, add these env declarations to your agent configuration:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: agent-container
          env:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: <KEY_NAME>
                  name: <SECRET_NAME>
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: <KEY_NAME>
                  name: <SECRET_NAME>

The standard and secure way to provide secrets to the running container in k8s is through Secret resource. You can also provide these credentials directly in the Deployment, but we do not recommend this method.

If you already have these credentials deployed in a Secret in your cluster, replace the KEY_NAME and SECRET_NAME placeholders accordingly.

If you do not have such Secret, follow the below example that shows how to create it:

apiVersion: v1
kind: Secret
metadata:
  name: nobl9-agent-user-amazon-credentials
  namespace: default
type: Opaque
stringData:
  aws_access_key_id: "ASIA!XAE6VILQ54NYAOV"
  aws_secret_access_key: "Q16Vep/WU1JHCLo1oMIYOoGAB6j2MFtjZPqm0ac"
note

aws_access_key_id and aws_secret_access_key contain exemplary values.

Now, replace the placeholders from the above YAML:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: agent-container
          env:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: aws_access_key_id
                  name: nobl9-agent-user-amazon-credentials
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: aws_secret_access_key
                  name: nobl9-agent-user-amazon-credentials

Defining user’s permission​

The agent connection to AWS is fully configured once you deploy the agent with the credentials exposed to the running container.

Make sure that the user you have generated the credentials for has the correct permissions to query metrics. To do that, click the Add permissions button in the Permissions section on the user view in the AWS console.

We recommend using a predefined permission policy AmazonPrometheusQueryAccess.

You can also define your own policy. To do that, Amazon Prometheus needs the aps:QueryMetrics permission. The below example shows a JSON configuration of the permission:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "aps:QueryMetrics"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

The configuration is complete once you have assigned the above permission to a user.

Defining permissions for shared roles​

If you do not want to grant the permission directly to a user, AWS allows you to create a shared role that this user can assume.

To assign permissions to a shared role, follow all the authentication path steps outlined above. Using a shared role method, you still need to provide the agent with the users' credentials. However, instead of assigning permission directly to a user, you make it assume a shared role with the sts:AssumeRole permission.

To assign permissions to a shared role:

  1. Grant the user with sts:AssumeRole that points to the role ARN you wish to assume. For example, you can do this through the following JSON configuration:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Resource": "arn:aws:iam::123123123123:role/amp-query-role"
        }
      ]
    }
Assumptions

The above example assumes that your role ARN is arn:aws:iam::123123123123:role/amp-query-role and it has the sts:QueryMetrics permission. You can add permission policies following the same steps that you assign them to an individual user.

  1. Tell the role you want to assume to trust this user
    1. Use the Trust relationships tab on the Role view in your AWS Console.
    2. Click Edit trust relationships. Below is an example of JSON configuration:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123123123123:user/agent-user"
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}
Assumptions

This example assumes that your user's ARN is arn:aws:iam::123123123123:user/agent-user.

  1. Now, set the env variable N9_PROMETHEUS_AWS_ROLE_ARN for the agent. Configure it in the agent's Deployment:
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: agent-container
          env:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: aws_access_key_id
                  name: nobl9-agent-user-amazon-credentials
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: aws_secret_access_key
                  name: nobl9-agent-user-amazon-credentials
            - name: N9_PROMETHEUS_AWS_ROLE_ARN
              value: arn:aws:iam::123123123123:role/amp-query-role

Role-based access​

This method differs from a user-based authentication. In the role-based access method, you are not providing any static credentials to the agent. The role credentials used to sign the request are supplied by the ServiceAccount that you attach to the Pod in the Kubernetes cluster.

IAM roles for service accounts

For more details on IAM roles for service accounts, refer to the AWS guide available here

Before creating the ServiceAccount and configuring the Trust relationship for the role you want the ServiceAccount to provide, make sure an OIDC provider is running in your cluster. For more details, refer to Create an IAM OIDC provider for your cluster.

Roles for Amazon EC2

For the time being, Nobl9 does not support IAM roles for Amazon EC2.

Defining permissions for a role​

If you want to bind a role to the ServiceAccount, the role has to have aps:QueryMetrics permission.

Important notes

  • Refer to the above section on the Access keys for more details.

  • To create a role that can be linked to the ServiceAccount, refer to this guide.

  1. Update the Trust relationship of the created role. Follow the below JSON declaration:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Federated": "arn:aws:iam::123123123123:oidc-provider/${OIDC_PROVIDER}"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:amp-query-role"
            }
          }
        }
      ]
    }

  2. Now, create a ServiceAccount for this role:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: amp-query-role
      namespace: default
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123123123123:role/amp-query-role

  3. To complete the configuration, you need to tell the agent's Deployment that you want this ServiceAccount to be attached to the pods that run under it. To do this, modify the Deployment as follows:

    apiVersion: apps/v1
    kind: Deployment
    spec:
      template:
        spec:
          serviceAccount: amp-query-role
          serviceAccountName: amp-query-role

Role chaining​

This case builds on top of the previous one and differs in a similar way in which the user with permission differs from the user with the assumed role.

Our role does not have the permissions defined itself but instead relies on a different role to provide it:

  1. The JSON declarations for both roles will look the same as in the previous case, Trust relationship for arn:aws:iam::123123123123:role/amp-query-role:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::123123123123:user/the-role-which-wants-to-assume"
          },
          "Action": "sts:AssumeRole",
          "Condition": {}
        }
      ]
    }

  2. Apply Permissions for the arn:aws:iam::123123123123:role/the-role-which-wants-to-assume:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Resource": "arn:aws:iam::123123123123:role/amp-query-role"
        }
      ]
    }

  3. Create and configure ServiceAccount for the the-role-which-wants-to-assume just as in:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: the-role-which-wants-to-assume
      namespace: default
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123123123123:role/the-role-which-wants-to-assume

  4. To finish the configuration, set N9_PROMETHEUS_AWS_ROLE_ARN in the agent's environment:

    apiVersion: apps/v1
    kind: Deployment
    spec:
      template:
        spec:
          serviceAccount: the-role-which-wants-to-assume
          serviceAccountName: the-role-which-wants-to-assume
          containers:
            - name: agent-container
              env:
                - name: N9_PROMETHEUS_AWS_ROLE_ARN
                  value: arn:aws:iam::123123123123:role/amp-query-role
Check out these related guides and references:
Create an IAM OIDC provider for your clusterAmazon Prometheus docsCreating an IAM role and Policy for your service accountAmazon Prometheus docsAWS identity and access management (IAM)Amazon Prometheus docsAmazon resource names (ARNs)Amazon Prometheus docs
On this page