Agent

Setting Up With Thanos

Configuring external_labels for Thanos

Due to the way we extract node and container information from the Prometheus metrics aggregated by Thanos, we recommend that the keys configured for podLabel, nodeLabel and containerLabel be renamed if already used in your Prometheus external labels.

Setting up the thanos receive component

Due to availability concerns with using the sidecar, we HEAVILY RECOMMEND using the thanos receive component which emulates all the realtime results of Prometheus that the agent needs to get metrics.

Kindly review the setup docs here and configure the remote_write sections of your prometheus installation(s).

Here's a sample of the prometheus configuration desired to work with the thanos receive component

# prometheus.yaml
remote_write:
  url: <thanos-receive-url>/api/v1/receive
  headers:
    - THANOS-TENANT: <replex-cluster-name>

Configuring the Querier With The Replex Agent

The Querier / Query Gateway is a part of the thanos components which provides a Prometheus compatible endpoint that works just fine with the replex agent.

If you have a Thanos instance set up, verify there is support for access to this endpoint using a simple curl command, replace THANOS_QUERIER_URL with the url of your thanos instance

curl $THANOS_QUERIER_URL/api/v1/query?query=up

This should provide reasonable output similar to this which indicates that endpoint is prometheus compatible and would work with the agent.

up{instance="replex.io:9090", job="prometheus"} 1
up{instance="replex.io:9091", job="pushgateway"} 1
up{instance="replex.io:9093", job="alertmanager"} 1
up{instance="replex.io:9100", job="node"} 1

From the previous guide, provide the THANOS_QUERIER_URL under the prometheus.url variable as depicted in the agent docs presented in this section, and you're up and running with Thanos on Replex.

Configuring Self-Signed SSL Certificates (On-Prem Only)

For on-prem pushgateway deployments, if the puhgateway is served with a self-signed SSL certificate, the agent may encounter errors when trying to sync with the pushgateway.

To resolve this, you can use the sslCertificate Helm chart parameter to pass your certificate into the agent.

Example:

sslCertificate: “-----BEGIN CERTIFICATE-----\nMIIC1TCCAb2gAwIBAgIJAKbCs/2knCwGMA0GCSqGSIb3DQEBBQUAMBoxGDAWBgNV\nZAeRdaEZS6Bs\n-----END CERTIFICATE——"

Filesystem Metrics (Prometheus)

This section is only for setups using prometheus as metrics provider. We use metrics from different sources for collecting PVC informations. The default setting uses cAdvisor's kubelet_volume* metrics.

cAdvisor

The default setup uses the cAdvisor metrics to get the PVC informations. In that case the METRICS_FILESYSTEM environment variable can be left at the default value that is cadvisor.

Metrics used:

Storage Metric

cAdvisor Metrics

Capacity

kubelet_volume_stats_capacity_bytes

Used

kubelet_volume_stats_used_bytes

CSI

If kubelet_volume* metrics are not available and you are using CSI plugins, you must set the METRICS_FILESYSTEM environment variable to csi. In that case kube-state-metrics is required. For csi, we get the PVC informations from the node_exporter and kube-state-metrics metrics.

Metrics used:

Storage Metric

node_exporter Metrics

kube-state-metrics Metrics

Capacity

node_filesystem_size_bytes

kube_persistentvolumeclaim_info

Used

node_filesystem_size_bytes - node_filesystem_free_bytes

kube_persistentvolumeclaim_info

Exposed Metrics

The Agent self exposes metrics. The metrics can be accessed via the /metrics route on port :8083.

Metric

Type

Labels

Description

replex_agent_provider_status

gauge

name: metrics provider name

Indicates whether or not a metrics provider is reachable. 1 if it is reachable and 0 if not

replex_agent_sync_duration_count

counter

agent_version, response_code

Total number of times the metrics were synchronized with the Replex server

replex_agent_sync_duration_sum

gauge

agent_version, response_code

The total duration of all sync requests to the Replex server in seconds

replex_agent_retry_cache_size

gauge

cluster_id

Count of cached metrics that are waiting to be re-sent to the replex server

replex_agent_failed_metrics_total

counter

cluster_id

The total count of once failed metrics

Used Metrics

These are the metrics the agent currently uses:

Property

Description

Prometheus

Instana (plugin: metric)

Stackdriver

Datadog

1

Container CPU Usage

container_cpu_usage_seconds_total

docker: cpu.total_usage

kubernetes.io/container/cpu/core_usage_time

kubernetes.cpu.usage.total

2

Container MEM Usage

container_memory_working_set_bytes

docker: memory.usage

kubernetes.io/container/memory/used_bytes

kubernetes.memory.working_set

3

Node CPU Usage

node_cpu_seconds_total

docker: cpu.total_usage

kubernetes.io/node/cpu/core_usage_time

kubernetes.cpu.usage.total

4

Node MEM Usage

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

docker: memory.usage

kubernetes.io/node/memory/used_bytes

kubernetes.memory.usage

5

Storage (Capacity)

kubelet_volume_stats_capacity_bytes, node_filesystem_size_bytes

-

kubernetes.io/pod/volume/total_bytes

kubernetes.kubelet.volume.stats.capacity_bytes

6

Storage (Used)

kubelet_volume_stats_used_bytes, node_filesystem_free_bytes

-

kubernetes.io/pod/volume/used_bytes

kubernetes.kubelet.volume.stats.used_bytes

7

Disk Capacity

container_fs_limit_bytes

-

-

system.disk.total

8

Disk Used

container_fs_usage_bytes

-

-

system.disk.used

9

Network I/O (Received)

container_network_receive_bytes_total

docker: network.rx.bytes

kubernetes.io/pod/network/received_bytes_count

kubernetes.network.rx_bytes

10

Network I/O (Sent)

container_network_transmit_bytes_total

docker: network.tx.bytes

kubernetes.io/pod/network/sent_bytes_count

kubernetes.network.tx_bytes

11

Disk I/O (Written)

container_fs_writes_bytes_total

docker: blkio.blk_write

-

kubernetes.io.write_bytes

12

Disk I/O (Read)

container_fs_reads_bytes_total

docker: blkio.blk_read

-

kubernetes.io.read_bytes

Environment Variables

Variable

Required

Default

Comment

1

REPLEX_TOKEN

Yes

2

METRIC_PROVIDER

Yes

Options: prometheus, datadog, stackdriver, instana, thanos

3

CLUSTER_ID

If "KUBERNETES_INFO_PROVIDER" == "kubernetes"

4

CLUSTER_NAME

If "KUBERNETES_INFO_PROVIDER" == "kubernetes"

5

PUSHGATEWAY_URL

No

https://pushgateway.replex.io/push

6

PROMETHEUS_SERVER_URL

If "METRIC_PROVIDER" == "prometheus"

7

DATADOG_API_KEY

If "METRIC_PROVIDER" == "datadog"

8

DATADOG_APPLICATION_KEY

If "METRIC_PROVIDER" == "datadog"

9

DATADOG_SITE

No

com

Options: com, eu

10

GCP_PROJECT_ID

If "METRIC_PROVIDER" == "stackdriver"

11

INSTANA_BASE_URL

If "METRIC_PROVIDER" == "instana"

Format: https://tenant-unit.instana.io

12

INSTANA_API_TOKEN

If "METRIC_PROVIDER" == "instana"

13

KUBERNETES_INFO_PROVIDER

No

kubernetes

Options: kubernetes, instana

14

INSTANA_CLUSTER_ID

No

15

ONLY_USE_READY_NODES

No

false

Track only nodes that are in "Ready" state

16

PROMETHEUS_NODE_LABEL

No

node

The label that represents the node in the Prometheus metrics

17

PROMETHEUS_CONTAINER_LABEL

No

container

The label that represents the container in the Prometheus metrics

18

PROMETHEUS_POD_LABEL

No

pod

The label that represents the pod in the Prometheus metrics

19

CLOUD_PROVIDER_OVERRIDE

No

Detecting automatically

Options: aws, azure, gce, custom, alibaba

20

USE_CONTROL_PLANE_COST

No

false

Track costs of the Kubernetes Control Plane

21

METRICS_FILESYSTEM

No

cadvisor

Specify the filesystem metric source. Options: cadvisor, csi

22

SYNC_INTERVAL_SECONDS

No

300

23

LOG_LEVEL

No

3

Higher value means higher verbosity

24

METRICS_RETRY_INTERVAL_SECONDS

No

300

25

METRICS_CACHE_DISK

No

true

Cache failed metrics on disk

26

METRICS_CACHE_DISK_DIR

No

/data/metrics

Directory to cache metrics if METRICS_CACHE_DISK == true

27

PROMETHEUS_BEARER_TOKEN

No

Prometheus server requests bearer token. Only if METRIC_PROVIDER == prometheus

PreviousCalculationsNextAggregator

Last updated