# Quick Start Guide ### Purpose The Replex Kubernetes Agent collects metadata from the local cluster via the kubernetes API and metrics from a metric provider (a local Prometheus instance, Datadog, Stackdriver or Instana) and sends this information to the replex pushgateway which stores it for further use. ### Requirements * A Kubernetes cluster to install the agent. * A metric provider. Either: * Prometheus instance that is running in the cluster and accessible via URL. * A thanos instance configured with a querier component * A Datadog account with API key and Application key provided. * A Stackdriver account. * An Instana account with an API token. * Helm 3 for the installation. * A replex token. ### Required parameters for the installation If only the Replex agent is to be used for data collection in the clusters, it must be installed in each individual cluster. The `kubernetesInfoProvider` parameter can be left at the default value of `kubernetes`. The helm installation requires a few mandatory parameters: | Parameter | Description | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | cluster.id | A string that identifies the cluster uniquely. This is used in order to distinguish two clusters that might have the same name (e.g. If a cluster called "development" is destroyed and later replaced by a new cluster that is also called "development" this ID should be different to identify both clusters uniquely). | | cluster.name | A human readable name for the cluster. | | replex.token | A token that is provided by Replex and is used by the pushgateway to authenticate requests from the agent. If the agent is installed in multiple clusters the same token can be used for all deployments. | | metrics.provider | The metric provider to be used. Can be either *prometheus*, *thanos*, *stackdriver*, *instana* or *datadog*. | | (Optional) pushgateway.url | The full URL to the replex pushgateway. Format: (Note the `/push` path). | | (Optional) onlyUseReadyNodes | Consider nodes without 'Ready' status as not running at all. | | (Optional) useControlPlaneCost | Track costs of the Kubernetes Control Plane. | ### Metric Provider Parameters * **prometheus** or **thanos**: | Parameter | Description | Default | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --------- | | prometheus.url | The API URL of the local Prometheus instance, for example | | | prometheus.nodeLabel | Sets the value which represents the 'node label'. This label is usually `node` or `instance`. | node | | prometheus.containerLabel | Sets the value which represents the 'container label'. This label is usually `container` or `container_name`. | container | | prometheus.podLabel | Sets the value which represents the 'pod label'. This label is usually `pod` or `pod_name`. | pod | Parameters `prometheus.*Label` allow overriding default labels in Prometheus installation. This will be the label that contains actual entity name in a Prometheus time-series like `container_cpu_usage_seconds_total`. * **datadog**: | Parameter | Description | Default | | ----------------------- | --------------------------------------------------------------------------------- | ------- | | datadog.apiKey | Your Datadog API Key. | | | datadog.applicationKey | Your Datadog Application Key. | | | (Optional) datadog.site | Either `com` if you are on Datadog US site or `eu` if you are on Datadog EU site. | com | **Note**: Retrieve your Datadog keys from [the Datadog settings page](https://app.datadoghq.com/account/settings#api). * **stackdriver**: | Parameter | Description | Default | | --------------------- | ------------------------------------------------------------------------------------------------------------------- | ------- | | stackdriver.projectId | Your GCP project ID. The agent will authenticate with Google using the service account provided by the environment. | | * **instana**: | Parameter | Description | Default | | ---------------- | ------------------------------------------------------------------------------ | ------- | | instana.baseUrl | This is the base URL of a tenant unit, e.g. . | | | instana.apiToken | Valid Instana API token. | | #### Instana Single-Agent Setup This section is relevant if you already use Instana to monitor your Kubernetes clusters. With a single deployment of the replex Agent, replex can pull all metrics and kubernetes information from the Instana API for all your clusters. This way, you don't need to install the replex Agent on all the clusters that you want to monitor costs for. To activate this feature it is required to set the Helm parameter **kubernetesInfoProvider** to `instana`. Additionally, the Instana base URL (`instana.baseUrl`) and API Token (`instana.apiToken`) are required. For the single agent setup, the `cluster.id` and `cluster.name` parameters should not be set since the agent automatically uses the cluster IDs from Instana. ### Installation 1. Add the Replex Helm repository ```bash helm repo add replex https://registry.replex.io/chartrepo/public ``` 2. Create a namespace to install the agent in ```bash kubectl create namespace replex-k8s-agent ``` 3. Create a file called `values.yaml` to specify Helm parameters. The file should look like this: * **prometheus** or **thanos**: ```yaml cluster: id: name: replex: token: metrics: provider: prometheus prometheus: url: containerLabel: podLabel: nodeLabel: ``` * **datadog**: ```yaml cluster: id: name: replex: token: metrics: provider: datadog datadog: apiKey: applicationKey: site: ``` * **stackdriver**: ```yaml cluster: id: name: replex: token: metrics: provider: stackdriver stackdriver: projectId: ``` * **instana**: ```yaml cluster: id: name: replex: token: metrics: provider: instana instana: baseUrl: apiToken: ``` 4. Installation with Helm ```bash helm install replex/replex-k8s-agent --namespace replex-k8s-agent -f values.yaml ``` Where `release-name` is any arbitrary string to identify the Helm installation e.g. `replex-agent`. 5. Installation without Helm You will still need Helm to render the template locally. However, the installation itself will be done with `kubectl` directly, not with Helm: ```bash helm template replex replex/replex-k8s-agent --namespace replex-k8s-agent -f values.yaml | kubectl apply -f - ``` After these steps the agent will start sending data to the pushgateway. ### Updating To update the agent, follow these steps: 1. Update the Helm repo. ``` helm repo update ``` 2. Upgrade the replex release. ``` helm upgrade replex/replex-k8s-agent --namespace -f values.yaml ``` ### Logging The replex-kubernetes-agent runs as a pod in the namespace it was installed in and its logs can be consulted using kubectl. For example, `kubectl -n replex-k8s-agent logs ` ### Retry policy Replex agent has a retry policy when one of metric push fails. By default, retry policy is turned on, so if one of metric pushes to Pushgateway fails, it will be stored on disk in `persistentVolume.mountPath` dir. If `retry.diskCache` is false, metrics will be stored in agents memory. `retry.intervalSeconds` stands for the interval when agent will try to resend metrics from the cache, by default it is 5 minutes (300s). To set up retry policy next three variables can be used: * (Optional) **retry.intervalSeconds**: Interval between a retry to push metrics that failed previously. * (Optional) **retry.diskCache**: Cache failed metrics on disk. * (Optional) **persistentVolume.mountPath**: If `retry.diskCache` is true, path to store metrics. ### Chart Values | Key | Type | Default | Description | | --------------------------------- | ------ | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | cloudProviderOverride | string | `""` | force agent to use specified cloud provider | | cluster.id | string | `""` | Unique cluster identifier | | cluster.name | string | `""` | Cluster name displayed in dashboard | | datadog.apiKey | string | `""` | Your Datadog API Key | | datadog.applicationKey | string | `""` | Your Datadog Application Key | | datadog.site | string | `"com"` | Either `com` if you are on Datadog US site or `eu` if you are on Datadog EU site. | | extraInitContainers | object | `{}` | Add init containers to replex agent container | | extraVolumeMounts | object | `{}` | Add extra volumeMounts to replex agent container | | extraVolumes | list | `[]` | Add extra volumes to replex agent container | | image.pullPolicy | string | `"Always"` | Image pull policy | | image.repository | string | `"registry.replex.io/public/replex-k8s-agent"` | Image repository | | image.tag | string | `""` | Override AppVersion with specific tag from image repository | | instana.apiToken | string | `""` | Valid Instana API token | | instana.baseUrl | string | `""` | Base URL of a tenant unit, of form: | | kubernetesInfoProvider | string | `"kubernetes"` | Specify Kubernetes info provider. Options are \`kubernetes' or 'instana' | | metrics.filesystem | string | `"cadvisor"` | Either `cadvisor` for using cAdvisor filesystem metrics or `csi` if using CSI drivers. `csi` requires . | | metrics.provider | string | `""` | Metrics provider to use. | | nodeSelector | object | `{}` | Pod node selector Key-Value pair | | onlyUseReadyNodes | bool | `false` | Consider nodes without 'Ready' status as not running at all. | | persistentVolume.mountPath | string | `"/data/metrics"` | Persistent volume mount path | | persistentVolume.size | string | `"10Gi"` | Size of agent persistent volume | | persistentVolume.storageClassName | string | `""` | name of the storage class to configure the PVC on | | prometheus.bearerToken | string | `""` | Bearer token for prometheus server requests authentication. | | prometheus.containerLabel | string | `"container"` | The label representing "container name" on prometheus time series. | | prometheus.nodeLabel | string | `"node"` | The label representing "node name" on prometheus time series. | | prometheus.podLabel | string | `"pod"` | The label representing "pod name" on prometheus time series. | | prometheus.url | string | `""` | URL of the Prometheus instance | | pushgateway.url | string | `""` | Full URL to the replex pushgateway. Format: (Note the `/push` path). | | replex.token | string | `""` | Agent authentication token. | | resources.requests.cpu | string | `"50m"` | Specify the cpu units requests of the agent container | | resources.requests.memory | string | `"100Mi"` | Specify the memory bytes requests of the agent container | | retry.diskCache | bool | `true` | | | retry.intervalSeconds | int | `300` | | | securityContext | object | `{}` | Deployment security context | | sslCertificate | string | `""` | SSL certificate string. Can be used to add a custom ssl certificate to the agent. | | stackdriver.projectId | string | `""` | Your GCP project ID | | tokenSecret.create | bool | `true` | Set to 'false' to skip creation of Secret object | | tokenSecret.key | string | `""` | Override key in token Secret object (for custom secret keys) | | tokenSecret.name | string | `""` | Override name of token Secret object | | tolerations | list | `[]` | array of tolerations for the pod scheduling | | useControlPlaneCost | string | `""` | (bool) Track costs of the Kubernetes Control Plane. | # Getting Started This section describes the various dashboards and views in the Replex UI. Detailed descriptions are provided for the following dashboards: ### [Cluster Dashboard](/getting-started/cluster-dashboard) ### [Teams Dashboard](/getting-started/teams-dashboard) ### [Namespace Dashboard](/getting-started/namespace-dashboard) ### [Node Dashboard](/getting-started/node-dashboard) ### [Settings Page](/getting-started/settings-page) # Cluster Dashboard The **Cluster Dashboard** provides a number of views, ranging from an overview of all connected Kubernetes clusters to detailed views outlining metrics for individual clusters as well as the Kubernetes artefacts belonging to those clusters. Below are some of the views provided: ### Kubernetes Custer Overview The **Kubernetes Cluster Overview** provides overall cluster metrics for all connected clusters across cloud providers and on-premise. The metrics provided include **Cluster Score, Next 30 Days Forecast** and **Total Cost.** These metrics are provided for the time frame chosen using the date picker in the top right. ![Kubernetes Cluster Overview](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Ma9LUE-KlehtlMsAhbF%2F-Ma9M42rCBMVEDUUnQnf%2FKubernetes-Clusters-Replex%20\(1\).png?alt=media\&token=9852af81-cbe8-4cd0-bf6a-800f3b83f604) **Cluster Score** is a measure of cluster utilization. The following utilization tiers are applied when calculating the cluster score: **0-30 = low** **31-60 = medium** **61-100 = high** **Next 30 Days Forecast** is based on the costs for the previous 7 days. The forecast is displayed only when the date range chosen includes the previous 7 days. Date range can be chosen using the date picker in the top right. **Total Cost** represents the total costs of that cluster for the date range chosen. The **Cluster Overview** also outlines the number of nodes currently running in that cluster, the total number of unique nodes that belonged to that cluster in the timeframe chosen as well as when the metrics were last updated. ![Kubernetes Cluster Overview](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Ma9JtgA3lGLl4vDYTyl%2F-Ma9Ksh5F3nOCf5aTDxO%2FKubernetes-Clusters-Replex.png?alt=media\&token=fcc57885-d604-4cc0-a76b-79b0ae533922) In the screenshot above 3 nodes are currently running in the **replex** cluster while 9 unique nodes have belonged to the cluster during the timeframe chosen. ### Detailed Cluster View The **Detailed Cluster View** can be accessed by clicking on one of the listed clusters in the [**Cluster Overview**](https://docs.replex.io/getting-started/cluster-dashboard#kubernetes-custer-overview). The **Detailed Cluster View** outlines **Total Monthly Cost**, **Idle Cost**, **Savings** and **Efficiency** metrics for individual Kubernetes clusters. **Efficiency** metrics are provided for all three cluster resources including **CPU**, **RAM** and **Disk** and are color coded to provide an indication of **Used Resources** and **Idle Resources**. The **Detailed Cluster View** also provides a cost progression chart of **Total Monthly Cluster Costs** for the previous 6 months. ![Detailed Cluster View](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MRzmFq7rYg3CW4qkq7W%2F-MRzmYw8UY9jg2aQaT6r%2Fk8s_31_replex1-Replex.png?alt=media\&token=325382ca-b61a-4b81-ba89-8f77497cf15a) The **Widgets** at the bottom of the view provide cost allocation metrics for all **Namespaces** and **Deployments** in that cluster for the last 24 hours. The **Namespaces Widget** outlines costs for all Namespaces in the cluster as a percentage of total cluster total. Similarly, the **Deployments Widget** outlines costs for all Deployments in the cluster as a percentage of total cluster cluster costs. ![Cost Allocation Widgets](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MS7wDYAfh5g8RjuKmEk%2F-MS83k2pQo4z-aq-iymL%2Fk8s_50_replex1-Replex.png?alt=media\&token=5504bc71-86d2-4e16-9f29-7f925ab9f1e4) ### Namespaces Overview The **Namespaces Overview** outlines cost metrics for all Namespaces belonging to that cluster. It can be accessed by clicking on the Namespaces Widget in the [**Detailed Cluster View**](https://docs.replex.io/getting-started/cluster-dashboard#detailed-cluster-view). Cost metrics are provided as two charts: **Allocated Cost Combined** and **Allocated Cost Detailed**. **Allocated Cost Combined** represents the progression of total costs for all **Namespaces** in the cluster across the entire duration chosen. The **Allocated Cost Detailed** chart dynamically groups total costs for all **Namespaces** on a weekly, daily or hourly basis, based on the duration chosen. The individual bars representing total namespace costs are broken down into **RAM**, **CPU,** **PVC** and **Node Disk Costs**, for a more informative and easy-to-understand cost analysis. ![Total Namespace Costs](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdMlg70_fEku0vTDzA5%2F-MdMmIJk4NLKrTv9uznR%2Freplex-1-Namespaces-Replex%20\(1\).png?alt=media\&token=8d93191d-c8cb-4dac-9ba0-8570516e8f64) The **Cost Details** section of the **Namespaces Overview** provides detailed cost metrics for individual **Namespaces** in the cluster. The metrics provided include the **Utilized Cost** for all three resources (CPU, RAM and Storage) in that particular namespace, the **Idle Cost** of that **Namespace** (both as an absolute $ number and a percentage), [**External Cost**](/settings/integrate-out-of-cluster-costs-cloud-costs) and the **Total Cost.** These metrics are outlined in the **Summaries** tab of the **Cost Details** section. ![Cost Details Section - Summaries Tab](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdMmvO_CFgdkSpbOcR8%2F-MdMnMqBTlya09533Zav%2Freplex-1-Namespaces-Replex%20\(2\).png?alt=media\&token=ad8aa6f3-ecc1-4785-80ff-b536adc53d48) The **Cost Development** tab of the **Cost Details** section charts cost progression for each namespace individually across the duration chosen. Namespace charts can be sorted **Alphabetically**, **Total Cost, Idle Cost** or by **Variance.** ![Cost Details Section - Cost Development Tab ](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdMnUW9n2rOTXZSiFUN%2F-MdMo3Anvd9_EgjGs6yn%2Freplex-1-Namespaces-Replex%20\(3\).png?alt=media\&token=82ab7d9b-aff5-41b9-9e10-72706a29b4f3) Sorting by **Total Cost** will organize namespace charts, showing the namespace with the highest cost first. ![Sorting Namespaces by Total Cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdMoHKmZ3O4yjJnwBHI%2F-MdMoouV0uwaFC-ZbnNT%2Freplex-1-Namespaces-Replex%20\(4\).png?alt=media\&token=20bb82a3-14b5-4a25-9df4-2410aa2d70fb) Sorting by **Idle Cost** will organize namespace charts, showing the namespace with the highest change in cost first. ![Sorting Namespaces by Idle Cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdMoHKmZ3O4yjJnwBHI%2F-MdMpONTCo5a9-pN7EJJ%2Freplex-1-Namespaces-Replex%20\(5\).png?alt=media\&token=a4ebc098-63aa-4283-aa3e-f2e0a5edb000) Sorting by **Variance** will organize namespace charts, showing the namespace with the highest change in cost first. ![Sorting Namespaces by Variance](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdMnUW9n2rOTXZSiFUN%2F-MdMo3Anvd9_EgjGs6yn%2Freplex-1-Namespaces-Replex%20\(3\).png?alt=media\&token=82ab7d9b-aff5-41b9-9e10-72706a29b4f3) #### Namespace Overview Filters The **Namespace Overview** can be filtered using the **+** **Add Filter** button in the top left of the screen. **+** **Add Filter** allows Namespaces to be filtered using either **Namespace Name** or **Kubernetes Labels** with unique key/value combinations. The Namespace Overview can also be filtered using a combination of both **Namespace Name** and **Kubernetes Labels**. #### Filter by Namespace Name To filter the **Namespace Overview** by Namespace Name, click on **+** **Add Filter**, input the name of the namespace you want to filter by in the **Filter by namespaces** field or choose one from the drop down. This will filter the Namespace Overview to display cost metrics for the namespace chosen: ![Filter Namespace Overview by Namespace Name](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdMrOxMgt-jrRAvj7Ng%2F-MdMsXeg4EU74gf2nFpX%2Freplex-1-Namespaces-Replex%20\(6\).png?alt=media\&token=64a108ac-45ce-47fc-88e1-8d06f374f5dd) #### Filter by Label (Key/Value Combination) To filter the **Namespace Overview** by Label, click **+ Add Filter**, input both the **Key** and **Value** of a Kubernetes Label in the **Filter by labels** field or choose one from the drop down. This will filter the Namespace Overview to display cost metrics for that specific key/value combination. ![Filter Namespace Overview by Label](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdMt4KPHgZ3v6TAqCPS%2F-MdMtgGisWMoKkqcj06L%2Freplex-1-Namespaces-Replex%20\(7\).png?alt=media\&token=778321db-795c-4389-8c63-93bb068ac84a) #### Filter by Namepace Name and Label Key The **Namespace Overview** can also be filtered using a combination of **Namespace Name** and **Labels**. To do this click **+ Add Filter**, input both a **Namespace Name** and **Key Value** combination in the appropriate fields or choose from the drop down. This will filter the Namespace Overview to display cost metrics for that specific Namespace Name/Label combination. ![Filter Namespace Overview by Namespace Name and Label](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdMuC3fl8Tl9ZqhTe90%2F-MdMuuua2pjaW1o5cyxk%2Freplex-1-Namespaces-Replex%20\(8\).png?alt=media\&token=9903108f-ad82-42cb-a82a-f2187481af2e) #### **Filter by Label Key** The **Namespace Overview** can also be filtered using only Label Keys. ‌To do this, click **Group by Label** and choose a Label Key in the **Select label to group by** pop-up. This will filter the Namespace Overview to display cost metrics for that specific Label Key. ![Filter Namespace Overview by Label Key](https://lh3.googleusercontent.com/H5RVwxTnhM9tayigO6Ds19jiRPBBK6_zZD5SEyUk8rdD3u7g88tQbFhZwF_R7AIEUjxhBfx62WPCr7k7nINroryZE7KDughaGGO59D29db4Zd_5i7mlyr_Wb4ILDhZvjZzNad6d1) ### **‌**Other Overviews #### **Overviews for Native Kubernetes Artefacts** **Overviews** similar to the one for Namespaces are provided for other Kubernetes artefacts including **Deployments**, **StatefulSets**, **DaemonSets**, **Services** and **Jobs**. #### Deployments Overview ![Deployments Overview](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdN-hNNqlABE-qgbFNm%2F-MdN4-awcaPkpqTRk33y%2Freplex-1-Deployments-Replex.png?alt=media\&token=825b59e6-f17a-42d1-a26a-fef346e2184e) #### StatefulSets Overview ![Statefulsets Overview](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdN45BUYod0VfIOGAvP%2F-MdN4s_MKshFPXc1g7YR%2Freplex-1-StatefulSets-Replex.png?alt=media\&token=e207ba51-3b1c-4eb9-b700-47618ecf07c4) #### Services Overview ![Services Overview](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdN45BUYod0VfIOGAvP%2F-MdN5O5qWVOupfqIJl5p%2Freplex-1-Services-Replex.png?alt=media\&token=929b99a3-486e-44e4-a6ae-b8ac331cef24) #### **Overviews for Custom Resource Groupings** Overviews are also provided for **Custom Resource Groupings** including **Team** and **App**. These overviews can be accessed in the tabs list at the top, and are hardcoded in the Replex UI. For example the **App Overview** lists metrics for all Kubernetes artefacts with the Label Key **App.** ![App Overview](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdN5d2-C6zxwqJ2MqkD%2F-MdN67Snvd1mBKoiTEkO%2Freplex-1-App-Replex.png?alt=media\&token=7c568ee2-a6f7-4363-9e0e-1bf7992c6226) Similarly, the **Team Overview**, outlines metrics for all Kubernetes artefacts with the Label Key **Team**. ![Team Overview](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdN5d2-C6zxwqJ2MqkD%2F-MdN6W8uljvh_fDcoA4t%2Freplex-1-Team-Replex.png?alt=media\&token=48a3751b-efee-4e93-93fc-d90e6937b934) ### Detailed Namespace View The **Detailed Namespace View** provides cost metrics for an individual **Namespace** as well as for the Kubernetes artefacts running in it, including **Jobs**, **Deployments** and **Services** etc. The **Detailed Namespace View** can be accessed by clicking on a Namespace in the [Namespace Overview](https://docs.replex.io/getting-started/cluster-dashboard#namespaces-overview). ![Detailed Namespace View](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdN8jl66Rc_BQn1ebwp%2F-MdNAc34DV1bGybNRfWw%2Freplex-1-Namespaces-Replex-Replex.png?alt=media\&token=4762b003-7312-4884-9437-100b013eb2b9) The **Cost Overview** section of the **Detailed Namespace View** outlines the **Total Cost** of that namespace as well as the **Utilized Cost** for **RAM**, **CPU** and **Storage**. It also provides the **Idle Cost** of that Namespace both as an absolute $ number as well as a percentage. Finally, the **Cost Overview** section also outlines the **External Cost** of that Namespace. ![Cost Overview Section](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR_I60Oj4tsClAxb3OC%2F-MR_IHjPUHxcoQpk_mmw%2Fk8s_11_replex-1-Namespaces-Replex-Replex.png?alt=media\&token=1f079c72-0b39-40cb-813b-4450cbde0ae9) The **Detailed Namespace View** also provides a cost breakdown for individual out of cluster resources allocated to the Namespace. It outlines both the **Resource Type** as well as its cost for the time period chosen**.** Cost breakdowns for out of cluster resources can be viewed by clicking on **Show External Resources.** ![Cost Breakdown for External Resources](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdRAENPv8NUKQej6W-0%2F-MdRCrtZ0nJIsKiCsc2X%2Freplex-1-Namespaces-Replex-Replex%20\(2\).png?alt=media\&token=8ac8dbde-2c33-43c7-b227-bb0b203936a2) Next, on the **Detailed Namespace View** are two charts: **Allocated Cost Combined** and **Allocated Cost Detailed**. **Allocated Cost Combined** is a cost progression chart for that **Namespace** for the entire duration chosen. The **Allocated Cost Detailed** chart dynamically groups the costs of that **Namespace** on a weekly, daily or hourly basis, based on the duration chosen. ![Allocated Cost Combined and Allocated Cost Detailed](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdWpAWS5JbP755eQkYn%2F-MdWxEM_sp5-0aZ2lT0V%2Freplex-1-Namespaces-Replex-Replex%20\(3\).png?alt=media\&token=d739db69-f51e-4c40-aedb-b2b786499629) The **Cost Details** section of the **Detailed** **Namespaces View** provides detailed cost metrics for the Kubernetes artefacts (Deployments, Services, Jobs, StatefulSets etc) running in the Namespace. The metrics provided include the Utilized cost for **RAM**, **CPU** and **Storage**, the **Total Cost** of the artefact as well as the **Idle Cost**. Idle Cost is outlined both as an absolute $ number and a percentage. These metrics are outlined in the **Summaries** tab of the **Cost Details** section. Below is a screenshot with detailed cost metrics for all **Deployments** in the Namespace. ![Cost Metrics for Deployments](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MReRgcLhsGeQtdJWX_H%2F-MReScT6rB7h4uBQt136%2Fk8s_14_replex-1-Namespaces-Replex-Replex.png?alt=media\&token=425e2424-a08a-4b0c-9f26-df983a950004) Cost metrics of other Kubernetes artefacts (Services, Jobs, StatefulSets) running in the **Namespace**, can be viewed by selecting one from the drop down in the top right of the **Cost Details** section. The screenshot below shows detailed cost metrics for all **Services** running in the Namespace. ![Cost Metrics for Services](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MReUkebXYo8c4ZkBCnu%2F-MReUn7iPqZMF6_D7gRC%2Fk8s_15_replex-1-Namespaces-Replex-Replex.png?alt=media\&token=2e4cd724-d72c-4492-809b-430346d766ab) The **Cost Development** tab of the **Cost Details** section charts cost progression for Kubernetes artefacts running in the **Namespace** for the duration chosen. Below are the cost progression charts for all **Services** running in the Namespace: ![Cost Progression for all Services running in the Namespace](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MReVYA12cLcyyH8laYp%2F-MReW7hKQ4CXMMsx4KHz%2Fk8s_16_replex-1-Namespaces-Replex-Replex.png?alt=media\&token=ee05a6e6-8976-42bf-945d-feed1201574b) And for all **Deployments** running in the Namespace: ![Cost Progression for all Deployments running in the Namespace](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MReWOvueBKwgxwjJjtI%2F-MReWrsXoRM7LLcZoCAz%2Fk8s_17_replex-1-Namespaces-Replex-Replex.png?alt=media\&token=b2ff0724-59ac-4a5b-ae0c-ecd43444c88b) All cost progression charts can be sorted **Alphabetically**, by **Total Cost** , **Idle Cost** or **Variance.** ### Detailed Deployments View The **Detailed Deployments View** outlines cost metrics for an individual **Deployment** as well as the **Pods** running in it. ![Detailed Namespace View](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR_vMzg3DskYyj6m83X%2F-MR_weQWAQxhJ_Ro93m9%2Fk8s_11_replex-1-Namespaces-Replex-Replex.png?alt=media\&token=7b4570d4-9f37-4227-accd-bfc8ecbb6652) The first three sections of the **Detailed Deployment View** are similar to the **Detailed Namespace View.** These include the **Cost Overview** section - which outlines the **Utilized Cost** for **CPU**, **RAM** and **Storage**, the **Idle Cost** and the **Total Cost** of the Deployment - the **Allocated Cost Combined** and the **Allocated Cost Detailed** sections. ![Cost Overview Section](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdX6g7dxUAQ6-ZHWbyt%2F-MdX7GVhZGqI_XMeAQOP%2Freplex-1-Deployments-Pushgateway-Replex.png?alt=media\&token=2099bb4a-486a-40a9-b548-cdaca65a03a1) ![Allocated Cost Combined and Allocated Cost Detailed](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdX7L5JysUTs5k0sB7c%2F-MdX7XG9PMN3qN8zIDOc%2Freplex-1-Deployments-Pushgateway-Replex%20\(1\).png?alt=media\&token=0d4b3a8a-4d24-47fc-87c5-f07578747fba) The **Cost Details** section of the **Detailed** **Namespaces View** is replaced by the **Pod** section in the **Detailed Deployments View**. This section outlines metrics for all **Pods** running in the **Deployment**. The metrics provided for each **Pod** include **Requests**, **Usage** and **Cost metrics** for **CPU** and **RAM** resources, **Capacity** and **Cost** metrics for **Storage** resources and the **Total Cost**. ![Pod Metrics](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MS2zDttViLpXlq8-LTy%2F-MS2zXr1ENpKVRAF_oPU%2Fk8s_42_replex1-DaemonSets-Instana-agent-Replex.png?alt=media\&token=4fcedd44-50a4-4a77-8453-1cf5730437dd) Inactive Pods are grayed out, as can be seen in the screenshot above. **Pod Metrics** for CPU, RAM and Storage are also color-coded for intuitive and easy to understand utilization information. Pod Metrics are color coded as Red whenever the utilization falls in the following ranges: For **CPU**: * Below 60% and above 95% For **RAM**: * Below 60% and above 100% For **Storage**: * Above 95% ### Other Detailed Views **Detailed Views for Native Kubernetes Artefacts** **Detailed Views** similar to the **Detailed Deployments View** are provided for other Kubernetes artefacts including for **StatefulSets**, **DaemonSets**, **Services** and **Jobs**. **Detailed Services View:** ![Detailed Services View](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdX7s6XwP1JRfWsYbDD%2F-MdX8aNcloD6LRAR9yDA%2Freplex-1-Services-Postgres-Replex%20\(1\).png?alt=media\&token=ff6a150d-9cfb-4c55-a78c-be9c1060a5f9) **Detailed StatefulSets View:** ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdXEGmIJfMAoml8ikZB%2F-MdXElXrEGBochjvxY0T%2Freplex-1-StatefulSets-Postgres-Replex.png?alt=media\&token=19faddcf-8615-4915-b90d-ff438d9e4e12) #### Detailed DaemonSets View ![Detailed DaemonSets View](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdXEGmIJfMAoml8ikZB%2F-MdXZ79_RCHZGBZ7WuVn%2Freplex-1-DaemonSets-Instana-agent-Replex.png?alt=media\&token=609c8765-93d6-4578-a3dd-43cf70a8ff2f) #### **Detailed Views for Custom Resource Groupings** Detailed Views are also provided for **Custom Resource Groupings** including **Team** and **App**. These detailed views can be accessed by clicking on an individual custom resource from the **Team** or **App** **Overview**. Following is the **Detailed Team View**: ![Detailed Team View](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdXZDnZwu2DWGukkz-V%2F-MdX_VFWr4zuMrvxxGeP%2Freplex-1-Team-Instana-Replex.png?alt=media\&token=a88ea3ce-1b41-458c-a825-707937eb1a84) And the **Detailed App View:** ![Detailed App View](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Md_qBvW-kfGGNXFzhd4%2F-MdaMUR8rGiI7Kb5bp8m%2Freplex-1-App-Postgres-Replex.png?alt=media\&token=aa1b389f-691d-402f-981b-0e0aaf777325) The **Cost Details** sections of the **Detailed** **Team and App Views** outline cost metrics for all Kubernetes artefacts (Namespaces, Deployments, Services, Jobs, StatefulSets etc) belonging to that **Team** or **App**. Cost metrics of these other Kubernetes artefacts can be viewed by selecting one from the drop down in the top right of the **Cost Details** section. # Teams Dashboard The **Teams Dashboard** provides detailed cost and resource metrics for each **Team** with access to the cluster. The metrics provided include the number of the following resources owned by each team: * **Namespaces** * **Pods** The **Teams Dashboard** also outlines the **Cost for Last 30 Days** incurred by each team. **Cost for Last 30 Days** represents the cost of all Kubernetes resources owned by the team for the last 30 days. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MlEzQnJ_5U1s6UV95TT%2F-MlF1TKv1NgvUXPANhjD%2FTeams-Replex.png?alt=media\&token=0df7e4ee-dc89-4a94-9fdb-131a9d6b860e) ### Detailed Team View Clicking on a team name will take you to the **Detailed Team View**. The **Detailed Team View** outlines the **Idle** **Cost**, **Utilized** **Cost, External Cost** and **Total** **Cost** of the team along-with the total number of team **Members** and **Pods** owned. These metrics are provided for the time frame chosen using the date picker in the top right. The **Allocated Cost** chart graphs total costs for that team across the entire duration chosen. ![Individual Team View](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MYjqDl4nlTYW8AUtVn1%2F-MYjuMnfdW5BziPl6X0J%2FReplex-teams-k8s-cost-allocation-chargeback.png?alt=media\&token=63e0a075-c635-48a6-aae3-07fb13af82dc) The **Detailed Team View** also provides information about the **Clusters**, **Namespaces** and **Labels** owned by the team. ![Namespaces Owned by Team](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Mdq6C5WSGDkXK0nOfch%2F-MdqSAEdlJX_eJVYkxt4%2FEmmanuel-s-Test-Team-Replex.png?alt=media\&token=a57183af-a0a0-411a-bd58-575c882df7a7) ![Labels owned by Team](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Mdq6C5WSGDkXK0nOfch%2F-MdqSrCp2nrW4nD7edLK%2FEmmanuel-s-Test-Team-Replex%20\(1\).png?alt=media\&token=291e9de5-1030-4fe3-8408-7e0dde3777f1) Users can also review the **Budgets** and **External Costs** of that **Team** by clicking on the **Budgets** or **External** tabs respectively. ![Budgets Tab](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MYjqDl4nlTYW8AUtVn1%2F-MYjveKxsLKDKoU7uLFx%2FReplex-teams-budgets.png?alt=media\&token=c07e9768-9f09-4c67-9aad-02af3f6a717c) The **External** tab provides a breakdown of external resources allocated to that team by **Type** and **Cost**. ![External Tab](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MYjqDl4nlTYW8AUtVn1%2F-MYjwU9QGSyipaUA5Duq%2FReplex-teams-external-cloud-costs-out-of-cluster-costs.png?alt=media\&token=73c905cb-2c37-44f5-bd0c-128515ecc861) ### Add New Teams Cluster Admins can also create new teams using the **Teams Dashboard** . To create a new team, click on ***Create Team +*** in the top right corner of the dashboard. A new window will pop up, asking you to enter the name of the team. ![Create a new team](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFeKIJYpWEkmT5Rpa6g%2F-MFeKkWWgWP5XQ7PIDfI%2FScreenshot%202020-08-26%20at%2012.35.54.png?alt=media\&token=e57cb4f5-5db2-4191-9001-cd9032cb6a59) Enter a team name and click ***Create Team***. This will create a new team, however there are no team members or resources in this newly created team. ![Setup Team](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFeKIJYpWEkmT5Rpa6g%2F-MFeLAxPFkEF3LTvmSVU%2FScreenshot%202020-08-26%20at%2012.37.51.png?alt=media\&token=72a96a04-2570-4f68-91ee-c2f0ce2bafc2) To add resources and members to the newly created team click on ***Setup Team.*** The next screen allows you to add members to the newly created team. Use the search box to search for members that are already part of your organisation on Replex, tick the checkbox next to their names and click on ***Next Step*** to proceed. ![Add members to team](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFeKIJYpWEkmT5Rpa6g%2F-MFeNJv4KWDY2cXwmybH%2FScreenshot%202020-08-26%20at%2012.46.44.png?alt=media\&token=65d6f0a5-2577-44e5-8a15-eb88265ad3ae) On the next screen you can add Kubernetes resources to the newly created team. Three types of resources can be added to the team: * **Clusters** * **Namespaces** * **Labels** Choose the clusters, Namespaces and Labels that you want to add to the team, using the search box below each resource. ![Add resources to team](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFeKIJYpWEkmT5Rpa6g%2F-MFeOn7xGYUVQWckNvI7%2FScreenshot%202020-08-26%20at%2012.53.43.png?alt=media\&token=06c7c56e-81c0-462e-b80c-b3cdcb21ee63) Click on ***Finish Setup***. This will create the new team and will also add the members and Kubernetes resources you chose to it. The newly created team will show up on the **Teams Dashboard** where you can view the Kubernetes resources associated with it as well as its **Total Monthly Cost**. ### Edit Team Admins can edit **Teams** by clicking on the arrow next to the team name from the **Detailed Team View** and choosing **Edit Team.** ![Edit Team](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MS33Dp_dJaj-eB5mZfK%2F-MS34KZBABEKBKjriFSx%2FScreenshot%202021-01-27%20at%2014.41.34.png?alt=media\&token=18953b79-46f0-4d91-81c2-b66a30604ec2) The next screen allows Admins to edit the Team Name, change privacy settings for the team and add or remove members. ### Manage Team Resources To add or remove resources owned by a team click on the arrow next to the team name from the **Detailed Team View** and choose **Manage Resources.** On the next screen, Admins can add or remove resources (Clusters, Namespaces, Labels) associated with a Team. ### Remove Team Admins can also remove teams completely from the edit team screen by clicking on **Delete Team** in the top right. # Namespace Dashboard The **Namespace Dashboard** provides detailed cost metrics for each Namespace. The cost metrics provided for each Namespace include the **Utilized Cost** for each individual Kubernetes resource in the namespace (CPU, RAM, Storage), the **Idle Cost** (both as an absolute $ figure as well as a percentage) and the **Total Cost** of the Namespace. The Namespace Dashboard also outlines the **External Cost** for each namespace. Each metric is provided for a specific timeframe which can be chosen in the date picker in the top right of the screen. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdRax-ttpGl14d1Xash%2F-MdRbrK9GYB31lTDnJpc%2FKubernetes-Namespaces-Replex%20\(1\).png?alt=media\&token=2b39d4bf-98a5-4d78-9335-d2745770dc1f) ### Namespace Dashboard Filters The namespace dashboard can be filtered using the **Filter by clusters or namespaces** field in the top left of the screen. The following filters can be applied to the namespace dashboard: * Display metrics for individual namespaces or groups of namespaces * Display metrics for namespaces belonging to an individual cluster or group of clusters * And a combination of both ![Namespace Dashboard Filters](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdRax-ttpGl14d1Xash%2F-MdRglattP3MzfVWEM0j%2FKubernetes-Namespaces-Replex%20\(2\).png?alt=media\&token=f75a80d5-68a6-4809-9d3a-ceeb367fb548) ### Cross-Cluster vs Single Cluster Namespaces Most Kubernetes environments have multiple clusters with namespaces spanning across clusters or simply with the same naming convention. Cross-cluster namespaces can be differentiated by the arrow preceding the namespace name. Single cluster namespaces do not have the arrow preceding it. In the screenshot below, replex and replex-ns are cross-cluster namespaces, while dev\_namespace is a single cluster namespace. ![Cross-Cluster and Single Cluster Namespaces](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdRm5_vH-4S29fW1FUT%2F-MdRnbdNChVJugLpeOlb%2FKubernetes-Namespaces-Replex%20\(4\).png?alt=media\&token=7bb507d3-51a7-48bc-9051-ad694443acd5) In case a **Namespace** spans across multiple clusters, Replex will aggregate the cost metrics for all the resources that belong to that namespace in all clusters. Per cluster metrics for that namespace can be viewed by clicking on the arrow preceding each namespace name. ![Cross Cluster Namespaces](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdRh-crm4YvKXP7hmNc%2F-MdRhgbx8GJ52helxXu3%2FKubernetes-Namespaces-Replex%20\(3\).png?alt=media\&token=272a8f01-9823-4195-bbbc-3e335cee4c22) The screenshot above outlines cost metrics for the **replex** namespace across all clusters including **replex-1**, **replex-gke** and **replex- eks** etc. ### Detailed Namespace View Clicking on a single cluster namespace will take you to the [**Detailed Namespace View**](https://docs.replex.io/getting-started/cluster-dashboard#detailed-namespace-view). Read more about the charts and metrics provided on the detailed namespace view [here](https://docs.replex.io/getting-started/cluster-dashboard#detailed-namespace-view). ### Detailed Cross-Cluster Namespace View Clicking on a cross-cluster namespace will take you to the **Detailed Cross Cluster Namespace View.** The **Detailed Cross-Cluster Namespace View** outlines aggregated metics for all instances of that namespace across all clusters. ![Detailed Cross-Cluster Namespace View](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdRqWbYZG90z-1FRM_v%2F-MdRqzKio-3UH5xRoTDv%2FNamespaces-Replex-Replex.png?alt=media\&token=1e10ecf4-fa40-43f2-9aca-fff43c9518d0) The **Cost Overview** section of the **Detailed Cross-Cluster Namespace View** outlines the aggregated **Total Cost** of that namespace as well as the **Utilized Cost** for **RAM**, **CPU** and **Storage** for all instances of that namespace. It also provides the **Idle Cost** of that cross-cluster namespace both as an absolute $ number as well as a percentage. Finally, the **Cost Overview** section also outlines the **External Cost** of that cross-cluster namespace. ![Cost Overview](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdRqWbYZG90z-1FRM_v%2F-MdRrEdZv2rUJP8HKvSv%2FNamespaces-Replex-Replex%20\(1\).png?alt=media\&token=840c3b72-3ed6-44fd-be0c-41a1d51380bc) The **Detailed Cross-Cluster Namespace View** also provides a cost breakdown for individual out of cluster resources allocated to the cross-cluster namespace. It outlines both the **Resource Type** as well as its cost for the time period chosen**.** Cost breakdowns for external resources (out of cluster resources) can be viewed by clicking on **Show External Resources.** ![Cost Breakdown for External Resources](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdRrJsHPBrv2ZAmKy4u%2F-MdRriqOCKD_QQemJSRk%2FNamespaces-Replex-Replex%20\(2\).png?alt=media\&token=7e9d9bfd-46ce-46ce-890a-58293c547829) In the section below the **Cost Overview**, the **Detailed Cross-Cluster Namespace View** outlines all instances of the namespace across all clusters as well as the cost metrics associated with each instance. ![Cost Metrics for each Cross-Cluster Namespace Instance](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdRrsZJ8JI-wEEExPgc%2F-MdRsU1K_Q28fvaY3bXH%2FNamespaces-Replex-Replex%20\(3\).png?alt=media\&token=905f47e2-c523-406b-aa9e-03806582bff3) Next, on the **Detailed Cross-Cluster Namespace View** are two charts: **Allocated Cost Combined** and **Allocated Cost Detailed**. **Allocated Cost Combined** is a cost progression chart for that instances of that namespace for the entire duration chosen. The **Allocated Cost Detailed** chart dynamically groups the costs of the cross-cluster namespace on a weekly, daily or hourly basis, based on the duration chosen. The individual bars representing total cross-cluster namespace costs are broken down into **RAM**, **CPU,** **PVC** and **Node Disk Costs**, for a more informative and easy-to-understand cost analysis. ![Allocated Cost Combined and Allocated Cost Detailed](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdRtZRab42YVvUCtTOr%2F-MdRtamBZNlb4ILrC4_l%2FNamespaces-Replex-Replex%20\(4\).png?alt=media\&token=95ad68e5-bfa0-433e-86c3-ccc781c1d168) # Node Dashboard The **Node Dashboard** outlines detailed **Cost** and **Utilization** metrics for each node across all connected clusters. Utilization metrics are provided for each node resource (CPU and RAM), as well as the number of CPU cores and RAM capacity. The **Node Dashboard** also outlines the node/instance type of the **Node**, the cluster it belongs to as well as its **Total Cost**. The **Node Dashboard** is populated automatically once the [**Replex Kubernetes** **Agent**](/) is installed in a cluster. ![Node Dashboard](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdpzmGvXvIPpTuRjsLr%2F-Mdq2RWLehyDO82CXiOG%2FKubernetes-Nodes-Replex.png?alt=media\&token=5ee6ca1c-c5b5-4416-8a09-97935f4e5c24) # Settings Page The **Settings Page** has two main sections: **General Settings** and **User Settings**. **User Settings** allow users to edit and update personal details including **First** and **Last Name** and the **email** associated with their account. Users can also update their passwords. ![User Settings](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFjCbDSSCWo6ae3JT3h%2F-MFjDVl3OXReb_FeZgrL%2FUser-Profile-Replex.png?alt=media\&token=c03a1508-c634-4ae7-8db6-d923040377ee) **General Settings** allow account admins to add new users or admins, keep track of account invitations, add billing credentials, configure authentication, add custom cost models and generate API tokens. For a detailed review of each functionality, check out the [Settings](/settings) section of this documentation. # Concepts / Calculations This section describes the concepts and calculations that are a part of the Replex UI. ## Calculations Descriptions are provided for the following calculations: ### [Forecasted cost](/concepts/calculations) ### [Cost previous 30 days](/concepts/calculations) ### [Total cost](/concepts/calculations) ### [Idle cost](/concepts/calculations) ### [Savings](/concepts/calculations) ### [Utilized cost](/concepts/calculations) ### [Cluster score](/concepts/calculations) ### [CPU](/concepts/calculations) ### [RAM](/concepts/calculations) ### [Storage](/concepts/calculations) ### [Attached Node Disks](/concepts/calculations) ## Concepts The following concepts are described: ### [Agent](/concepts/agent) ### [Aggregator](/concepts/aggregator) ### [Pushgateway](/concepts/pushgateway) ### [Server](/concepts/server) ### [Pricing API](/concepts/pricingapi) # Calculations In this section we review metrics and calculations provided on the Replex dashboards. Here is a list of all the calculations described in this section: * Forecasted cost * Cost previous 30 days * Total cost * Idle cost * Savings * Utilized cost * Cluster score * CPU * RAM * Storage * Attached Node Disks ### Forecasted Cost **Next 30 days forecast** is a prediction of the next month’s costs based on the historical data of the past 7 days. ![Next 30 days forecast](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf278ZEh-QboaOedEQ%2F-MFf2OyoIsY7F-RZZSOq%2FScreenshot%202020-08-07%20at%2015.45.37.png?alt=media\&token=6ce0cc45-af5d-4c86-bcd1-dbd9bb71a471) ### Costs previous 30 days **Costs previous 30 days** are the total costs of the cluster resources (VM instances, storage) for the past 30 days. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf278ZEh-QboaOedEQ%2F-MFf2lcts4QpR9YEfWdx%2FScreenshot%202020-08-07%20at%2015.45.37.png?alt=media\&token=9aeb26ab-21ab-4a35-84b6-9d65b4a8a640) ### Total cost **Total Cost** is calculated as the sum of **Idle Cost** and **Utilized Cost**. For example **Cluster Total cost** is calculated as the sum of **Idle Cost** and **Utilized Cost** of all resources in that cluster. ![Cluster Total Cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf278ZEh-QboaOedEQ%2F-MFf39rJ911ugKWwspTF%2FScreenshot%202020-08-07%20at%2015.47.43.png?alt=media\&token=92eef768-aacc-4b68-bbe6-013a58e4c41d) **Namespace Total cost** is calculated as the sum of **Idle Cost** and **Utilized Cost** for all pods running in that Namespace. ![Namespace Total cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf5b0AQgl23_5bDfDV%2F-MFf64FhhnUcZ03pT2gA%2FScreenshot%202020-08-07%20at%2016.27.05.png?alt=media\&token=cfe86ec9-3577-46c2-994d-387d748c0d01) Similarly, **Team Total cost** is calculated as the sum **Idle Cost** and **Utilized Cost** for all pods labelled with that specific team. ![Team Total cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf5b0AQgl23_5bDfDV%2F-MFf66UG9w7mj6h3ku0a%2FScreenshot%202020-08-07%20at%2015.59.18.png?alt=media\&token=90f8b2e0-e20f-47ab-8b3a-0a885725654f) ### **Idle cost** **Cluster Idle cost** is calculated as the total cost of cluster resources that are unused. ``` idle cluster cost = idle CPU cost + idle RAM cost + idle PVC cost ``` In the above calculation the **Idle CPU cost** is a sum of idle CPU cost for all nodes belonging to the cluster. Similarly the **Idle RAM cost** is a sum of idle RAM cost for all nodes belonging to that cluster. ![Cluster Idle cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf278ZEh-QboaOedEQ%2F-MFf39rJ911ugKWwspTF%2FScreenshot%202020-08-07%20at%2015.47.43.png?alt=media\&token=92eef768-aacc-4b68-bbe6-013a58e4c41d) **Node Idle CPU cost** is calculated as the cost of un-utilized CPU on that node. ``` Node idle CPU cost = cost of un-utilized node CPU ``` Similarly **Node idle RAM cost** is calculated as the cost of un-utilized RAM on that node. ``` Node idle RAM cost = cost of un-utilized node RAM ``` Lastly, **Node idle PVC cost** is calculated as the cost of un-utilized Persistent Volume Storage provisioned for that cluster. ``` idle PVC cost = cost of un-utilized Persistent Volume Storage ``` ### Savings **Cluster Savings** represent the cost savings that can be made by optimising cluster resource utilization. ![Savings](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf278ZEh-QboaOedEQ%2F-MFf39rJ911ugKWwspTF%2FScreenshot%202020-08-07%20at%2015.47.43.png?alt=media\&token=92eef768-aacc-4b68-bbe6-013a58e4c41d) On the Replex UI **Cluster** **Savings** are calculated as the deviation of the **total cluster cost** from a best practice utilization baseline of 70%. ``` Cluster Savings = total cluster costs - (used costs / 0.7) ``` Here is an example to make this clearer: Assume we have a cluster with a **total cluster cost** of $100 and used costs of $25. In this scenario cluster savings are calculated as follows: ``` Cluster Savings = $100 - ($25 / 0.7) = $64.29 ``` This means that cluster costs can be reduced by $64.29, with a cluster utilization of 70%. Once optimised, the new cluster cost is $35.71. ``` New cluster cost = $100 - $64.29 = $35.71 ``` Similarly, the new cluster utilization is 70.01% ``` New cluster utilization = $25 / $35.71 * 100 = 70.01% ``` ### Utili**z**ed cost **Utilized Cost** is calculated as the sum of **Pod CPU cost**, **Pod RAM cost** and **Storage cost** for all pods. For example **Namespace Utilized costs** are calculated as the sum of **Pod CPU cost**, **Pod RAM cost** and **Storage cost** for all pods running in that namespace. ![Namespace Utilized cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf5b0AQgl23_5bDfDV%2F-MFf6ZkugbWQ7Ehe1vWE%2FScreenshot%202020-08-07%20at%2015.59.18.png?alt=media\&token=40e9c22a-2bc9-4aa5-935d-8fa09c9018c5) Similarly, **Team Utilized costs** are calculated as the sum of **Pod CPU cost**, **Pod RAM cost** and **Storage cost** for all pods labelled with that specific team. ![Team Utilized cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf5b0AQgl23_5bDfDV%2F-MFf6gTW5HTGg0DNvu_n%2FScreenshot%202020-08-07%20at%2016.27.05%20\(1\).png?alt=media\&token=4bbb03b4-16bd-4833-954c-e449f3047ed0) ### Cluster score **Cluster score** is calculated based on how much resources were used within the cluster. The efficiency score is the deviation of actually used resources from the expected resources baseline. ![Cluster score](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf278ZEh-QboaOedEQ%2F-MFf4yqmDGBhtEEtYWW7%2FScreenshot%202020-08-07%20at%2015.45.37%20\(1\).png?alt=media\&token=82c41405-2a26-4752-b031-11fe16213265) ``` Expected resources baseline = used resources (100 - 30)% = 70% ``` where **30%** is allowed idle cost. #### Example If used resources are **58%** of total cost: ``` efficiency score = (1 - ((70 - 58) / 70)) * 100 = 83% ``` Note, that if used resources go above the requested resources, the efficiency will not exceed 100%, instead the idle costs will increase. #### Efficiency Score in the Replex Interface The icon next to the efficiency score is displayed in different colours, based on the actual score value. This ensures a quick overview of the general efficiency of the cluster, wether it is in the cluster list or on the cluster dashboard. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf278ZEh-QboaOedEQ%2F-MFf5C1H_esjg-Bjv8E2%2Fcluster-score-green.png?alt=media\&token=aa6425b7-9572-43ef-85bc-dd5770bd490a) A score between 61 and 100 means that the available cluster resources are used in an efficient way. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf278ZEh-QboaOedEQ%2F-MFf5EFWB-AFj-ZDoNMR%2Fcluster-score-yellow.png?alt=media\&token=ab79d607-7b79-409c-8a82-21239fe1112a) A score between 31 and 60 means that the cluster configuration should be improved. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf278ZEh-QboaOedEQ%2F-MFf5GArRcaVQBPNSAGR%2Fcluster-score-red.png?alt=media\&token=c6d10eb9-92b1-4ac6-b8bf-07d48459dca1) A score between 0 and 30 means that most of the resources in the cluster are idling and the cluster should be downscaled in order to safe money. ### CPU **Pod Requested CPU** is calculated as the sum of CPU requests for all containers in the pod. ![Pod Requested CPU](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf7l5Lm9VZUavq7V6Y%2F-MFf9PYKLsocbVpp-k4K%2Fk8s_pods_requests_used_cost%20\(2\).png?alt=media\&token=e9df6bc3-4153-4272-a713-c6c403c5cc9a) Each container can request CPU resources either as a fraction (0.5) or as millicpu (500m) using `spec.containers[].resources.requests.cpu` In Kubernetes one CPU is equivalent to 1 vCPU/Core for cloud providers and 1 hyperthread on bare-metal Intel processors. Below is an example of a pod spec. The pod has two containers, each requesting 250 millicpu of CPU. The first container requests CPU resources as a fraction (.25), whereas the second requests CPU resources in millicpu (250m). ``` apiVersion: v1 kind: Pod metadata: name: frontend spec: containers: - name: app1 image: images.replex.example/app1:v2 env: resources: requests: memory: "64Mi" cpu: ".25" limits: memory: "128Mi" cpu: ".5" - name: app2 image: images.replex.example/app2:v3 resources: requests: memory: "128Mi" cpu: "250m" limits: memory: "256Mi" cpu: "500m" ``` **Pod Used CPU** is calculated as the sum of CPU resources consumed by all containers in the pod expressed as a percentage. ![Pod Used CPU](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf7l5Lm9VZUavq7V6Y%2F-MFf9PYKLsocbVpp-k4K%2Fk8s_pods_requests_used_cost%20\(2\).png?alt=media\&token=e9df6bc3-4153-4272-a713-c6c403c5cc9a) **Pod CPU cost** is calculated as the sum of **Container CPU cost** for all containers in the pod. ![Pod CPU cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf7l5Lm9VZUavq7V6Y%2F-MFf9PYKLsocbVpp-k4K%2Fk8s_pods_requests_used_cost%20\(2\).png?alt=media\&token=e9df6bc3-4153-4272-a713-c6c403c5cc9a) **Pod Minimum CPU Usage** is the minimum amount of CPU used by the pod during the selected time range. **Pod Maximum CPU Usage** is the maximum amount of CPU used by the pod during the selected time range. **Recommended CPU** is generated by the Replex system and is the recommended CPU requests value for that pod. Recommended CPU is calculated as follows: ``` Recommended CPU = (Average CPU usage + CPU Usage stddev) / % Target CPU Utilization ``` {% hint style="info" %} Target CPU Utilization in the calculation above is 70% {% endhint %} ![Pod Min, Max CPU Usage](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MfhOiZzsmp8AVYLZKEl%2F-MfhPJ5QR_gWRkC_hY_y%2Freplex1-StatefulSets-Postgres-Replex.png?alt=media\&token=75b9fcb7-8208-4d90-8899-ba11b63306ff) **Standard Deviation** outlined in the CPU column is the stddev in CPU usage for that pod during the selected time range. ![Standard Deviation in CPU Usage](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Mfhg7EsL4RGfIMXNVGZ%2F-MfhhI-kJSbHXl8GjM7A%2Freplex1-StatefulSets-Postgres-Replex.png?alt=media\&token=888d396b-0ec3-4b22-999c-10f1a799a788) **Container CPU cost** is the product of **node CPU cost** and the fraction of CPU resources consumed by the container during a specific time interval. ``` Container CPU cost = node CPU cost * fraction of container node CPU usage ``` **Node CPU cost** is calculated as the product of **CPU cost per core** and the total number of cores on that node, for a specific time duration. ``` node CPU cost = (cpuCoreHourly * cores of node) / 60 * interval duration in minutes ``` **cpuCoreHourly** (CPU per core per hour) costs are fetched from the cloud providers using Replex’s pricing API. ### RAM **Pod Requested RAM** is calculated as the sum of memory requests for all containers in the pod. ![Pod Requested RAM](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf7l5Lm9VZUavq7V6Y%2F-MFf9PYKLsocbVpp-k4K%2Fk8s_pods_requests_used_cost%20\(2\).png?alt=media\&token=e9df6bc3-4153-4272-a713-c6c403c5cc9a) Memory requests are measured in bytes. Each container can request memory resources as a plain integer or as a fixed-point integer with one of these suffixes: E, P, T, G, M, K using `spec.containers[].resources.requests.memory.` Memory resources can also be requested as power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. Below is an example of a pod spec with two containers. The first container has RAM requests of 128Mi whereas the second has RAM requests of 256 Mi. ``` apiVersion: v1 kind: Pod metadata: name: frontend spec: containers: - name: app1 image: images.replex.example/app1:v2 env: resources: requests: memory: "64Mi" cpu: ".25" limits: memory: "128Mi" cpu: ".5" - name: app2 image: images.replex.example/app2:v3 resources: requests: memory: "128Mi" cpu: "250m" limits: memory: "256Mi" cpu: "500m" ``` **Pod Used RAM** is calculated as the sum of memory resources consumed by all containers in the pod expressed as a percentage. ![Pod Used RAM](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf7l5Lm9VZUavq7V6Y%2F-MFf9PYKLsocbVpp-k4K%2Fk8s_pods_requests_used_cost%20\(2\).png?alt=media\&token=e9df6bc3-4153-4272-a713-c6c403c5cc9a) **Pod RAM cost** is calculated as the sum of **Container RAM cost** for all containers in the pod. ![Pod RAM cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf7l5Lm9VZUavq7V6Y%2F-MFf9PYKLsocbVpp-k4K%2Fk8s_pods_requests_used_cost%20\(2\).png?alt=media\&token=e9df6bc3-4153-4272-a713-c6c403c5cc9a) **Pod Minimum RAM Usage** is the minimum amount of RAM used by the pod during the selected time range. **Pod Maximum RAM Usage** is the maximum amount of RAM used by the pod during the selected time range. **Recommended RAM** is generated by the Replex system and is the recommended RAM requests value for that pod. Recommended RAM is calculated as follows: ``` Recommended RAM = (Average RAM usage + RAM Usage stddev) / % Target RAM Utilization ``` {% hint style="info" %} Target RAM Utilization in the calculation above is 70% {% endhint %} ![Pod Min, Max RAM Usage](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MfhOiZzsmp8AVYLZKEl%2F-MfhPJ5QR_gWRkC_hY_y%2Freplex1-StatefulSets-Postgres-Replex.png?alt=media\&token=75b9fcb7-8208-4d90-8899-ba11b63306ff) **Standard Deviation** outlined in the RAM column is the stddev in RAM usage for that pod during the selected time range. ![Standard Deviation in RAM Usage](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Mfhg7EsL4RGfIMXNVGZ%2F-Mfhhojt1NUi16dcsIq0%2Freplex1-StatefulSets-Postgres-Replex.png?alt=media\&token=f6838d03-103c-45c4-a0c3-d8a0afd76654) **Container RAM cost** is the product of **node RAM cost** and the fraction of memory resources consumed by the container during a specific time interval. ``` Container RAM cost = node RAM cost * fraction of container node RAM usage ``` **Node RAM cost** is the product of **RAM cost per core** and the total number of cores on that node, for a specific time duration. ``` node CPU cost = (ramGBHourly ram of node in GB) / 60 * interval duration in minutes ``` **ramGBHourly** (RAM per GB per hour) costs are fetched from the cloud providers using Replex’s pricing API. ### **Storage** **Pod Storage Capacity** is calculated as the size of the Persistent Volume Claim (PVC) that is provisioned for that pod. This number is based on the pod spec. ![Pod Storage Capacity](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf7l5Lm9VZUavq7V6Y%2F-MFf9PYKLsocbVpp-k4K%2Fk8s_pods_requests_used_cost%20\(2\).png?alt=media\&token=e9df6bc3-4153-4272-a713-c6c403c5cc9a) **Pod Storage cost** is calculated as the cost of PVCs attached to that pod. ![Pod Storage cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf7l5Lm9VZUavq7V6Y%2F-MFf9PYKLsocbVpp-k4K%2Fk8s_pods_requests_used_cost%20\(2\).png?alt=media\&token=e9df6bc3-4153-4272-a713-c6c403c5cc9a) **Total Pod cost** is calculated as the sum of **CPU cost**, **RAM cost** and **Storage cost** for all containers in that pod. ![Pod Total Cost](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MFf7l5Lm9VZUavq7V6Y%2F-MFf9PYKLsocbVpp-k4K%2Fk8s_pods_requests_used_cost%20\(2\).png?alt=media\&token=e9df6bc3-4153-4272-a713-c6c403c5cc9a) ### Attached Node Disks **Node Disk Cost** is calculated with the standard GB price of the particular cloud provider. The price is multiplied with the disk capacity and scaled to the required time interval. ``` node disk cost = (standard GB price per hour) / 60 * capacity * interval duration in minutes ``` **Container Disk Cost** are calculated by distributing the node disk cost equally among all containers running on the node. ``` container disk cost = (node disk cost)/ number of containers running on the node ``` # Agent ### Setting Up With Thanos #### Configuring external\_labels for Thanos Due to the way we extract node and container information from the Prometheus metrics aggregated by Thanos, we recommend that the keys configured for `podLabel`, `nodeLabel` and `containerLabel` be renamed if already used in your Prometheus external labels. #### Setting up the thanos receive component Due to availability concerns with using the sidecar, we HEAVILY RECOMMEND using the `thanos receive` component which emulates all the realtime results of Prometheus that the agent needs to get metrics. Kindly review the setup docs [here](https://thanos.io/tip/components/receive.md/) and configure the remote\_write sections of your prometheus installation(s). Here's a sample of the prometheus configuration desired to work with the `thanos receive` component ```yaml # prometheus.yaml remote_write: url: /api/v1/receive headers: - THANOS-TENANT: ``` #### Configuring the Querier With The Replex Agent The Querier / Query Gateway is a part of the [thanos components](https://thanos.io/v0.6/thanos/getting-started.md/#components) which provides a Prometheus compatible endpoint that works just fine with the replex agent. If you have a Thanos instance set up, verify there is support for access to this endpoint using a simple curl command, replace `THANOS_QUERIER_URL` with the url of your thanos instance ```bash curl $THANOS_QUERIER_URL/api/v1/query?query=up ``` This should provide reasonable output similar to this which indicates that endpoint is prometheus compatible and would work with the agent. ``` up{instance="replex.io:9090", job="prometheus"} 1 up{instance="replex.io:9091", job="pushgateway"} 1 up{instance="replex.io:9093", job="alertmanager"} 1 up{instance="replex.io:9100", job="node"} 1 ``` From the previous guide, provide the `THANOS_QUERIER_URL` under the [prometheus.url](https://docs.replex.io/#metric-provider-parameters) variable as depicted in the agent docs presented in this section, and you're up and running with Thanos on Replex. ### Configuring Self-Signed SSL Certificates (On-Prem Only) For on-prem pushgateway deployments, if the puhgateway is served with a self-signed SSL certificate, the agent may encounter errors when trying to sync with the pushgateway. To resolve this, you can use the `sslCertificate` Helm chart parameter to pass your certificate into the agent. Example: ```yaml sslCertificate: “-----BEGIN CERTIFICATE-----\nMIIC1TCCAb2gAwIBAgIJAKbCs/2knCwGMA0GCSqGSIb3DQEBBQUAMBoxGDAWBgNV\nZAeRdaEZS6Bs\n-----END CERTIFICATE——" ``` #### Filesystem Metrics (Prometheus) This section is only for setups using prometheus as metrics provider. We use metrics from different sources for collecting PVC informations. The default setting uses [cAdvisor's](https://github.com/google/cadvisor) `kubelet_volume*` metrics. **cAdvisor** The default setup uses the `cAdvisor` metrics to get the PVC informations. In that case the `METRICS_FILESYSTEM` environment variable can be left at the default value that is `cadvisor`. Metrics used: | Storage Metric | `cAdvisor` Metrics | | -------------- | ------------------------------------- | | Capacity | `kubelet_volume_stats_capacity_bytes` | | Used | `kubelet_volume_stats_used_bytes` | **CSI** If `kubelet_volume*` metrics are not available and you are using [CSI](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/) plugins, you must set the `METRICS_FILESYSTEM` environment variable to `csi`. In that case [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) is required. For `csi`, we get the PVC informations from the `node_exporter` and `kube-state-metrics` metrics. Metrics used: | Storage Metric | `node_exporter` Metrics | `kube-state-metrics` Metrics | | -------------- | ----------------------------------------------------------- | --------------------------------- | | Capacity | `node_filesystem_size_bytes` | `kube_persistentvolumeclaim_info` | | Used | `node_filesystem_size_bytes` - `node_filesystem_free_bytes` | `kube_persistentvolumeclaim_info` | ### Exposed Metrics The Agent self exposes metrics. The metrics can be accessed via the `/metrics` route on port `:8083`. | Metric | Type | Labels | Description | | ----------------------------------- | ------- | -------------------------------- | ------------------------------------------------------------------------------------------- | | `replex_agent_provider_status` | gauge | `name`: metrics provider name | Indicates whether or not a metrics provider is reachable. 1 if it is reachable and 0 if not | | `replex_agent_sync_duration_count` | counter | `agent_version`, `response_code` | Total number of times the metrics were synchronized with the Replex server | | `replex_agent_sync_duration_sum` | gauge | `agent_version`, `response_code` | The total duration of all sync requests to the Replex server in seconds | | `replex_agent_retry_cache_size` | gauge | `cluster_id` | Count of cached metrics that are waiting to be re-sent to the replex server | | `replex_agent_failed_metrics_total` | counter | `cluster_id` | The total count of once failed metrics | ### Used Metrics These are the metrics the agent currently uses: | Property | Description | Prometheus | Instana `(plugin: metric)` | [Stackdriver](https://cloud.google.com/monitoring/api/metrics_kubernetes) | [Datadog](https://docs.datadoghq.com/integrations/kubernetes/) | | -------- | ---------------------- | ---------------------------------------------------------------------- | -------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------- | | 1 | Container CPU Usage | container\_cpu\_usage\_seconds\_total | *docker*: cpu.total\_usage | kubernetes.io/container/cpu/core\_usage\_time | kubernetes.cpu.usage.total | | 2 | Container MEM Usage | container\_memory\_working\_set\_bytes | *docker*: memory.usage | kubernetes.io/container/memory/used\_bytes | kubernetes.memory.working\_set | | 3 | Node CPU Usage | node\_cpu\_seconds\_total | *docker*: cpu.total\_usage | kubernetes.io/node/cpu/core\_usage\_time | kubernetes.cpu.usage.total | | 4 | Node MEM Usage | node\_memory\_MemTotal\_bytes - node\_memory\_MemAvailable\_bytes | *docker*: memory.usage | kubernetes.io/node/memory/used\_bytes | kubernetes.memory.usage | | 5 | Storage (Capacity) | kubelet\_volume\_stats\_capacity\_bytes, node\_filesystem\_size\_bytes | - | kubernetes.io/pod/volume/total\_bytes | kubernetes.kubelet.volume.stats.capacity\_bytes | | 6 | Storage (Used) | kubelet\_volume\_stats\_used\_bytes, node\_filesystem\_free\_bytes | - | kubernetes.io/pod/volume/used\_bytes | kubernetes.kubelet.volume.stats.used\_bytes | | 7 | Disk Capacity | container\_fs\_limit\_bytes | - | - | system.disk.total | | 8 | Disk Used | container\_fs\_usage\_bytes | - | - | system.disk.used | | 9 | Network I/O (Received) | container\_network\_receive\_bytes\_total | *docker*: network.rx.bytes | kubernetes.io/pod/network/received\_bytes\_count | kubernetes.network.rx\_bytes | | 10 | Network I/O (Sent) | container\_network\_transmit\_bytes\_total | *docker*: network.tx.bytes | kubernetes.io/pod/network/sent\_bytes\_count | kubernetes.network.tx\_bytes | | 11 | Disk I/O (Written) | container\_fs\_writes\_bytes\_total | *docker*: blkio.blk\_write | - | kubernetes.io.write\_bytes | | 12 | Disk I/O (Read) | container\_fs\_reads\_bytes\_total | *docker*: blkio.blk\_read | - | kubernetes.io.read\_bytes | ### Environment Variables | | Variable | Required | Default | Comment | | -- | --------------------------------- | ----------------------------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------- | | 1 | REPLEX\_TOKEN | Yes | | | | 2 | METRIC\_PROVIDER | Yes | | Options: `prometheus`, `datadog`, `stackdriver`, `instana`, `thanos` | | 3 | CLUSTER\_ID | If "KUBERNETES\_INFO\_PROVIDER" == "kubernetes" | | | | 4 | CLUSTER\_NAME | If "KUBERNETES\_INFO\_PROVIDER" == "kubernetes" | | | | 5 | PUSHGATEWAY\_URL | No | | | | 6 | PROMETHEUS\_SERVER\_URL | If "METRIC\_PROVIDER" == "prometheus" | | | | 7 | DATADOG\_API\_KEY | If "METRIC\_PROVIDER" == "datadog" | | | | 8 | DATADOG\_APPLICATION\_KEY | If "METRIC\_PROVIDER" == "datadog" | | | | 9 | DATADOG\_SITE | No | com | Options: `com`, `eu` | | 10 | GCP\_PROJECT\_ID | If "METRIC\_PROVIDER" == "stackdriver" | | | | 11 | INSTANA\_BASE\_URL | If "METRIC\_PROVIDER" == "instana" | | Format: `https://tenant-unit.instana.io` | | 12 | INSTANA\_API\_TOKEN | If "METRIC\_PROVIDER" == "instana" | | | | 13 | KUBERNETES\_INFO\_PROVIDER | No | kubernetes | Options: `kubernetes`, `instana` | | 14 | INSTANA\_CLUSTER\_ID | No | | | | 15 | ONLY\_USE\_READY\_NODES | No | false | Track only nodes that are in "Ready" state | | 16 | PROMETHEUS\_NODE\_LABEL | No | node | The label that represents the node in the Prometheus metrics | | 17 | PROMETHEUS\_CONTAINER\_LABEL | No | container | The label that represents the container in the Prometheus metrics | | 18 | PROMETHEUS\_POD\_LABEL | No | pod | The label that represents the pod in the Prometheus metrics | | 19 | CLOUD\_PROVIDER\_OVERRIDE | No | Detecting automatically | Options: `aws`, `azure`, `gce`, `custom`, `alibaba` | | 20 | USE\_CONTROL\_PLANE\_COST | No | false | Track costs of the Kubernetes Control Plane | | 21 | METRICS\_FILESYSTEM | No | cadvisor | Specify the filesystem metric source. Options: `cadvisor`, `csi` | | 22 | SYNC\_INTERVAL\_SECONDS | No | 300 | | | 23 | LOG\_LEVEL | No | 3 | Higher value means higher verbosity | | 24 | METRICS\_RETRY\_INTERVAL\_SECONDS | No | 300 | | | 25 | METRICS\_CACHE\_DISK | No | true | Cache failed metrics on disk | | 26 | METRICS\_CACHE\_DISK\_DIR | No | /data/metrics | Directory to cache metrics if `METRICS_CACHE_DISK` == `true` | | 27 | PROMETHEUS\_BEARER\_TOKEN | No | | Prometheus server requests bearer token. Only if `METRIC_PROVIDER` == `prometheus` | # Aggregator **Environment Variables** These are the environment variables used by the application: | | Variable | Required | Default | Description | | -- | ----------------------------------- | -------- | -------------------------- | --------------------------------------- | | 1 | POSTGRES\_USER | Yes | | | | 2 | POSTGRES\_PASSWORD | Yes | | | | 3 | POSTGRES\_DB | Yes | | | | 4 | POSTGRES\_HOST | Yes | | | | 5 | POSTGRES\_PORT | No | 5432 | | | 6 | POSTGRES\_SSL\_MODE | No | verify-full | | | 7 | CRON\_INTERVAL\_MINUTES | No | 30 | Metrics aggregation interval in minutes | | 8 | LOG\_LEVEL | No | 3 | | | 9 | METRICS\_RETENTION\_THRESHOLD\_DAYS | No | 7 | Days to keep raw metrics | | 10 | SERVER\_HOST | No | | | # Pushgateway **Environment Variables** These are the environment variables used by the application: | | Variable | Required | Default | Description | | -- | ------------------------------------- | ------------------------------- | --------------------------- | ---------------------------------------------------------------------------------- | | 1 | POSTGRES\_DB | Yes | | | | 2 | POSTGRES\_USER | Yes | | | | 3 | POSTGRES\_PASSWORD | Yes | | | | 4 | POSTGRES\_HOST | Yes | | | | 5 | POSTGRES\_SSL\_MODE | No | verify-full | | | 6 | POSTGRES\_PORT | No | 5432 | | | 7 | PUBLIC\_KEY | Yes | | Public key string (used for JWT). Prefix with `file:` if specifying a file path. | | 8 | PRIVATE\_KEY | Yes | | Private key string (used for JWT). Prefix with `file:` if specifying a file path. | | 9 | PRICING\_API\_HOST | No | | | | 10 | PRICING\_API\_KEY | If pricing mode is not disabled | | | | 11 | PRICING\_API\_SYNC\_INTERVAL\_SECONDS | No | 86400 (24hrs) | | | 12 | PRICING\_API\_MODE | No | `all` | `all`, `disable`, any number of \[`gcp`, `azure`, `aws`, `custom`] comma-separated | | 13 | LOG\_LEVEL | No | 4 | | | 14 | PRICING\_CPU\_CORE\_HOUR | No | 0.031611 | Default price of cpu core per hour. Used if pricing cannot be found. | | 15 | PRICING\_RAM\_GB\_HOUR | No | 0.004237 | Default price of memory GB per hour. Used if pricing cannot be found. | | 16 | PRICING\_STORAGE\_GB\_HOUR | No | 0.00005479452 | Default price of storage GB per hour. Used if pricing cannot be found. | | 17 | PRICING\_GPU\_HOUR | No | 0.95 | Default price for GPU per hour. Used if pricing cannot be found. | | 18 | METRICS\_ARCHIVE | No | disable | `disable`, the name of the Google Cloud storage bucket to archive metrics | | 19 | ENV | No | `test` | Options: `production`, `development`, `test` | | 20 | QUEUE\_PROCESS\_HOUR | No | 6 | Internal | # Server ### Multitenancy Setup This section describes how to create new tenants (also called organizations) in the replex software. The user who can set up another organization/tenant must have a `SUPER_ADMIN` role. If you do not yet have your authentication token, you must first log in to receive the token. If you already have the token, you can jump to step 3 and use the token in the authentication header. 1. Ensure the replex API is accessible. If you have exposed the API over a public URL, then you can use that. Otherwise, access the API locally by port-forwarding the service with the following command: ``` kubectl port-forward --namespace replex svc/replex-server 3100 ``` The following steps will assume the replex API is running at `http://localhost:3100`. 1. Login with super admin credentials. Next you'll login to the API to retrieve your authentication token for further requests to the API. ``` curl --location --request POST 'http://localhost:3100/api/v1/auth/login' \ --header 'Content-Type: application/json' \ --data-raw '{ "email": "", "password": "", "tenant": "" }' ``` The output will be similar to: ```javascript { "status": true, "data": { "token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJvcmdh...." } } ``` The contents of the `data.token` field is the authentication token we need. 1. Call the 'create organization' endpoint. ``` curl --location --request POST 'http://localhost:3100/api/v1/organizations' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "name": "", "email": "", "tenantID": "", "admin":{ "firstName": "--", "lastName": "--", "password": "--" } }' ``` A user with the user details provided in `admin` is automatically created for the new tenant. The login details are the values in `email` and `admin.password`. NOTE: The user that creates the new tenant is not automatically a member of the new tenant, but they can login to the tenant because the software allows `SUPER_ADMIN` users to login to any tenant on an installation. The output will be similar to: ```javascript { "status": true, "data": { "id": "eafedacd-4fba-4d31-be52-69567262a372", "defaultCurrency": "USD", "billingStatus": true, "name": "New Organization", "email": "john.doe@hello.com", "tenantID": "neworganization", "active": true, "updated_at": "2020-10-20T15:54:48.100Z", "created_at": "2020-10-20T15:54:48.100Z" } } ``` The new organization/tenancy has been successfully created. Once the software detects that more than one tenant exists, it automatically adds a new 'Tenant ID' field to the login form. The value of this field is the tenant you want to log in to. The email / password combination must exist in the tenant. 1. Generate Replex token. For configuring replex agents a Replex token is needed. To generate the token we need the organization ID from the previous step to specify for which organization we want to generate a token. The ID can be found in the response under `data.id`. We'll make use of the authentication token to get the Replex token: ```bash curl --location --request GET 'http://localhost:3100/api/v1/organizations//token' \ --header 'Authorization: Bearer ' ``` The response contains a new token which can be used as the replex.token Helm parameter in the agent Helm chart. ### Replex filter query language This section describes cluster stats querying basics and provides examples. **NOTE**: Examples below require authentication token, that can be retrieved at '' An instruction can be found in the section `Multitenancy Setup` above on the step 2. Cluster stats endpoints supports different parameters depending on the type of resource. Assume that a labels range stats endpoint is queried. It requires a couple of parameters: * start: timestamp that represents the beginning of the period where to start aggregate metrics. * end: timestamp that represents the end of the period where to stop aggregating metrics. * filters: a query string in [Lucene](https://lucene.apache.org/core/2_9_4/queryparsersyntax.html) query syntax. * clusterID: the unique ID of the cluster to query stats for. Available lucene keys to filter by are: * `labels.:` - represents resource labels, where key is `LABEL_NAME` and value is `LABEL_VALUE`, e.g. `labels.app:replex-server`. * `namespace:` - represents resources namespace, e.g. `namespace:monitoring`. * `key:` - represents label name that must appear in stats, e.g. `key:app`. A basic examples of `filters` parameter to query stats: | | Parameter value | Description | | - | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | 1 | `filters=labels.labelKey1:labelValue1 OR (labels.labelKey1:labelValue2 AND namespace:nsName1)` | Query resources that have labels with key `labelKey1` and value `labelValue1` or resources that are in namespace `nsName1` and have labels key `labelKey1` with value `labelValue2`. | | 2 | `filters=labels.labelKey1:labelValue1 AND namespace:nsName1` | Query resources that have labels with key `labelKey1` and value `labelValue1` and belong to namespace `nsName1`. | | 3 | `filters=namespace:nsName1 OR namespace:nsName2` | Query resources that belong to namespace `nsName1` or `nsName2`. | | 4 | `filters=namespace:nsName1 AND key:labelKey1` | Query resources that belong to namespace `nsName1` and have label keys such as `labelKey1`. | ### Environment Variables These are the environment variables used by the application: | | Variable | Required | Default | Description | | -- | ----------------------------------- | ------------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | | 1 | POSTGRES\_DB | Yes | | | | 2 | POSTGRES\_USER | Yes | | | | 3 | POSTGRES\_PASSWORD | Yes | | | | 4 | POSTGRES\_HOST | Yes | | Database host, or Master if using replication | | 5 | POSTGRES\_PORT | No | 5432 | | | 6 | ENV | No | development | Options: `production`, `development`, `test` | | 7 | PORT | No | 3100 | | | 8 | PUBLIC\_KEY | Yes | | Public key string (used for JWT). Prefix with `file:` if specifying a file path. | | 9 | PRIVATE\_KEY | Yes | | Private key string (used for JWT). Prefix with `file:` if specifying a file path. | | 10 | ADMIN\_FIRST\_NAME | No | | Initial Admin user credentials | | 11 | ADMIN\_LAST\_NAME | No | | | | 12 | ADMIN\_EMAIL | No | | | | 13 | ADMIN\_PASSWORD | No | | | | 14 | ORGANIZATION\_NAME | No | | Initial organization name | | 15 | SECRET | Yes | | Used for internal password recovery token en-/decryption | | 16 | CODE\_ACTIVE\_MINUTES | No | 15 | Token lifetime in minutes | | 17 | EMAIL\_MODE | No | enable | Options: `enable`, `disable` | | 18 | SMTP\_HOST | If `EMAIL_MODE` is enable | | | | 19 | SMTP\_PORT | If `EMAIL_MODE` is enable | | | | 20 | SMTP\_USER | If `EMAIL_MODE` is enable | | | | 21 | SMTP\_PASS | If `EMAIL_MODE` is enable | | | | 22 | SMTP\_SECURE | If `EMAIL_MODE` is enable | | | | 23 | HOST\_URL | No (required for on-prem) | | Server host url. | | 24 | PRICING\_API\_MODE | No | enable | Options: `enable`, `disable` | | 25 | PRICING\_API\_KEY | If `PRICING_API_MODE` is enable | | | | 26 | PRICING\_API\_HOST | If `PRICING_API_MODE` is enable | | | | 27 | MAX\_POOL\_SIZE | No | 20 | Max database connection pool size | | 28 | LOG\_LEVEL | No | 6 | Higher value represents higher verbosity (correspond to `syslog` levels - ) | | 29 | INGRESS\_NAME | No | | Ingress to be configured with new host information for organization | | 30 | INGRESS\_HOST\_SUFFIX | No | `replex.io` | Suffix appended to organization tenant ID to form Ingress host | | 31 | STRIPE\_API\_KEY | No | | Stripe API key. Used for Servicebot self checkout | | 32 | STRIPE\_WEBHOOK\_SECRET | No | | Stripe webhook secret. Used for verifying stripe webhook requests. | | 33 | SERVICEBOT\_PORTAL\_ID | No | | Servicebot Billing Page ID of the Customer Portal. | | 34 | SERVICEBOT\_CHECKOUT\_ID | No | | Servicebot Checkout Page ID. | | 35 | STRIPE\_PRODUCT\_ID | No | | Stripe Product ID. | | 36 | SAML\_DECRYPTION\_PRIVATE\_KEY | No | | SAML SP Private Key to decrypt SAMLResponses. Prefix with `file:` if specifying a file path. | | 37 | SAML\_DECRYPTION\_PUBLIC\_CERT | No | | SAML SP Public certificate to decrypt SAMLResponses. Prefix with `file:` if specifying a file path. | | 38 | SAML\_SIGNING\_PRIVATE\_KEY | No | | SAML SP Private Key to sign SAMLRequests. Prefix with `file:` if specifying a file path. | | 39 | SAML\_SIGNING\_PUBLIC\_CERT | No | | SAML SP Public Certificate to sign SAMLRequests. Prefix with `file:` if specifying a file path. | | 40 | METRICS\_RETENTION\_THRESHOLD\_DAYS | No | 7 | Represents the period when the raw metrics are saved in the db and could be accessed in short time range stats queries | | 41 | CLOUDCOST\_TAG\_KEY\_NAMESPACE | No | CLOUDCOST\_TAG\_KEY\_NAMESPACE | Tag key representing namespaces in cloud billing data. | | 42 | DB\_READ\_HOSTS | NO | | Comma-separated list of read replica IP addresses, e.g. 1.1.1.1,8.8.9.9,192.168.5.255 | # Pricing API **Environment Variables** These are the environment variables used by the application: | | Variable | Required | Default | Description | | -- | ---------------------------- | ---------------------------------------------- | ------- | --------------------------------------------------------------------------------------------- | | 1 | POSTGRES\_DB | Yes | | | | 2 | POSTGRES\_USER | Yes | | | | 3 | POSTGRES\_PASSWORD | Yes | | | | 4 | POSTGRES\_HOST | Yes | | | | 5 | POSTGRES\_PORT | No | 5432 | | | 6 | PRICING\_API\_KEY | Yes | | API key used for authorization | | 7 | GOOGLE\_API\_KEY | If pricing mode is 'all' or includes 'gcp' | | | | 8 | ENV | No | | Options: `production`, `development`, `test` | | 9 | BILLING\_ENCRYPT\_PASS | No | | For en-/decrypting billing integration data | | 10 | BILLING\_ENCRYPT\_SALT | No | | For en-/decrypting billing integration data | | 11 | PRICING\_API\_MODE | No | `all` | `all`, `disable`, any number of \[`gcp`, `azure`, `aws`, `custom`, `alibaba`] comma-separated | | 12 | ALIBABA\_ACCESS\_KEY\_ID | If pricing mode is 'all' or includes 'alibaba' | | Specifies the Access Key for the associated Alibaba account | | 13 | ALIBABA\_SECRET\_ACCESS\_KEY | If pricing mode is 'all' or includes 'alibaba' | | Specifies the Secret Key for the associated Alibaba account | # Monitoring and Alerts Describes the process of configuring your Grafana, Prometheus and Alertmanager instances to monitor your Replex deployments. ## Quick Note {% hint style="info" %} This documentation is meant to be used by on-prem Replex installations.\ \ Clients hosted on `*.replex.io` need not worry about the content of this documentation.\ \ All alerts and configurations here are handled for you. {% endhint %} ## Dependencies Setting up your monitoring stack requires you to have Grafana, Prometheus and Alertmanager installed on your desired cluster.\ \ Ensure Grafana version 5.3.4+ is installed as the charts JSON specification is only compatible from there. ## Setting up Prometheus Prometheus is used for our metrics as we expose them from our core applications using this metrics provider.\ \ We already configure our deployment spec with the required annotation (as shown below) so your Prometheus instance would scrape the metrics automatically. ``` # k8s-file.yaml ... annotations: prometheus.io/scrape: "true" ``` \ To access the metrics please add the following scrape config named `kubernetes-pods` from [here](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L254) to your `prometheus.yml` file to complete the configuration. \ \ You may ignore this step if the configuration above is in line with what is provided on your Prometheus installation. ## Configuring AlertManager (Optional) Alerts can also be configured based on certain metrics exposed to your Prometheus instance. This step is only necessary if you are installing AlertManager for the first time, there's a very nice guide on getting it set up and configuring your receivers [here](http://elatov.github.io/2020/01/alerting-with-prometheus-on-kubernetes/#install-alertmanager).\ \ Despite the earlier focus on installing AlertManager, the scope of the doc is outside the installation process.\ \ The first step to setting up alerts is to confirm that your Prometheus instance has already been configured to point to AlertManager correctly. We use the following configuration for our instance ```bash # prometheus.yml rule_files: - /etc/prometheus-rules/rules # This points to where the rules are stored alerting: alertmanagers: - static_configs: - targets: - alertmanager..svc.cluster.local:9093 #FQDN to your alertmanager instance ``` {% hint style="info" %} Restarting your Prometheus instance is required after changing this configuration {% endhint %} \ You can check out [this](https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/) article about alerting rules and pointing Prometheus to Alert Manager.
## Setting up Alerts Once Alert Manager is properly configured with Prometheus, you may then add the following rules specified here to your Prometheus rules configuration.\ \ You can copy and modify the current template below to fit your use cases or preferred alerting messages: ```bash # /etc/prometheus-rules/rules groups: - name: uptime rules: - alert: CAdvisorHostDown expr: up{job="kubernetes-cadvisor"} == 0 for: 1m labels: severity: high annotations: summary: CAdvisort reports Host {{ $labels.instance }} is down, investigate immediately! - alert: NodeExporterNodeDown expr: up{job="kubernetes-nodes"} == 0 for: 1m labels: severity: high annotations: summary: NodeExporter reports {{ $labels.instance }} is down, investigate immediately! - alert: APIServerDown expr: up{job="kubernetes-apiservers"} == 0 for: 1m labels: severity: high annotations: summary: APIServer {{ $labels.instance }} is down, investigate immediately! - alert: PodDown expr: up{job="kubernetes-pods"} == 0 for: 1m labels: severity: high annotations: summary: Pod {{ $labels.kubernetes_namespace }}/{{ $labels.kubernetes_pod_name}} is down, investigate immediately! - name: pvc rules: - alert: VolumeRequestThresholdExceeded expr: (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) > 0.9 for: 1m labels: severity: high annotations: summary: Volume {{ $labels.persistentvolumeclaim }} in namespace {{ $labels.namespace }} on node {{ $labels.kubernetes_io_hostname }} exceeded threshold capacity of 90% - alert: UnboundedPV expr: kube_persistentvolume_status_phase{phase != "Bound"} == 1 for: 1d labels: severity: high annotations: summary: PV {{ $labels.persistentvolume }} has been in phase {{ $labels.phase }} for more than 1 day. - alert: UnboundedPVC expr: kube_persistentvolumeclaim_status_phase{phase != "Bound"} == 1 for: 5m labels: severity: high annotations: summary: PVC {{ $labels.persistentvolumeclaim }} in namespace {{ $labels.namespace }} is currently in phase {{ $labels.phase }}. - name: replex rules: - alert: ServerErrorAlert expr: sum by (kubernetes_namespace, kubernetes_pod_name) (changes(server_http_request_duration_seconds_count{job="kubernetes-pods",status_code=~"5.*"}[1m])) > 0 for: 30s labels: severity: medium annotations: summary: 5xx errors on {{$labels.kubernetes_namespace}}/{{ $labels.kubernetes_pod_name }} for url {{ $labels.url }} exceeded threshold of 1 requests in 1 minute - alert: PushGatewayError expr: sum by (kubernetes_namespace, kubernetes_pod_name) (changes(pushgateway_push_requests_duration_seconds_count{job="kubernetes-pods", status=~"5.*"}[30m])) > 0 for: 30s labels: severity: medium annotations: summary: 5xx errors on pod {{$labels.kubernetes_namespace}}/{{ $labels.kubernetes_pod_name }} exceeded threshold of 1 requests within 30 minutes - alert: PricingAPIError expr: sum by (kubernetes_namespace, kubernetes_pod_name) (changes(pricingapi_http_request_duration_seconds_count{job="kubernetes-pods",status_code=~"5.*"}[1m])) > 0 for: 30s labels: severity: medium annotations: summary: 5xx errors on pod {{$labels.kubernetes_namespace}}/{{ $labels.kubernetes_pod_name }} exceeded threshold of 1 requests in 1 minute - alert: AggregatorErrors expr: changes(aggregator_aggregation_duration_seconds_count{job="kubernetes-pods",status="0"}[15m]) > 0 for: 15m labels: severity: medium annotations: summary: Failed aggregation on pod {{$labels.kubernetes_namespace}}/{{ $labels.kubernetes_pod_name }} exceeded threshold of 1 requests in 15 minute - name: database rules: - alert: ReplicationStopped expr: pg_repl_stream_active{job="kubernetes-pods"} == 0 for: 10s labels: severity: high annotations: summary: Replication slot {{ $labels.slot_name }} for server {{ $labels.server }} is no longer active - alert: PushGateWayDatabaseUnavailable expr: pushgateway_database_status{job="kubernetes-pods"} == 0 for: 1m labels: severity: high annotations: summary: Database connection for pod {{$labels.kubernetes_namespace}}/{{ $labels.kubernetes_pod_name }} is no longer active description: Database connection for the pod {{$labels.kubernetes_namespace}}/{{ $labels.kubernetes_pod_name }} is no longer active, consider restarting the pod - alert: AggregatorDatabaseUnavailable expr: aggregator_database_status{job="kubernetes-pods"} == 0 for: 1m labels: severity: high annotations: summary: Database connection for pod {{$labels.kubernetes_namespace}}/{{ $labels.kubernetes_pod_name }} is no longer active description: Database connection for the pod {{$labels.kubernetes_namespace}}/{{ $labels.kubernetes_pod_name }} is no longer active, consider restarting the pod - alert: FailedAggregationEvent expr: changes(aggregator_query_duration_seconds_count{job="kubernetes-pods",status="0"}[15m]) > 0 for: 1m labels: severity: high annotations: summary: "{{ $labels.aggregation_type }} cron job failed on pod {{$labels.kubernetes_namespace}}/{{ $labels.kubernetes_pod_name }}" description: A failed aggregation event has occurred on {{ $labels.aggregation_type }} on pod {{$labels.kubernetes_namespace}}/{{ $labels.kubernetes_pod_name }} - alert: HighNumberOfConnections expr: pg_stat_database_numbackends{datname="postgres"} > 30 for: 1m labels: severity: high annotations: summary: "More than 30 connections to database {{$labels.datname}} on server {{ $labels.server }}" description: "More than 30 connections to database {{$labels.datname}} on server {{ $labels.server }}" ``` After copying the template above and modifying (if necessary) into the Prometheus rules file, you can check your Prometheus dashboard to verify the alerts are registered on the alert page {% hint style="info" %} Restarting your Prometheus instance is required after editing the rules {% endhint %} ![Snapshot of correctly configured Prometheus alerts](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MM0wF6T78aKqZ7cpPgx%2F-MM1jjckiwmX-JGArSXu%2Fimage.png?alt=media\&token=0418f2c9-2d4c-45bc-a385-987f5ce58642) ## Finishing with Grafana Once Prometheus is configured, you can then proceed to install the Grafana charts.
The charts are hosted and maintained publicly on Grafana | Name | URL | | ---------------- | ---------------------------------------------- | | Request Metrics | | | Database Metrics | | Provided the metrics from the Replex components are exposed properly, you should access dashboards similar to this: ![Request Metrics Dashboard](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MM0wF6T78aKqZ7cpPgx%2F-MM1nk6pnJ0E6E7SdGl8%2Fimage.png?alt=media\&token=f6b24ac1-50aa-4286-b8a1-bcd72011f817) ![Database Metrics Dashboard](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MM0wF6T78aKqZ7cpPgx%2F-MM1ntVvLXdwcbY9qPri%2Fimage.png?alt=media\&token=3470ca31-f442-4d2b-9447-8fc130ec32ab) # Settings This section outlines advanced configuration options for the **Replex Kubernetes agent**. We will walkthrough the process of adding new users or admins, keeping track of account invitations, adding billing credentials and custom cost models, configuring authentication and generating API tokens. ### [Users](/settings/users) ### [Pending Invitations](/settings/pending-invitations) ### [Billing Credentials](/settings/billing-credentials) ### [Single Sign On (SAML) ](/settings/single-sign-on-saml) ### [Custom Cost ](/settings/custom-cost) ### [Integrate Out of Cluster Costs (Cloud Costs)](/settings/integrate-out-of-cluster-costs-cloud-costs) ### [API Tokens ](/settings/api-tokens) #### # Users The **Users** section provides an overview of all the admins or users with access to the account. Admins can quickly review existing roles and privileges, add and remove **Users** or change existing privileges. Each member can be associated with one of two Roles: * **Admin** * Has **full** access within organization * **User** * Has **View** Access within organization ![Users Section](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MN3ZvObI1xvzjblFPKP%2F-MN3cHNJu_y_SgWALpCc%2FUsers-Replex.png?alt=media\&token=58f078c1-6f84-4cf2-91f0-764c646d4d2f) ### Add new Admins or Users To add new users or admins first navigate to the **Users** section of the **Settings** page. The **Settings** page can be accessed in the left hand panel of the Replex UI. Click on **+ Invite People** in the top right corner of the **Users** section. On the next screen enter the **First** and **Last Name** of the new user or admin, their **Email** address and the associated **Role**. Click on **Send Invitations.** The new users/admin will receive an email with instructions on how to setup their account. ![Add New Users/Admins](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MN3dp1aj4aCdjPoZ1vL%2F-MN3eJ6OJW2kVogM2jAe%2FUsers-Replex_3.png?alt=media\&token=f26c1d2c-2cb4-44ba-86fa-28b7173308f6) ### Add Multiple Users Admins can also add multiple **Users** at once. To do this click on **add multiple people at once** from the **Invite People** screen**.** On the next screen add multiple comma separated email addresses to invite new users in bulk. ![Add new users/admins in bulk](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MN3dp1aj4aCdjPoZ1vL%2F-MN3fkGL6BMIFGWIZEsk%2FUsers-Replex_4.png?alt=media\&token=4679ad88-09cc-4f0d-b240-1d49aedc57ed) ### Edit and Remove Users Admins can edit **User** information as well as the roles or privileges associated with them by clicking the edit icon in front of each User. ![Edit User](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR4yRdpmy07oQ9JoBqI%2F-MR50FwKOf3jFTTzFTQG%2Fusers_1212.png?alt=media\&token=619a5570-02e3-4df7-ad1c-8a164e68c640) Users can be removed right from the edit screen above or by clicking the delete icon in front of each User. # Pending Invitations The **Pending Invitations** section of the **Settings** Page allows admins to keep track of invitations sent out but not yet accepted. These are the users/admins who have been invited but have not yet accepted or completed their account setup. Admins can resend invitations as a reminder by clicking on the **Resend** button or cancel these invitations by clicking on **Deny**. ![Pending Invitations](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MN3iZL14UkDAcdDAmO2%2F-MN3j7Xn4Pd-a2ronB_C%2FUsers-Replex_6.png?alt=media\&token=70c01237-24f8-4250-8ed5-df66c4be7e0a) # Billing Credentials The **Billing Credentials** section of the **Settings** page, allows admins to add, review edit and delete cloud provider billing credentials to their account. By default Replex uses each cloud provider's list price to help chargeback and allocate Kubernetes costs. To accurately allocate and chargeback costs, it is recommended to enable Replex to access your cloud provider's billing data. ## Add Billing Credentials To add cloud provider billing credentials click **+ Add Credentials** in the top right of the **Billing Credentials** section. On the next screen, choose the cloud provider you are adding billing credentials for. Billing credentials can be added for **AWS**, **Azure,** **Alibaba and Google Cloud**. ![Choose Cloud Provider](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MN476wHo1X0zIVerGte%2F-MN4AVOPmiFycpBg4ZmJ%2FBilling-Credentials-Replex.png?alt=media\&token=e03ca378-c752-43d7-92c3-f7d03ee19684) On the next screen enter the **Credential Name** and choose the clusters that the credentials will be applied to. To apply these credentials to all clusters currently in the account as well as to those that will be created in the future, click on the **Set as default** toggle. ![Enter Billing credentials](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MNNdUhT7E0xsmhjQFQF%2F-MNNdun-04dDRTL4uVB6%2FNew-AWS-Credentials-Replex.png?alt=media\&token=18650ffe-8a95-43e9-98be-51a18c0e26bd) Next enter the credentials manually or as JSON by filling out each required field. Finally, click **Save** to add the billing credentials to your account. Once added, billing credentials will show up in the **Billing Credentials** section, where admins can review, edit or delete them. ![Billing credentials section](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR0eXO8oPKn7lNVgNuK%2F-MR0fYmn8aL6HDd3jLVl%2FBilling-Credentials-Replex.png?alt=media\&token=7019dd16-9a2b-4984-a0bb-fa4aec08a913) ![Billing credentials section](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR0eXO8oPKn7lNVgNuK%2F-MR0fYmn8aL6HDd3jLVl%2FBilling-Credentials-Replex.png?alt=media\&token=7019dd16-9a2b-4984-a0bb-fa4aec08a913) Below is the list of required fields for each cloud provider: ### AWS * Report Name * Access Key ID * Secret Access Key {% hint style="info" %} To create Cost and Usage Reports you need a user with advanced access. This user must have at least this policies: 1. AmazonS3FullAccess (or similar that can create, read, list and attach bucket policies) 2. Billing (to access billing console and create report) 3. AWSCloudFormationFullAccess (to create a stack that will get data from bucket and put it to Athena DB) 4. IAMFullAccess (or similar to create user, create policy, attach policy, get access keys) 5. EC2:DescribeInstances (to get information about available instance types) 6. EC2:DescribeVolumes (to get information about attached node disks) {% endhint %} {% hint style="info" %} **Report Name** refers to AWS Cost and Usage Reports. Follow the steps outlined below to setup reports on your AWS console {% endhint %} a. Click the “Create report button” [here](https://console.aws.amazon.com/billing/home#/reports) and follow the following steps to create a billing report: 1. Enter the report's name. 2. Check the "Include resource IDs" and click next. 3. Click "Configure" in "S3 Bucket section" and enter a bucket name in "Create a bucket" section. 4. Enter report path prefix. 5. Check "Daily" in "Time granularity" section. 6. Check "Amazon Athena" in "Enable report data integration for" section. 7. Click "Next", review entered data and save. b. Create a CloudFormation Stack using doc from [here](https://docs.aws.amazon.com/cur/latest/userguide/use-athena-cf.html). c. Create a new AWS User, that will query billing data, [here](https://console.aws.amazon.com/iam/). d. Grant the newly created user the following access roles: {% hint style="info" %} Make sure you replace the `{BUCKET_NAME}` with the actual name of the bucket in the `VisualEditor4` block and do not remove the trailing asterisk. {% endhint %} \ `{`\ `"Version": "2012-10-17",`\ `"Statement": [`\ `{`\ `"Sid": "VisualEditor1",`\ `"Effect": "Allow",`\ `"Action": [`\ `"athena:startQueryExecution",`\ `"athena:getQueryExecution",`\ `"cur:DescribeReportDefinitions",`\ `"glue:GetTable",`\ `"glue:GetPartitions",`\ `"glue:GetDatabase",`\ `"ec2:DescribeInstances",`\ `"ec2:DescribeVolumes"`\ `],`\ `"Resource": "*"`\ `},`\ `{`\ `"Sid": "VisualEditor2",`\ `"Effect": "Allow",`\ `"Action": [`\ `"s3:PutObject",`\ `"s3:GetObject",`\ `"s3:ListBucketMultipartUploads",`\ `"s3:AbortMultipartUpload",`\ `"s3:CreateBucket",`\ `"s3:ListBucket",`\ `"s3:GetBucketLocation",`\ `"s3:ListMultipartUploadParts"`\ `],`\ `"Resource": "arn:aws:s3:::aws-athena-query-results-*"`\ `},`\ `{`\ `"Sid": "VisualEditor4",`\ `"Effect": "Allow",`\ `"Action": [`\ `"s3:PutObject",`\ `"s3:GetObject",`\ `"s3:ListBucketMultipartUploads",`\ `"s3:AbortMultipartUpload",`\ `"s3:CreateBucket",`\ `"s3:ListBucket",`\ `"s3:GetBucketLocation",`\ `"s3:ListMultipartUploadParts"`\ `],`\ `"Resource": "arn:aws:s3:::{BUCKET_NAME}*"`\ `}`\ `]`\ `}` \ e. Proceed to creation of billing integration in Replex settings. Fields can be added both as JSON as well as manually. Example JSON input: ``` { "accessKeyId": "", "secretAccessKey": "", "reportName": "" } ``` ![Enter AWS billing credentials as JSON](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MNZ2dbyPJiNhZbLAbBG%2F-MNZ3GmAYZXTtRQLDPEM%2FNew-AWS-Credentials-Replex%20\(1\).png?alt=media\&token=3584b4bb-3395-4e84-b9f9-661feb1810fb) ![Enter AWS billing credentials manually](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MNZ2dbyPJiNhZbLAbBG%2F-MNZ3kBW6nMwD2WuQ6B3%2FNew-AWS-Credentials-Replex%20\(1\).png?alt=media\&token=59e434f6-8b14-4422-b9d0-1fd2c5c9a4f7) ### Google Cloud * Type * Project ID * Private Key ID * Private Key * Client Email * Client ID * Auth URI * Token URI * Auth Provider X509 Cert URL * Client X509 Cert URL * Big Query Integration - Dataset Name * Big Query Integration - Table Name * Big Query Integration - Dataset Location {% hint style="info" %} Make sure you configure Billing export to BigQuery by following the steps outlined below: {% endhint %} a. Create a new service account by following [this guide](https://cloud.google.com/docs/authentication/getting-started). b. Grant the newly created service account the following access roles: * BigQuery Data Viewer * BigQuery Job User c. Follow the instructions [here]() to enable and set up Billing export to BigQuery. d. Input required fields in the Billing Credentials section either as JSON or manually. Example JSON input: ``` { "serviceAccount": { "type": "", "project_id": "", "private_key_id": "", "private_key": "", "client_email": "", "client_id": "", "auth_uri": "", "token_uri": "", "auth_provider_x509_cert_url": "", "client_x509_cert_url": "" }, "bigQueryIntegration": { "datasetName": "", "tableName": "", "datasetLocation": "" } } ``` ![Enter GCP billing credentials as JSON](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MNZ2dbyPJiNhZbLAbBG%2F-MNZ5IhADMm-q2gb2z7i%2Fgcp_2.png?alt=media\&token=bc08bc03-67aa-4adb-bf15-888396d1f534) ![Enter GCP billing credentials manually](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MNZ2dbyPJiNhZbLAbBG%2F-MNZ4xvSQga9UNsJr1PY%2FGCP.png?alt=media\&token=7b13efed-eb50-4ddd-b438-cd7359d81160) ### Azure * Client ID * Subscription ID * Secret * Domain {% hint style="info" %} Make sure the account you use to add billing credentials has the "User Access Administrator" rights. {% endhint %} a. Get the subscription ID that contains the information about billing data. b. Make sure Azure Active Directory is available and App Registrations are enabled. c. Get the AAD ID and create the auth key. 1. Open the AAD page, then click on the App registrations sidebar menu. Open the corresponding app and copy the Application (client) ID and the Directory (tenant) ID. 2. Click on the Certificates & secrets tab and create a client secret. d. Grant permissions 1. Go to subscriptions and select the corresponding subscription. 2. Click on Access Control (IAM) in the sidebar, open the Role assignments tab and add a new role assignment for the chosen app. 3. Next click on the Add button, in the right pop-up modal, select roles "Cost Management Reader" and "Virtual Machine Contributor", assign access to "Azure AD user, group, or service principal", then in the Select choose the app. e. Once the above steps have been completed, input required fields in the Billing Credentials section either as JSON or manually. Example JSON input: ``` { "clientID": "", "subscriptionID": "", "secret": "", "domain": "" } ``` ![Enter Azure billing credentials as JSON](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MNNhoSIh2PvjOe6oO4X%2F-MNNjlYJN1sNJkHUPtjx%2FNew-Azure-Credentials-Replex.png?alt=media\&token=72cb5f14-da6d-42be-872b-3144de0f681b) ![Enter Azure billing credentials manually](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR0gPao0mxyZ-O894cM%2F-MR0hJhIndF11lceRQwU%2FNew-Azure-Credentials-Replex.png?alt=media\&token=90d046cd-6630-4fd8-92c1-3e11958f149b) ### Alibaba * Access Key ID * Access Key Secret {% hint style="info" %} Make sure the account you use to add billing credentials has the "AliyunBSSReadOnlyAccess" permission. {% endhint %} a. Follow [this](https://www.alibabacloud.com/help/faq-detail/63482.htm) guide to obtain your Access Key ID and Access Key Secret. b. Enter both in the replex Billing Credentials settings page. \ \ Example JSON input: ``` { accessKeyID: "", accessKeySecret: "" } ``` ![Enter Alibaba billing credentials as JSON](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdHfp56MjNRCgTlrJbt%2F-MdIQO4IEBGlvzfl7Usf%2FNew-Credentials-Replex.png?alt=media\&token=18a9d8ea-3a28-4362-8c69-6fcaec1c33b5) ![Enter Alibaba billing credentials manually](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MdHfp56MjNRCgTlrJbt%2F-MdIQoLdMgo0Z7HT-j7Q%2FNew-Credentials-Replex%20\(2\).png?alt=media\&token=47a1ab4e-ced0-4327-abeb-796e9c894131) ## Edit or Remove Credentials Admins can edit previously added **Billing Credentials** by clicking the edit icon in front of each credential. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR54F41TrBlHSOFf9ja%2F-MR55GS3hae43xWPo5Lo%2Fsettings-1-Edit-GCP-Credentials-Replex.png?alt=media\&token=53194789-370f-46e7-be3a-0c983869cebd) Previously added billing credentials can be deleted by clicking on the delete icon in front of each credential. ### # Single Sign On (SAML) Replex’s SAML implementation allows admins to configure SSO authentication using any one of a number of compliant identity providers (IdPs). Admins can configure any IdP that conforms to the SAML form of authentication, to be used with Replex. These include but are not limited to the following IdPs: * Okta * OneLogin * Azure Active Directory * SecureAuth * TrustBuilder * adAS * ADFS ### Add SAML Integration To configure SAML, click on **Settings** in the left hand panel and then click on **Single Sign On (SAML)**. If you are configuring SAML for the first time you will see a screen with no entries. To add your first SAML integration click on **+ Add SAML Integration** in the top right corner. On the next screen, enter the **Name** and **Provider** of your SAML integration. ![Integrating Single Sign On (SAML)](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR4ezBYNxQLEp4tufWB%2F-MR4fLb3ZSdVMjksDVrQ%2Fokta-11-New-SAML-Integration-Replex.png?alt=media\&token=63cfebb6-9b45-4186-8983-70909e14a94d) To proceed follow one of the two implementation methods: manual or automatic setup. #### Automatic Setup The automatic setup allows admins to easily integrate SAML by exchanging configuration metadata with the IdP. {% hint style="info" %} Automatic setup is only supported by some IdPs. {% endhint %} Follow the steps outlined below to integrate SAML using the automatic setup method: 1. Download the **Configuration Metadata** using the button right at the start of the Automatic setup section. 2. Upload the metadata file downloaded in the previous step to your IdP. 3. Download the **IdP-metadata** file issued from your IdP or note the ssoLink and the certificate provided by the IdP to sign SAML messages. 4. To complete the integration, upload the IdP-metadata file downloaded in the previous step in the **IdP Metadata** section right at the end of the screen. Make sure you choose **File**, before uploading the IdP-metadata file. 5. Integration can also be completed by choosing **Manual Input** in the **IdP Metadata** section and entering the ssoLink and signing certificate noted in step 3. #### Manual Setup To complete the integration using the manual method follow the steps below: 1. Create a SAML app in your IdP. During the SAML app creation you will need to provide the following four values to your IdP (copy these values using the button in front of each field and paste them into your IdP): * ACS URL * Logout URL * Audience/SP Entity ID * Name ID Format 2. Download the **IdP-metadata** file issued from your IdP or note the ssoLink and the certificate provided by the IdP to sign SAML messages. 3. To complete the integration, upload the IdP-metadata file downloaded in the previous step in the **IdP Metadata** section right at the end of the screen. Make sure you choose **File**, before uploading the IdP-metadata file. 4. Integration can also be completed by choosing **Manual Input** in the **IdP Metadata** section and entering the ssoLink and signing certificate noted in step 2. ![Manual Setup](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR5JLtHsPX499ix2fUY%2F-MR5JzFGgCtd-W78oJlz%2F1-sso-New-SAML-Integration-Replex.png?alt=media\&token=7de67024-ed23-4871-bd7e-15e1ec8e5eeb) ### Edit or Remove Integration Admins can edit previously added **SAML Integrations** by clicking the edit icon in front of each integration. ![Edit SAML Integration](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR56Pq-YuTnkU-m3qv1%2F-MR57k09FgWBIaxNZg8q%2Fokta-21-Edit-SAML-Integration-Replex.png?alt=media\&token=9fcfd153-a6f3-42f5-bb30-516d3c310fb9) Previously added **SAML Integrations** can be deleted by clicking on the delete icon in front of each integration. ### Okta Integration In this section of the documentation, we will provide a detailed walkthrough of integrating Okta with Replex. Since Okta does not provide a way to upload configuration metadata from the service provider, we will be following the manual method of SAML integration. {% hint style="info" %} You will require an account with administrator privileges in Okta to complete the integration {% endhint %} Navigate to the **Single Sign On (SAML)** screen by clicking on **Settings** in the left hand panel of the Replex UI. Once there, enter the **Name** and **Provider** of the SAML implementation. Here we use "Okta SSO" as the **Name** and "Okta" as the **Provider**. Download the **Configuration Metadata** file. ![Enter Name and Provider](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR4ifhdakbV-hnd0Rpp%2F-MR4ivj8v2ZVRE4WqufI%2Fokta-11-New-SAML-Integration-Replex.png?alt=media\&token=553b7727-5b07-474f-afa3-9b9891f7bbf6) #### Create SAML App in Okta Open Okta in your browser and sign in using an account with administrator privileges. Navigate to **Applications**. ![Okta Applications](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Mfs4GF7g4PIFaAzk5A0%2F-Mfs5SepPXmYTcI8mkha%2Fokta-dev-37030484-Applications.png?alt=media\&token=71d50349-8d3e-4ec6-bdb5-8363ae9c65f1) Click **Create App Integration**. In the **Create a new app integration** pop-up, choose **Saml 2.0** as the sign-in method and click **Next.** ![Choose sign-in method](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Mfs4GF7g4PIFaAzk5A0%2F-Mfs6LhdFbj2Jn0nAi9Z%2Fokta-dev-37030484-Applications%20\(1\).png?alt=media\&token=1de0aab4-1ee2-490b-9d7d-a10f6712c116) This will open the **Create SAML Integration** Wizard, which will guide you through the process of creating a new app in Okta. In the **General Settings** tab, enter **Replex** as the **App Name** and optionally upload the Replex logo. You can also choose whether to display the application icon, in the **App visibility** section. ![General Settings Tab](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Mfs6htGbJT9Muibn51o%2F-Mfs7c7h9z3xrc2QQYEI%2Fokta-dev-37030484-Applications%20\(2\).png?alt=media\&token=f95ef5a9-988a-4c9b-ac46-a5e710329dab) Click **Next** to proceed. In the **Configure SAML** tab, copy and paste the **ACS URL** from the Replex UI into the **Single sign on URL** field. Similarly, copy and paste the **Audience/SP Entity ID** from the Replex UI into the **Audience URI (SP Entity ID)** field. Next choose **EmailAddress** from the drop down list in front of the **Name ID format** field. Similarly, choose **Email** from the drop down in front of the **Application username** field. Next choose **EmailAddress** from the drop down list in front of the **Name ID format** field. Similarly, choose **Email** from the drop down in front of the **Application username** field. ![Configure SAML](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Mg54LcYV5hhLMG0tLoN%2F-Mg5L60yeQVGBaVstNi7%2Fokta-dev-37030484-Applications%20\(3\).png?alt=media\&token=df842f8c-7520-4262-9a2c-f3f3639f6569) Click **Show Advanced Settings** and set the **Assertion Encryption** to **Encrypted**. Create a file **sp\_certificate.pem**, open the previously downloaded sp metadata xml file and copy the encryption certificate. Paste the encryption certificate into the new .pem file replacing {ENCRYPTION\_CERTIFICATE}. ``` -----BEGIN CERTIFICATE----- {ENCRYPTION_CERTIFICATE} -----END CERTIFICATE----- ``` Upload **sp\_certificate.pem** file in the **Encryption Certificate** field. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MfrBueLYdwpUVpig4HL%2F-MfrLaay6JW0qiiPJE9v%2FScreenshot%202021-07-30%20at%2015.18.31.png?alt=media\&token=f1aa1a54-ef66-4c1e-aab8-9c1b833a5f43) You can verify the information entered above by scrolling down to Section B and clicking on **Preview the SAML Assertion** button. ![Preview the SAML Assertion](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MgAVRxaSblbKB3FTrGA%2F-MgAVY_r267Lp4PPb8rN%2Fokta-dev-37030484-Applications%20\(4\).png?alt=media\&token=5286d025-ae8e-42f7-a2bf-40a13aa77da9) Click **Next** to proceed. Fill out the required **Feedback** tab and click **Finish**. This will create the new Okta application and you will be redirected to the **Sign On** tab of the newly created application overview screen. Download the **identity provider metadata** file at the bottom of the screen. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MgAVueZSM89HZRVfi3y%2F-MgAWfUWQvfWOBGt2lAc%2Fokta-dev-37030484-replex-replex.png?alt=media\&token=a37d81a3-13bd-42bf-97df-dd604d9c24f1) #### Upload Configuration file to Replex Switch to the Replex UI and upload the **identity provider metadata** file downloaded in the previous step in the **IdP Metadata** section. Next click on **Save** to complete the setup and activate the Okta integration. ![Upload identity provider metadata file](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR4kWjft4nt1JLXvMN7%2F-MR4lRbFhyAHBoi0wbah%2Fokta-12-New-SAML-Integration-Replex.png?alt=media\&token=de63d25b-6733-4b37-9615-2b59b0bf02de) To enable users to access Replex via Okta, they have to get it assigned to them first. To do this, navigate to the **Assignments** tab of the **Applications** view in your Okta account, click on **Assign** to assign Okta to either people to groups. ![Assign to People or Groups](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MgAVueZSM89HZRVfi3y%2F-MgAXy-BOkvRm9hmY3Ey%2Fokta-dev-37030484-replex-replex%20\(1\).png?alt=media\&token=4cc36970-1186-4583-bb39-f34e7813a939) Each new user assigned will have the **User** role assigned to them when first logging in. In the Replex UI admins can review the new Okta integration in the **Single Sign On (SAML)** section of the **Settings** Page. They can also edit or delete the integration using the respective icons in front of the integration. ![Review, edit, delete SAML integrations](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MR4kWjft4nt1JLXvMN7%2F-MR4m5r7UjlcyokQjOrG%2Fokta-10-Single-Sign-On-SAML-Replex.png?alt=media\&token=9c0b484e-1ce5-4348-ae65-36522e343ab4) ### Keycloak Integration In this section of the documentation, we will provide a detailed walkthrough of integrating Keycloak with Replex. {% hint style="info" %} You will require an account with administrator privileges in Keycloak to complete the integration {% endhint %} Navigate to the **Single Sign On (SAML)** screen by clicking on **Settings** in the left hand panel of the Replex UI. Once there, enter the **Name** and **Provider** of the SAML implementation. Here we use "Keycloak SSO" as the Name and "Keycloak" as the Provider. ![Enter Name and Provider](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MT_Cft17Y0R3YlToCsA%2F-MT_hud2F8bfx12caH_p%2Fk8s_101_New-SAML-Integration-Replex.png?alt=media\&token=65fd2537-5636-43e3-ab6a-813057549712) Download the configuration metadata file using the green **Configuration Metadata** download button. Now switch to the Keycloak UI and create a new Realm for Replex. ![Create new Keycloak realm](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MT_p6pMMCz9PXkhg9L9%2F-MT_paklZ2Y-ZmDaLNwq%2Fk8s_110_Keycloak-Admin-Console.png?alt=media\&token=20ffa237-ac0d-42c3-bdd6-bff5535b3464) Next create a SAML client in Keycloak by navigating to the **Clients** section of the **Configure Realm** panel and click on create in the top right corner. ![Create SAML Client](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MTpUGY_wYhDCI7EEv9y%2F-MTpUZXmxhBMCE5VAvfL%2Fk8s_103_Keycloak-Admin-Console.png?alt=media\&token=24573d5a-eb44-4c08-96b1-aa223c5e67cb) In the next screen, click on **Select File** and choose the **Configuration Metadata** file downloaded from the Replex UI. ![Upload Medata File](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MTpUGY_wYhDCI7EEv9y%2F-MTpUhUFVG2P4n1bkGgn%2Fk8s_104_Keycloak-Admin-Console.png?alt=media\&token=f8049133-fc96-41ed-86f7-e404e4505f95) The form fields will populate automatically, once the file has been selected. Click **Save**. Make sure that the "Client Signature Required" switch is disabled. {% hint style="info" %} To ensure that the replex profile is complete, it is recommended to map users personal info (like first and last name) to saml responses. To do this, open the “Mappers” section and click “Add Builtin”. In the “Add” column select “X500 givenName” and “X500 surname” rows and click “Add selected”. This data can also be provided in custom user properties mappers under “firstName” and “lastName” SAML attributes. {% endhint %} Click **Save** again at the bottom of the page. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MTpUGY_wYhDCI7EEv9y%2F-MTpUol7CAXqxBCmfN93%2Fk8s_105_Keycloak-Admin-Console.png?alt=media\&token=d1532fe7-c84a-412b-849a-0dd81d7da092) Click on Settings in the **Configure Realm** panel and navigate to the **General** tab. Click on the SAML 2.0 Identity Provider Metadata link and save the file as idp\_metadata.xml. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MTpUGY_wYhDCI7EEv9y%2F-MTpUuQE_9BUvyi5U7X0%2Fk8s_106_Keycloak-Admin-Console.png?alt=media\&token=832a2d7d-d277-4770-a2c2-6c68376f6114) Switch to the **Add SAML** **Integration** section of the Replex UI and scroll down to the IdP Metadata section. Upload the previously saved idp\_metadata.xml file and click save. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MTpV33xPX46zMBapoqF%2F-MTpV5ZWlurs4-wnnY1o%2Fk8s_107_New-SAML-Integration-Replex.png?alt=media\&token=e6dc7d16-0593-4dae-86af-e22b4e8a57e5) This will activate the keycloak integration. To enable users to access Replex via Keycloak, add users by navigating to the **Users** section of the **Manage Realm** panel in Keycloak. ![Add Users to Keycloak](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MTvLIAZhlCuvVGWVLxG%2F-MTvNFkfivZrgBRKQzlW%2Fk8s_118_Keycloak-Admin-Console.png?alt=media\&token=d71aba24-c53b-4be7-ab9a-a408a47e9d44) ![Add Users to Keycloak](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MTvLIAZhlCuvVGWVLxG%2F-MTvMvz6i2rpEsHK9rkd%2Fk8s_117_Keycloak-Admin-Console.png?alt=media\&token=c386f556-4385-4bea-99e2-d156d5cf65ea) In the Replex UI admins can review the new Keycloak integration in the **Single Sign On (SAML)** section of the **Settings** Page. They can also edit or delete the integration using the respective icons in front of the integration. ![Review Keycloak Integration](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MTvNaM1PlUmd49KO6k-%2F-MTvNmBFFHHXqzh8MxFu%2Fk8s_119_Single-Sign-On-SAML-Replex.png?alt=media\&token=f0d655d9-5a49-4290-b3d6-11a9cf9fbb99) ### **On-Prem installation** For on-prem installations the following environment variables are required to make SAML auth work properly:‌ * SAML\_DECRYPTION\_PRIVATE\_KEY * SAML\_DECRYPTION\_PUBLIC\_CERT * SAML\_SIGNING\_PRIVATE\_KEY * SAML\_SIGNING\_PUBLIC\_CERT‌ These variables are used to generate Service Provider metadata and to sign SAML requests.‌ `SAML_DECRYPTION_PRIVATE_KEY` and `SAML_DECRYPTION_PUBLIC_CERT` are used to share encrypted SAML messages. In most cases encryption is not used, but in cases where it is needed these two variables take in place. During the SAML setup, the IdP receives Public Cert (from SP Metadata) so it can encrypt SAML message with it, and then on SP side ‌it could be decrypted using Private Key. \ `SAML_DECRYPTION_PRIVATE_KEY` is a rsa sha256 private key.‌\ `SAML_DECRYPTION_PUBLIC_CERT` is a pair to the private key in x509 certificate format.‌ `SAML_SIGNING_PRIVATE_KEY` and `SAML_SIGNING_PUBLIC_CERT` are used to share signed SAML Authentication requests to the IdP. This is a common practice to provide trusted info about the issuer of the request (SP). \ `SAML_SIGNING_PRIVATE_KEY` is a rsa sha256 private key.‌\ `SAML_SIGNING_PUBLIC_CERT` is a pair to the private key in x509 certificate format.‌ First of all, the keys must be generated, e.g. using an openssl cli util:\ \ *For decryption:*\ `openssl req -x509 -nodes -sha256 -days 3650 -newkey rsa:2048 -keyout decryption_private_key.pem -out decryption_public_certificate.pem`*For encryption:*`openssl req -x509 -nodes -sha256 -days 3650 -newkey rsa:2048 -keyout signing_private_key.pem -out signing_public_certificate.pem`‌ These two commands will create 4 files: * decryption\_private\_key.pem * decryption\_public\_certificate.pem * signing\_private\_key.pem * signing\_public\_certificate.pem Next step is to encode previously generated keys with base64 encoding. Following cli command can be used:‌ `base64 -w 0 {KEY_NAME}.pem`‌, where `KEY_NAME` is name of the key generated in the previous step.\ After encoding the 4 base64 encoded strings should be generated.\ Example:\ `>>> base64 -w 0 decryption_private_key.pem`\ `>>> LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJn...` Lastly update or create secrets file `server-secret.yaml` to add these values.\ If file was not created before, create it, and add following values from `data` so the final secrets file wil look like: \ `apiVersion: v1`\ `kind: Secret`\ `metadata:`\ `name: replex-rsa-keys`\ `type: Opaque`\ `data:`\ `signingPrivateKey: `\ `signingPublicCert: `\ `decryptionPrivateKey: `\ `decryptionPublicCert: ` {% hint style="info" %} Make sure that Secret name is the same as the on used in Deployment {% endhint %} Apply the secrets by using `kubectl apply`:\ `kubectl apply -f ./server-secret.yaml -n {REPLEX_SERVER_NS}`, where `REPLEX_SERVER_NS` is the namespace where Replex Server is deployed. # Custom Cost The **Custom Cost** section of the **Settings Page** allows admins to edit, review, delete and add custom cost models to their Kubernetes environments. This functionality is especially useful for organisations running Kubernetes environments on privately hosted infrastructure, on premises or private cloud. The Custom Cost feature supports both regular **Node Types** as well as **GPUs**. Admins can add **Node Types** with custom **CPU cores** and **RAM capacity** reflecting their private infrastructure. In addition, admins can also add a **Total Cost/h**, **CPU Cost/h** and **RAM Cost/h** for their custom nodes. Custom GPU types with specific **Models**, **Regions** and **Cost/h** can also be added. The ability to add custom node and GPU types enables Replex to provide accurate cost allocation and chargeback for privately hosted Kubernetes environments. ### Add Custom Node Type To add a new custom node, click on **+ Add Node Type** in the top right of the **Node Types** section. On the next screen, name the new node type and optionally input the **Operating System** and **Region** of the node. Next toggle between **Total Cost** and **Split Cost** to choose whether to input CPU and RAM costs for that node combined or individually. When entering combined costs (Total Cost) you are required to enter **Total Cost/h**, the **Number of CPU Cores** and **Gigabyte of RAM** for the custom node. ![Add New Custom Node Type](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-McydXTsBZ0M_QSM3n0m%2F-Mcye-H8cMNBRL4WInW_%2FNew-Node-Type-Replex.png?alt=media\&token=eccecac4-0a09-4131-b72d-b1afe9eca255) Alternatively, when entering cost individually (Split Cost) you are required to enter **CPU Cost/h** and **RAM Cost/h** for that node type. You can also optionally enter the **Number of CPU Cores** and **Gigabyte of RAM**. ![Add New Custom Node Type](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Md1gfDm9QvclGhP4nc4%2F-Md1h8bUBc9jJf3nYfAP%2FNew-Node-Type-Replex%20\(2\).png?alt=media\&token=ac99d8e6-9779-4194-a391-658d998165ed) Click **Save** to create the new node type. This will create the new custom node and populate it in the **Node Types** section of the **Custom Cost** screen. ### Add Multiple Node Types Multiple node types can be added using the csv file upload functionality. When adding multiple node types, Kubernetes admins can opt to split CPU and RAM costs for those nodes and upload the cost model using the **Split Cost** csv format. Alternatively they can combine CPU and RAM costs for node types and upload the cost model using the **Total Cost** csv format. {% hint style="info" %} Split Cost csv format for adding multiple node types: {% endhint %} | type; | cpu\_cost; | ram\_cost; | cpu\_cores; | ram\_gb | operating\_system; | region | | ------------------ | ---------- | ---------- | ----------- | ------- | ------------------ | ------ | | instance\_type\_a; | 0.09; | 0.37; | 4; | 16 | linux; | west-1 | | instance\_type\_b; | 0.117; | 0.8236; | 8; | 32 | linux; | west-2 | {% hint style="info" %} Total Cost csv model for adding multiple node types: {% endhint %} | type; | total\_cost; | cpu\_cores; | ram\_gb | operating\_system; | region | | ------------------ | ------------ | ----------- | ------- | ------------------ | ------ | | instance\_type\_a; | 0.09; | 4; | 16 | linux; | west-1 | | instance\_type\_b; | 0.117; | 8; | 32 | linux; | west-2 | Once the csv has been formatted correctly click **Upload CSV** in the top right of the **Node Types** section. Click **Choose File** in the pop-up screen, choose the csv and click **Upload**. ![Upload Custom Cost CSV](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Md1CJXlmvDRAYEKgrTb%2F-Md1gYA7_ZsIHViSz3Rs%2FCustom-Cost-Replex.png?alt=media\&token=0cb23a49-d976-43c2-bc5c-50ca4d5a50db) Once uploaded, the custom node types will be populated in the **Node Types** section of the **Custom Cost** screen.. ### Edit or Remove Node Types Admins can edit previously added **Node Types** by clicking the edit icon in front of each node. ![Edit Node Type](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Md1qVrxRdRyoM9_XLcq%2F-Md1qnvP59CpZyeHsc08%2FEdit-Node-Type-Replex.png?alt=media\&token=6ec40c80-06e0-403b-8006-259ca975f81f) Previously added **Node Types** can be deleted by clicking on the delete icon in front of each node. ### Add Custom GPU Type To add a new **GPU Type**, click on **+ Add GPU Type** in the top right of the **GPU Types** section of the **Custom Cost** screen. On the next screen, enter the **Model** and **Cost/h** of the GPU type. Optionally enter the **Region of the GPU**. ![Add New GPU Type](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Md1rFiq73MPw0u5iTNs%2F-Md1y9eAK1C6J9cTsBuB%2FNew-GPU-Type-Replex.png?alt=media\&token=3d434820-fc47-4fa2-b1c5-bab55f355fb8) Click **Save** to create the new GPU type. This will create the new GPU type and populate it in the **GPU Types** section of the **Custom Cost** screen. ### Add Multiple GPU Types Multiple GPU types can be added using the csv file upload functionality. To add a new csv file, click on **Upload CSV** in the top right of the **GPU Types** section. Click **Choose File** in the pop-up screen, choose the csv and click **Upload**. ![Add Multiple GPU Types](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Md2-BG2zRAI1v05SKcs%2F-Md204ZfmdG63Mw28vFp%2FCustom-Cost-Replex%20\(2\).png?alt=media\&token=24d537a3-6e62-4d3b-87df-cfe95e76a13e) {% hint style="info" %} The uploaded csv should follow the following format: {% endhint %} | model; | cost\_hourly; | region; | | --------------------- | ------------- | ------- | | nvidia-rtx-2080;0.09; | 5.5; | west-1 | | nvidia-tesla-t4; | 2.1; | west-2 | Once uploaded the new GPU types will be populated in the **GPU Types** section. ### Edit or Remove GPU Types Admins can edit previously added **GPU Types** by clicking the edit icon in front of each GPU. ![Edit GPU Type](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-Md22-t0fCU5aegTZE3S%2F-Md229OaJdpYhTtDySV-%2FEdit-GPU-Type-Replex.png?alt=media\&token=a149bb45-dd8a-458e-91ae-664fff7ff779) Previously added **GPU Types** can be deleted by clicking on the delete icon in front of each GPU. # Cloud Costs (Out of Cluster Costs) The **Cloud Cost** functionality allows Kubernetes admins to allocate out of cluster costs to Kubernetes environments. Out of cluster costs e.g. costs for cloud services like S3**,** RDS or Lambda can be allocated to individual Kubernetes artefacts like **Namespaces** as well as **Labels**. The ability to allocate out of cluster costs to Kubernetes artefacts provides a more accurate picture of the costs of that Kubernetes artefact and by extension of the entire Kubernetes environment. The **Cloud Cost** functionality supports **AWS** and **Azure** services as of now. Additionally, out of cluster costs can be allocated to Kubernetes Namespaces and Labels. Support for other cloud providers and allocating costs to other Kubernetes artefacts like clusters, deployments, services etc is on the roadmap. Below we will walkthrough the process of allocating cloud service costs for both AWS and Azure to Kubernetes Namespaces and Labels. Let's start off with AWS. ### Allocating AWS Service Costs To allocate AWS service costs you first need to add AWS billing credentials to your Replex account. Follow [this](/settings/billing-credentials) guide to add billing credentials for AWS.\ \ Please add the following permission to the policy: \ `{`\ `"Version": "2012-10-17",`\ `"Statement": [`\ `{`\ `"Sid": "VisualEditor1",`\ `"Effect": "Allow",`\ `"Action": [`\ `...`\ `"ce:GetTags"`\ `],`\ `"Resource": "*"`\ `},`\ `...` {% hint style="info" %} In addition, the following permission must be added for the AWS cost allocation tags to work:\ CostExplorer Permission: **ce:GetTags** {% endhint %} Once the billing integration is complete we are ready to allocate AWS service costs to Kubernetes Namespaces and Labels. #### Allocating AWS Service Costs to Namespaces Below we walkthrough an example of tagging S3 on AWS and allocating it's costs to a Namespace in our Kubernetes environment. Costs for other AWS services like RDS, Lambda and ElasticCache can also be allocated by following the steps outlined below. For more advanced tagging practices please refer to [this](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) AWS tagging guide. #### Tag Resources/Services To enable the allocation of external AWS service costs to Kubernetes artefacts the cloud cost functionality utilizes the AWS tagging mechanism. To apply the relevant tag, browse to the S3 dashboard on your AWS console and click on the bucket name you want to add the tag to. Next click on the **Properties** tab in the S3 bucket screen, scroll down to the **Tag** section and click **Edit**. On the next screen click **Add Tag**. Enter the following value in the tag **Key** field: **CLOUDCOST\_TAG\_KEY\_NAMESPACE** Enter the name of the Namespace you want to add the S3 costs to in the tag **Value** field. Click **Save changes**. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MYjgGmRFyUv-6r0DG8O%2F-MYjgTtQhN-3K9B8sfHZ%2FS3-Management-Console.png?alt=media\&token=18e049e6-4f4d-4b88-a2a1-c91275bde2e8) #### Enable User Defined Cost Allocation Tags The tags applied in the previous step need to be activated in the AWS console. To do this browse to the **AWS Billing & Cost Management Dashboard**. Click on **Cost allocation tags** in the panel on the left hand side. In the next screen, under the User-defined cost allocation tags section, choose the tag added in the previous step and click on **Activate**. Please be aware that tags added in the previous step may take several hours to populate in the User-defined cost allocation tags section. #### Allocating AWS Service Costs to Labels In this section we walkthrough an example of tagging S3 on AWS and allocating its costs to a **Label** in our Kubernetes environment. Follow the steps outlined in the Namespaces example above to navigate to the **"Edit Bucket Tagging"** screen for the S3 bucket you want to allocate costs for. Enter the following in the tag **Key** field: **CLOUDCOST\_TAG\_KEY\_LABEL\_\** and **\** in the tag **Value** field. {% hint style="info" %} **K8s\_Label\_Key** and **K8s\_Label\_Value** in the example above depend on the Kubernetes Label that you want to allocate the costs to. The following tag **Key** and **Value** will be used for the Kubernetes Label **"app" : "postgres":** Tag Key: **CLOUDCOST\_TAG\_KEY\_LABEL\_app** Tag Value: **postgres** Similarly for **"tier" : "frontend"** the following tag **Key** and **Value** will be used: Tag Key: **CLOUDCOST\_TAG\_KEY\_LABEL\_tier** Tag Value: **frontend** {% endhint %} For our example we use the "app" : "postgres" label. ![](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-McPMUVtejTmUiRUF2Du%2F-McPNW0zwWKD10bUJOE2%2FS3-Management-Console%20\(1\).png?alt=media\&token=083cb6a4-6f71-4112-ba65-d32b3eca2321) Click **Save changes**. Next activate the tag we just added by navigating to the **User-defined cost allocation tags** section of the **AWS Billing & Cost Management Dashboard.** ### Allocating Azure Service Costs As with AWS, the first step in allocating Azure service costs is to add Azure billing credentials to your Replex account. Follow [this](/settings/billing-credentials) guide to add billing credentials for Azure. Once the billing integration is complete we are ready to allocate Azure service costs to Kubernetes Namespaces and Labels. #### Allocating Azure Service Costs to Namespaces Below we walkthrough an example of tagging an Azure SQL database and allocating it's costs to a Namespace in our Kubernetes environment. Costs for other Azure services can also be allocated by following the steps outlined below. #### Tag Resources/Services Navigate to the relevant SQL database on your Azure console and click on the database name you want to add the tag to. Next click on **tags** in the left hand panel and enter the following in the **Name** field: **CLOUDCOST\_TAG\_KEY\_NAMESPACE** Enter the name of the Namespace you want to allocate the SQL costs to in the tag **Value** field. Click **Apply**. ![Tag SQL Database](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-McTuJpS3LU_NmZrXqtP%2F-McU77mzguayPgv6zR00%2FCloudCostsTest-cloudcost-testserver-CloudCostsTest-Microsoft-Azure.png?alt=media\&token=0b0873b7-feb8-452f-8284-497c6dc13c88) #### Activate Azure Tag The tags applied in the previous step need to be activated in the Azure console. To do this navigate to the **Azure Cost Management + Billing** service. In the **Azure Cost Management + Billing** dashboard click on **Cost Management** in the left hand panel. In the next screen, scroll down to the **Settings** section and click on **Configuration**. In the next screen click on **Tags** and add the tag we created in the previous step. Click **Apply**. ![Activate Azure Tag](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-McU7EIEtP6pEZDvF-9M%2F-McU96KGPUdhKOBI3GzL%2FTags-Microsoft-Azure.png?alt=media\&token=5a19c41b-a829-4f50-a5a4-86b3eea2010f) #### Allocating Azure Service Costs to Kubernetes Labels In this section we walkthrough an example of tagging an Azure SQL database and allocating its costs to a **Label** in our Kubernetes environment. Follow the steps outlined in the example above to navigate to the **Tags** section of the SQL database you want to allocate costs for. Enter the following in the tag **Key** field: **CLOUDCOST\_TAG\_KEY\_LABEL\_\** and **\** in the tag **Value** field. Click **Apply**. Activate the tag we just added by navigating to the **Configuration** section of the **Azure Cost Management + Billing** dashboard**.** ### Viewing Allocated Cloud Costs Once tagged the costs for the Kubernetes **Namespaces** or **Labels** will populate in the Replex UI. External cloud costs can be viewed in the **Namespaces Overview** screen of the **Cluster Dashboard**. External cloud costs are accumulated in both the **Allocated Cost Combined** and **Allocated Cost Detailed** charts of the **Namespaces Overview**. Additionally cloud costs for each namespace are provided in the **Cost Details** section of the **Namespace Overview** in a separate **External Cost** column. ![External Cloud Costs Namespace Overview](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MYJpV3J9Nbdpca5vJsw%2F-MYK-V5AW3b8ufl3z9n4%2Fexternal-cloud-costs-1.png?alt=media\&token=deaf7c29-ba1a-4b3b-9548-daaa21c7ef5a) The **Detailed Namespace View** also provides a cost breakdown for individual out of cluster resources allocated to a Namespace. It outlines both the **Resource Type** as well as its cost for the time period chosen**.** Cost breakdowns for out of cluster resources can be viewed by clicking on **Show External Resources** in the **Detailed Namespace View.** ![Cost Breakdown for Out of Cluster Resources](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MYK0__nvF3Kwjzb22td%2F-MYK1Cp_cOTgJtziTJSK%2Fexternal-cloud-costs-2.png?alt=media\&token=4d83f4cd-c30a-4add-8434-1ff3ae94b533) External cloud costs can also be viewed on the [Teams Dashboard](/getting-started/teams-dashboard). Each individual Team dashboard outlines external cloud costs under the External tab. The **External** tab breaks down cloud costs by Resource Type as well as the Namespace/Label they are assigned to. ![External Cloud Costs Teams Dashboard](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-McUYU3KxcdA4BY2Onck%2F-McUZkEmVAhMIm-gNKsb%2FTest-Team-Replex-cost-allocation.png?alt=media\&token=4d41a1e6-37ad-4109-8144-81ad7e4a5f42) # API Tokens The **API Tokens** section of the **Settings Page** allows admins to generate API tokens with granular access control settings. Admins can also review, edit and delete already existing API tokens. API tokens allow client applications to access the Replex API. The following access controls can be configured for each API token: * Agent * Api Key * Billing * Budget * Cluster * Custom Pricing * Invitation * Organization * Recommendations * Role * Saml Integration * Scopes * Teams * User Granular permissions are provided for each access control category outlined above. ### Create API Token To create a new API token, click on **+ Add API Token** in the top right of the **API Tokens** section. Name the API token and choose the access controls that you want to be a part of that token. Access controls can be turned on/off by clicking on the toggles under each **Access Control** heading. ![Create New API Token](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MTujSSr_ggegyuIOSI6%2F-MTujuhLLGcz0_FqjYe6%2Fk8s_116_New-API-Token-Replex%20\(1\).png?alt=media\&token=74d250f0-ed31-413b-98b2-94173074e8ca) Click save to create the new API token. This will create the new **API Token.** On the next pop-up screen, make sure you copy the newly created token by clicking on the copy symbol in front of the token text field. {% hint style="info" %} This is the only time the token can be copied. In case of loss you will need to create a new token. {% endhint %} ### Edit or Remove API Tokens Admins can edit previously added **API Tokens** by clicking the edit icon in front of each token. ![Edit API Token](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MTtu4c8wpi-9re_P1Nh%2F-MTtuFetXC_Tou6fYlZV%2Fk8s_113_Edit-API-Token-Replex.png?alt=media\&token=53e7a9ae-9810-4410-9e58-077c9c03a7cf) Previously added **API Tokens** can be deleted by clicking on the delete icon in front of each token. ### Using API Tokens To use the token to access the API it must be sent as authorization header in every request. Following a curl example request with the authorization header to get information about the organisation. ``` curl --location --request GET 'https://replex.replex.io/api/v1/organizations/:OrgID' \ --header 'Authorization: Api-Key ' ``` In Postman this can be configured in the Authorization tab as follows. ![Postman API-Token configuration](https://4068579783-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MFGA2pJIFrMdGZ5i84m%2F-MlG2Xvw9tquv7DposXH%2F-MlG30z0mJ_KRigIPqpw%2FBildschirmfoto%202021-10-05%20um%2015.52.31.png?alt=media\&token=7e42e6e1-4be0-4e6e-b750-9d452cdcf762) # API Access API documentation describing endpoints and parameters needed to interact with the Replex Server. [Please access the Replex server API here](https://replex.replex.io/api/docs/#/) # Agent Docker Image Repository: docker.replex.io/replex/k8sagent-go ## 1.9.2 - 2022-02-02 * Adds pushgateway error handling when it is not available. Docker Image: `docker.replex.io/replex/k8sagent-go:1.9.2` ## 1.9.1 - 2021-11-25 * Fix missing DCGM-Exporter metrics. Docker Image: `docker.replex.io/replex/k8sagent-go:1.9.1` ## 1.9.0 - 2021-11-11 * Adds env `PROMETHEUS_GPU_METRICS_SOURCE` to specify the prometheus GPU metrics source. Options `cadvisor` for cAdvisor accelerator metrics or `dcgm` to use nvidia DCGM-exporter metrics. Default cadvisor * Adds env `PROMETHEUS_HONOR_LABELS` set the prometheus honor\_labels scrape setting. Default false. Docker Image: `docker.replex.io/replex/k8sagent-go:1.9.0` ## 1.8.0 - 2021-07-22 * Handles all offline provider sync errors and retries syncing and pushing metrics once available * Improves thanos UX for better error handling * Removes env `PROMETHEUS_TSDB_BLOCK_DURATION` and defers to the thanos receive component alone for querier and store support Docker Image: `docker.replex.io/replex/k8sagent-go:1.8.0` ## 1.7.20 - 2021-07-13 * Adds env `PROMETHEUS_TSDB_BLOCK_DURATION` for Thanos querier support Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.20` ## 1.7.19 - 2021-06-16 * Bug fixes and improvements Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.19` ## 1.7.18 - 2021-06-01 * Bug fixes and improvements Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.18` ## 1.7.17 - 2021-05-31 * Adds a new optional environment variable: `PROMETHEUS_BEARER_TOKEN` to set bearer token for prometheus server requests. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.17` ## 1.7.16 - 2021-05-18 * Fix bug in Alicloud detection Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.16` ## 1.7.15 - 2021-05-18 * Bug fixes and improvements Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.15` ## 1.7.14 - 2021-05-10 * Add Thanos support. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.14` ## 1.7.13 - 2021-02-15 * Add Alibaba cloud support. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.13` ## 1.7.12 - 2021-01-14 * Adds node CPU usage support for Stackdriver metric providers. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.12` ## 1.7.11 - 2020-12-07 * Sets the default Prometheus node label to `node` (previously `instance`). Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.11` ## 1.7.10 - 2020-11-10 * Bug fix in node CPU usage calculation. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.10` ## 1.7.9 - 2020-11-09 * We now use actual node CPU usage instead of the sum of the container CPU usage. This helps to account for CPU usage by non-container workload. * Adds support for failed metrics synchronization on disk. * By setting the environment variable `METRICS_CACHE_DISK` to either `true/false`, you can specify whether the metrics are cached on disk or in memory. By default, `true`. * If metrics are cached on disk, you can specify the path on disk by setting the environment variable `METRICS_CACHE_DISK_DIR`. By default, `/data/metrics`. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.9` ## 1.7.8 - 2020-10-19 * Adds a new optional environment variable: `METRICS_RETRY_INTERVAL_SECONDS` to set interval between retry to push metrics. * Adds metric caching if push to the pushgateway fails and auto-retries the push once per value from `METRICS_RETRY_INTERVAL_SECONDS` or 300 seconds by default. * Adds two new metrics `replex_agent_retry_cache_size` and `replex_agent_failed_metrics_total`. First one stands for the amount of metrics that are waiting to be re-sent, second one represents the total count of once failed metric push. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.8` ## 1.7.7 - 2020-10-14 * Adds a new optional environment variable: `METRICS_FILESYSTEM` to specify the filesystem metric source. Options: `cadvisor`, `csi`. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.7` ## 1.7.6 - 2020-10-06 * Adds a new optional environment variable: `USE_CONTROL_PLANE_COST` to track costs of the Kubernetes Control Plane. Set to `true/false` to enable/disable the feature. * Adds node local disk support for Datadog metric provider. * Adds Daemon- and StatefulSet support for Instana. * Adds the collection of node provider id metric. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.6` ## 1.7.5 - 2020-09-25 * Adds a new optional environment variable: `CLOUD_PROVIDER_OVERRIDE` to manually overwrite the cloud provider. Set to `aws, azure, gce, custom` choose the provider. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.5` ## 1.7.4 - 2020-09-18 * Disabled logs for pods created within the last 5 minutes and for already succeeded pods. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.4` ## 1.7.3 - 2020-09-16 * Adds a new optional environment variable: `PROMETHEUS_NODE_LABEL` to specify the label that represents the node in the Prometheus metrics. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.3` ## 1.7.2 - 2020-09-16 * Skip pod metrics if node is not ready yet. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.2` ## 1.7.1 - 2020-09-15 * Adds a new optional environment variable: `ONLY_USE_READY_NODES` to track only nodes that are in "Ready" state. Set to `true/false` to enable/disable the feature. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.1` ## 1.7.0 - 2020-09-07 * Adds node local disk support for Prometheus metric provider. Docker Image: `docker.replex.io/replex/k8sagent-go:1.7.0` ## 1.6.1 - 2020-08-19 * Renaming the label that represents the node in the Prometheus metrics from `node` to `instance`. Docker Image: `docker.replex.io/replex/k8sagent-go:1.6.1` ## 1.6.0 - 2020-07-28 * Adds Instana topology provider support. * Adds a new optional environment variable: `KUBERNETES_INFO_PROVIDER` to set the topology provider. Set to `kubernetes, instana` to choose between Kubernetes and Instana. * Adds a new optional environment variable: `INSTANA_CLUSTER_ID` to set the Instana cluster id. Docker Image: `docker.replex.io/replex/k8sagent-go:1.6.0` ## 1.5.0 - 2020-06-26 * Adds the collection of cluster and Prometheus version. Docker Image: `docker.replex.io/replex/k8sagent-go:1.5.0` ## 1.4.2 - 2020-06-02 * Datadog: using `kubernetes.memory.working_set` for container memory metrics * Instana: using `memory.usage` of docker plugin for container memory metrics. Docker Image: `docker.replex.io/replex/k8sagent-go:1.4.2` ## 1.4.1 - 2020-05-27 * Renaming environment variable from `INSTANA_URL` to `INSTANA_BASE_URL` Docker Image: `docker.replex.io/replex/k8sagent-go:1.4.1` ## 1.4.0 - 2020-05-25 * Adds support for Instana metrics provider. * Skipping unmounted PVCs. Docker Image: `docker.replex.io/replex/k8sagent-go:1.4.0` ## 1.3.0 - 2020-02-29 * Adds support for Stackdriver \\(Google Operations\\) metrics provider. Docker Image: `docker.replex.io/replex/k8sagent-go:1.3.0` # Aggregator Docker Image Repository: docker.replex.io/replex/aggregator ## 1.0.8 - 2021-08-25 * Fix bug in aggregating past metrics. Docker Image: `docker.replex.io/replex/aggregator:1.0.8` ## 1.0.7 - 2021-08-23 * Bug fix in aggregation. Docker Image: `docker.replex.io/replex/aggregator:1.0.7` ## 1.0.6 - 2021-08-23 * Performance improvements in aggregations. * Other bug fixes and improvements. Docker Image: `docker.replex.io/replex/aggregator:1.0.6` ## 1.0.5 - 2021-06-01 * Add support for GPU costs. * Record database growth over time. * Bug fixes and improvements. Docker Image: `docker.replex.io/replex/aggregator:1.0.5` ## 1.0.4 - 2021-04-20 * Bug fixes and improvements Docker Image: `docker.replex.io/replex/aggregator:1.0.4` ## 1.0.3 - 2021-02-22 * Fix budget PVC cost aggregation. Docker Image: `docker.replex.io/replex/aggregator:1.0.3` ## 1.0.2 - 2020-12-18 * Bug fixes and improvements Docker Image: `docker.replex.io/replex/aggregator:1.0.2` ## 1.0.1 * Bug fixes and improvements Docker Image: `docker.replex.io/replex/aggregator:1.0.1` ## 1.0.0 * Adds pod metric aggregation. * Adds disk metric aggregation. * Adds control plane metric aggregation. * Adds a new optional environment variables: `HOURLY_INTERVAL_MINUTES` and `DAILY_INTERVAL_MINUTES` to configure hourly and daily aggregation intervals. Set to positive number to specify minutes. * Adds pod metric aggregation. * Adds a new optional environment variable: `LOG_LEVEL` to configure the logging level. Set to positive number to specify log level. * Aggregating node, container and pvc metrics on a hourly and daily base. * Adds optional `SERVER_HOST` environment variable for budgets Docker Image: `docker.replex.io/replex/aggregator:1.0.0` # Frontend ## 1.6.10-beta ### Added ### Changed * Improved team list * Improved budget feature * Namespace tables get shown before external costs are fully loaded Docker Image: `docker.replex.io/replex/client-vue-prototype:1.6.10-beta` ## 1.6.9 ### Added * Added default currency selection to tenant creation form * Added utilized and idle costs to namespaces table on the team dashboard * Added CPU and RAM recommendation to pods list * Added notice to cluster deletion dialog that the cluster ID cannot be reused for new clusters * Added consistent blank slates ### Changed * Costs are now displayed in the defaultCurrency of the organization * Costs less than one cent will have a tooltip for the exact value * Improved currency formatting for input fields * Improved cluster list loading time by not loading cluster efficiency via a separate call for each cluster * Navigation bar is now scrollable * Status column in billing credentials table now shows either Active or Error with the error description in a tooltip ### Fixed * Fixed a bug that led to spaces in downloaded CSV file names * Fixed NaN values in cost allocation tables and CSV downloads when total cost was 0 * Fixed false positive infobox for outdated cluster agents Docker Image: `docker.replex.io/replex/client-vue-prototype:1.6.9` ## 1.6.8 ### Added * Added tooltip showing the exact value for rounded custom cost values * Added option to add custom costs via a total cost value instead of separate CPU and RAM costs * Added cloud cost support for labels on team dashboard * Added GPU costs to node list, team dashboard and cluster efficiency panel * Added settings page for Alibaba billing credentials * Added GPU custom costs ### Changed * Cost input fields for custom costs now allow up to 10 decimal places * Cost Chart Grid sorting buttons are now on the right-hand side ### Fixed * Custom cost CPU and RAM costs were incorrectly labeled "per Core" and "per GB" * Changing date range on team dashboard did not update cloud costs * Cloud cost error notification was also shown when cloud costs were disabled Docker Image: `docker.replex.io/replex/client-vue-prototype:1.6.8` ## 1.6.7 ### Added * Added note regarding comma & semicolon support to custom cost CSV upload dialog * Added CSV download when there are more than 100 custom cost entries * Added sticky header with breadcrumbs and upper right action items * Added "DB Usage" page * Added GPU costs to cost allocation tables and bar charts * Added clusters and namespaces filter selection to cross cluster namespaces overview * Added number of unique nodes in the selected time frame to Kubernetes cluster list ### Changed * The dates in the Budget History panel now mark the end of each period instead of the beginning * Bar charts in the Kubernetes dashboards now show PVC and node disk costs separately * Changed grouping and filter selection design on Kubernetes dashboards * Charts show no line or bar for periods without any data ### Fixed * Charts sometimes were not sized properly on initial render in Firefox * Donut chart panels on the Kubernetes cluster dashboard were empty for a queued cluster * Cost allocation tables failed to load when the cloud cost endpoint returned an error * Cluster list showed the update-agent notice when the installed agent is newer than the most recent agent * Team stats panel layout was broken when the chart next to it failed to load Docker Image: `docker.replex.io/replex/client-vue-prototype:1.6.7` ## 1.6.6 ### Added * Added cloud cost support for namespaces * Added 404 handling for all pages * Added min, max usage and standard deviation for cpu and ram to pod cost allocation table * Added support for time selection on the date range picker ### Changed * List of clusters now displays data for selected date range * Load list of cluster namespaces from an optimized endpoint for the filter creation dialog * Changed charts y-axes to always start and end with a tick mark * Changed a line chart's line to span the whole width of the chart area ### Fixed * A redeployment triggered an error page for certain users Docker Image: `docker.replex.io/replex/client-vue-prototype:1.6.6` ## 1.6.5 ### Added * Added tooltip showing the total cost for nodes * Added pagination controls to the custom cost page * Added CSV download to cost the tables on the kubernetes dashboards * Added 404 page * Added "More" navigation item with links to documentation, support and logout * Added "available in" format for current cost on Budgets List * Added sorting for cluster column on nodes list * Added support for "operating system" and "region" to custom cost ### Changed * Changed navigation icon for Clusters, Namespaces and Nodes * Changed breadcrumb position to be further away from the headline * Replex logo links to the cluster list * Changed the cost in the teams namespace list to singular ### Fixed * Filter items did not properly wrap into new lines Docker Image: `docker.replex.io/replex/client-vue-prototype:1.6.5` ## 1.6.4 ### Added * Added user scope checks to budgets area * Added info in the Current Cost panel, when current cost is not available yet * Added option to enable One Way Login if at least one SAML integration exists * Added Alibaba cloud icon * Added empty state info to Replex panels * Added user scope checks to teams area * Added API error message handling for API token form inputs * Added download button for custom cost example CSV file ### Changed * Changed headline design * Changed breadcrumb design * Changed date picker design * Changed design and position of the context menu for teams and budgets * Tabs on the team page are never hidden ### Fixed * SAML login links were not shown for single tenancy setups Docker Image: `docker.replex.io/replex/client-vue-prototype:1.6.4` ## 1.6.3 ### Added * Added tooltip to private option toggle for teams * Added signing cert input for SAML integration ### Changed * Changed pagination design Docker Image: `docker.replex.io/replex/client-vue-prototype:1.6.3` ## 1.6.2 ### Added * Added option to sort by idle cost for the cost development tab on cost allocation dashboards * Added allocated cost chart to team page * Added user scope checks to kubernetes dashboards * Added user scope checks to global namespaces list * Added user scope checks to global nodes list * Added CSV upload option to custom cost settings Docker Image: `docker.replex.io/replex/client-vue-prototype:1.6.2` ## 1.6.1 ### Added * Added Replex Agent Setup page to the settings area * Added user scope checks to settings area * Added option to sort by utilized and idle cost in cost allocation tables * Added indicator for private teams * Added success notification when a team was set to private or public ### Changed * Changed tenant manager navigation icon * Changed navigation design * Changed api key scope list to include all possible scopes * Sorting cost allocation tables by a column that represents a cost value defaults to descending order * Improved scaling behavior of line and bar charts ### Removed * Removed deprecated tenant information from tenant manager * Removed page header ### Fixed * Team was not saved properly when only the private option changed * Login form had the autofocus on the wrong input field in some cases * Item count in the Cost Details panel did not update in some cases Docker Image: `docker.replex.io/replex/client-vue-prototype:1.6.1` ## 1.6.0 ### Added * Load user scopes from API * Added tooltip to cost development charts hinting the associated namespace * Added tenant manager ### Changed * Restructured cost allocation tables on kubernetes dashboards * Dropdown menus and success messages now use a dark theme * Clicking an API token name does not link to its edit page anymore Docker Image: `docker.replex.io/replex/client-vue-prototype:1.6.0` ## 1.5.4 ### Changed * Update API token scope names Docker Image: `docker.replex.io/replex/client-vue-prototype:1.5.4` ## 1.5.3 ### Added * Applied dashboard filters (using OR) are now stored in the URL * Added icon on linked panels on cluster overview dashboard to indicate that they are clickable * Added the option to make teams private * Added cost development chart grid to the cost details section on kubernetes dashboards * Added tooltip to labels tab on the teams page explaining total cost calculation ### Changed * Increase time before showing cluster agent inactivity warning to 10 minutes Docker Image: `docker.replex.io/replex/client-vue-prototype:1.5.3` ## 1.5.2 ### Added * Added signup page * Added Servicebot Portal * Added support for a one-time token to auth token exchange on the login page * Added SAML integration Docker Image: `docker.replex.io/replex/client-vue-prototype:1.5.2` ## 1.5.1 ### Added * Labels list on the teams page shows costs per cluster ### Changed * Names of labels found for only one cluster on the teams page now link to the corresponding kubernetes cost allocation dashboard * Highlight pods with a low CPU usage on kubernetes dashboards * Clusters that are marked for deletion can no longer be assigned to billing integrations or teams * Display workload names in plural on kubernetes dashboards Docker Image: `docker.replex.io/replex/client-vue-prototype:1.5.1` ## 1.5.0 ### Added * Added budgets tab to the teams page * Added settings area * Added billing credentials settings * Added custom cost settings * Added API tokens settings * Added number of items to the selection dropdown in the third level kubernetes cost allocation dashboard * Clicking (+X) in the labels and namespaces list for a team expands the children ### Changed * Valid input fields have no green styling anymore * Kubernetes cost allocation dashboards now support multiple filters * The button to delete a cluster can now be found on the cluster overview dashboard * Sidebar entries are now grouped and visually separated * The bar chart for the budget history now supports varying budget amounts for budgets of type "previous month's cost" Docker Image: `docker.replex.io/replex/client-vue-prototype:1.5.0` ## 1.4.1 ### Added * Added icons in budget overview list to differentiate between cluster-based and team-based budgets * Added overview panel to team page * List groups can also be expanded by clicking on the shortened children list ### Fixed * Font size in charts were not scaled properly in Firefox Docker Image: `docker.replex.io/replex/client-vue-prototype:1.4.1` ## 1.4.0 ### Added * Added Budgets feature Docker Image: `docker.replex.io/replex/client-vue-prototype:1.4.0` ## 1.3.3 ### Added * Include link to the agent changelog in the outdated agent tooltip ### Changed * Include disk cost into all storage cost values ## 1.3.2 ### Changed * Include idle cost in pod cost allocation list and show breakdown on hover tooltip ## 1.3.1 ### Fixed * Require password fields on account setup to be filled out ## 1.3.0 ### Added * Added Teams feature * Added list of ownerless pods * Added cross-cluster detail page for namespaces * Added "Resend Invite" button to invitations list * Added "Copy Invite Link" button to invitations list * Added node name to pod list * Added underutilization indicators to node list * Added indicator for outdated Replex-Agent in cluster list * Added indicator for when Replex-Agent did not report within the last 5 minutes ### Changed * Include Idle cost in charts * Adjust cluster deletion text * Replace default namespace icon with cloud provider icon in namespace lists ### Fixed * Spinner on users list stayed forever when no user exists * Charts showed error when data was empty # Pushgateway Docker Image Repository: docker.replex.io/replex/pushgateway ## 1.5.10 - 2021-08-23 * Add currency support. * Improve error handling in metrics queue. * Numerous bug fixes and improvements. Docker Image: `docker.replex.io/replex/pushgateway:1.5.10` ## 1.5.9 - 2021-06-23 * Bug fixes and improvements for Alibaba. * Improvements for disks attached to AWS nodes. * Extends support for AWS storage class types. Docker Image: `docker.replex.io/replex/pushgateway:1.5.9` ## 1.5.8 - 2021-06-16 * Adds AWS IOPS billing support. * Bug fixes. Docker Image: `docker.replex.io/replex/pushgateway:1.5.8` ## 1.5.7 - 2021-06-01 * Bug fixes and improvements. Docker Image: `docker.replex.io/replex/pushgateway:1.5.7` ## 1.5.6 - 2021-04-20 * Bug fixes and improvements. Docker Image: `docker.replex.io/replex/pushgateway:1.5.6` ## 1.5.5 - 2021-03-29 * Bug fixes and improvements. Docker Image: `docker.replex.io/replex/pushgateway:1.5.5` ## 1.5.4 - 2021-02-22 * Adds Alibaba support. * Removes the environment variables `PRIVATE_KEY_FILE` and `PUBLIC_KEY_FILE`. To use a key file, now use the path with the prefix `file:` as the key. Example: `file:path/to/file.pem`. * Bug fixes and improvements. Docker Image: `docker.replex.io/replex/pushgateway:1.5.4` ## 1.5.3 - 2020-12-18 * Bug fixes and improvements Docker Image: `docker.replex.io/replex/pushgateway:1.5.3` ## 1.5.2 * Adds version to exposed metrics. * Adds Billing integration logic. * Adds Billing API integration. * Fix archive metrics when saving failed. * Improved error handling for `/push` route. * Adds spot instances support. * Adds billing status support. * Adds reserved instance support. * Adds local node disk support. * Adds GCP sustained-use discounts support. * Adds Azure pricing discount support. * Adds control plane cost support. * Improved node utilization calculations. Docker Image: `docker.replex.io/replex/pushgateway:1.5.2` ## 1.5.1 - 2020-07-09 * Adds a new optional environment variable: `METRICS_ARCHIVE` to set the name of the Google Cloud storage bucket to archive metrics. Can be set to `disable` to disable this feature. Default disabled. * Fix wrong pod metric assignment. Docker Image: `docker.replex.io/replex/pushgateway:1.5.1` ## 1.5.0 - 2020-06-26 * Adds Azure pricing support. * Adds a new environment variable: `PRICING_API_MODE` to specify which pricing information's are being synced. Set to``all`, `disable`` or any number of ``[`gcp`, `azure`, `aws`, `custom`]`` comma-separated. * Adds custom instances handling for GCP. * Fix Azure storage pricing. * Adds cluster-based custom pricing. * Adds support for cluster and metrics provider version. Docker Image: `docker.replex.io/replex/pushgateway:1.5.0` ## 1.4.1 - 2020-04-16 * Adds a new environment variables: `PRICING_CPU_CORE_HOUR, PRICING_RAM_GB_HOUR, PRICING_STORAGE_GB_HOUR` to set default prices for CPU, Memory and Storage. Docker Image: `docker.replex.io/replex/pushgateway:1.4.1` ## 1.4.0 - 2020-04-16 * Adds support for custom pricing. * Adds currency support. * Adds a new environment variable: `PRICING_API_KEY` to set pricing API key. Docker Image: `docker.replex.io/replex/pushgateway:1.4.0` ## 1.3.0 - 2020-03-19 * Fix pvc hourly cost calculation. * Adds default pvc pricing for AWS and GCP cloud provider. Docker Image: `docker.replex.io/replex/pushgateway:1.3.0` ## 1.2.3 - 2020-03-04 # Server Docker Image Repository: docker.replex.io/replex/server ## 1.6.10 - 2021-08-23 * Bug fixes and improvements Docker Image: `docker.replex.io/replex/server:1.6.10` ## 1.6.9 - 2021-08-23 * Add new workload recommendations. * Performance improvements for teams and budgets. * Numerous bug fixes and improvements Docker Image: `docker.replex.io/replex/server:1.6.9` ## 1.6.8 - 2021-06-23 * Adds idle costs to team namespaces. * Adds GPU custom pricing support. * Fixed an issue with sending e-mails. * Adds support for assigning external cloud costs to Kubernetes labels. * Numerous bug fixes and improvements. Docker Image: `docker.replex.io/replex/server:1.6.8` ## 1.6.7 - 2021-06-01 * Deprecates `labels` and `namespaces` query parameters in cluster stats endpoints. * Introduces new `filters` query parameter instead of `labels` and `namespaces`. * Add support for GPU pricing. * Enhancements for custom costs management. * Add support for external cloud costs. * Numerous bug fixes and improvements. Docker Image: `docker.replex.io/replex/server:1.6.7` ## 1.6.6 - 2021-04-20 * Bug fixes and improvements. Docker Image: `docker.replex.io/replex/server:1.6.6` ## 1.6.5 - 2021-03-29 * Bug fixes and improvements. Docker Image: `docker.replex.io/replex/server:1.6.5` ## 1.6.4 - 2021-02-22 * A lot of Bug fixes and improvements. * Updates email design to replex CI. * Improves filtering by label and namespaces. Filters can now be combined by AND/OR. * Adds team cost time series endpoint. * Adds private teams feature. * Adds SSO (Single Sign On) support using SAML. For configuration the following [environment variable](https://docs.replex.io/concepts/server#environment-variables) are added: `SAML_DECRYPTION_PRIVATE_KEY`, `SAML_DECRYPTION_PUBLIC_CERT`, `SAML_SIGNING_PRIVATE_KEY`, `SAML_SIGNING_PUBLIC_CERT`. * Fix issues when accessing API with API token. Docker Image: `docker.replex.io/replex/server:1.6.4` ## 1.6.3 - 2020-12-18 * SAML integration * Pod label updates Docker Image: `docker.replex.io/replex/server:1.6.3` ## 1.6.2 - 2020-11-25 * Support multiple label filters with logically `OR`. Docker Image: `docker.replex.io/replex/server:1.6.2` ## 1.6.1 - 2020-11-19 * Adds new [environment variable](https://docs.replex.io/concepts/server#environment-variables): `LOG_LEVEL` for configuring log verbosity. * Imporovements to billing API integration and custom costs. Docker Image: `docker.replex.io/replex/server:1.6.1` ## 1.6.0 - 2020-10-08 * Adds custom user roles. * Adds API Key authorization. * Adds API Key- and user-role-scopes. * Fix idle cost calculations. * Fix pod stats cost calculations. * Adds Teams feature. * Adds node metrics endpoint. * Adds Budget feature. Docker Image: `docker.replex.io/replex/server:1.6.0` ## 1.5.2 - 2020-07-14 * Adds a new environment variables: `PRICING_API_KEY` and `PRICING_API_HOST` pricing API. Docker Image: `docker.replex.io/replex/server:1.5.2` ## 1.5.1 - 2020-07-09 * Adds Agent version endpoint. * Adds Invite feature. Docker Image: `docker.replex.io/replex/server:1.5.1` ## 1.5.0 - 2020-06-26 * Adds a new environment variables: `HOST_URL` to set host url for on-premise installations. * Adds password recovery endpoint. * Adds a new environment variables: `SECRET` used for internal encrypting. * Adds idle cost support. Docker Image: `docker.replex.io/replex/server:1.5.0` ## 1.4.0 - 2020-04-29 * Adds currency support. Docker Image: `docker.replex.io/replex/server:1.4.0` ## 1.3.2 - 2020-04-05 * Adds custom pricing support. Docker Image: `docker.replex.io/replex/server:1.3.2` ## 1.3.1 - 2020-03-19 * Fix Memory cost. * Fix namespace filter in workload stats. * Adds and improves multiple filter options. * Adds universal superadmin login. Docker Image: `docker.replex.io/replex/server:1.3.1` ## 1.3.0 - 2020-03-19 * Improves label filtering. * Docker Image: `docker.replex.io/replex/server:1.3.0` ## 1.2.3 - 2020-03-04 # PricingAPI ## 1.2.10 - 2021-08-23 * Add multi-currency support for Azure. * Add GPU costs support. * Numerous bug fixes and improvements. Docker Image: `docker.replex.io/replex/pricingapi:1.2.10` ## 1.2.9 - 2021-06-23 * Adds Alibaba billing support. * Adds AWS attached node disk information to billing. * Extends support for AWS storage class types. * Bug fixes and improvements. Docker Image: `docker.replex.io/replex/pricingapi:1.2.9` ## 1.2.8 - 2021-06-08 * Fix Azure and AWS cloud cost support. Docker Image: `docker.replex.io/replex/pricingapi:1.2.8` ## 1.2.7 - 2021-06-01 * Improve support for external cloud costs. * Bug fixes and improvements. Docker Image: `docker.replex.io/replex/pricingapi:1.2.7` ## 1.2.6 - 2021-05-05 * Bug fixes and improvements. Docker Image: `docker.replex.io/replex/pricingapi:1.2.6` ## 1.2.5 - 2021-04-20 * Support for external cloud costs. Docker Image: `docker.replex.io/replex/pricingapi:1.2.5` ## 1.2.4 - 2021-03-29 * Bug fixes and improvements. Docker Image: `docker.replex.io/replex/pricingapi:1.2.4` ## 1.2.3 - 2021-02-22 * Adds endpoints to get available alibaba instance types for given cluster. Docker Image: `docker.replex.io/replex/pricingapi:1.2.3` ## 1.2.2 - 2020-09-25 * Adds API documentation. On `/api/docs` * Adds endpoint to update billing integration. * Auto-assign free clusters to first default billing integration. * Adds billing integration status indicator. * Adds support for Alibaba cloud provider. * Adds endpoints to get available instance types for given cluster. Docker Image: `docker.replex.io/replex/pricingapi:1.2.2-b6e685dd` ## 1.2.1 - 2020-09-23 * Adds a new environment variables: `PRICING_API_MODE` to enable cloud providers. * Adds support for control plane costs. * Adds support for uploading custom pricing files. Docker Image: `docker.replex.io/replex/pricingapi:1.2.1-0c3f776d` ## 1.2.0 - 2020-07-14 * Adds version metrics. * Adds Azure billing support. Docker Image: `docker.replex.io/replex/pricingapi:1.2.0-04395b88`