# Welcome to Steadybit Steadybit provides visibility into your systems to proactively identify resilience issues and reduce downtime. We follow the principles of chaos engineering and help you stay in control of your chaos experiments. ## Getting started with Steadybit We use an agent-based approach consisting of a central SaaS platform - being the center of control for you - and agents - deployed in your system to discover running hosts, containers and applications. If you are new to resilience engineering, use our "Quick Start" to begin and run your first assisted experiments. If you are a pro already, look at the detailed docs and dive into the depth of our software. [Contact us](https://www.steadybit.com/contact) if you need help. # First Steps You are completely new to Steadybit? Excellent, this section is just right for you! Below you find useful instructions to make your start as smooth as possible. ## Start using Steadybit {% hint style="info" %} Want to explore Steadybit without installing anything? Use the [built-in sample data](/quick-start/sample-data) to explore Steadybit. If you don't want to get started on your own system, you can also use the [sample application on Minikube or AWS EKS](/quick-start/deploy-example-application). {% endhint %} 1. [Sign up for a new account](https://signup.steadybit.com/) for your own Steadybit tenant 2. [Install agents and extensions](/quick-start/set-up-agents) to discover your system *or* use [built-in sample data](/quick-start/sample-data) to explore Steadybit 3. Start using Steadybit by [designing and running your first experiment](/quick-start/run-experiment) ## Looking for more advanced steps? If you feel comfortable and want to take the next steps, how about the following: * [How to install more agents](/install-and-configure/install-agent) * [Invite more users and group them to teams](/install-and-configure/manage-teams-and-users) * [Structure your system by setting up Environments](/install-and-configure/manage-environments) # Explore with Sample Data Want to explore Steadybit without installing agents or extensions into your environment? You can use sample data to get a hands-on experience with Steadybit's features. ## What is Sample Data? Sample data is a pre-configured dataset that simulates a realistic Kubernetes-based microservice application. It allows you to: * Explore a pre-configured [service](/use-steadybit/services) and explore Steadybit's service-centric view of your reliability work and risk calculation * Review [experiment designs](/use-steadybit/experiments/design) and see how chaos experiments are structured * Analyze [past experiment runs](/use-steadybit/experiments/run) and their results * Explore targets and environments in the [landscape](/use-steadybit/explorer/landscape) * Learn about [reliability advice](/use-steadybit/explorer/advice) without deploying anything All sample data is automatically assigned to a **Sample** [environment](/install-and-configure/manage-environments) in your Steadybit tenant. {% hint style="info" %} Sample data is designed for exploration and learning. The experiments cannot be executed since they target simulated infrastructure. To run real chaos experiments, you'll need to install agents and extensions in your environment. {% endhint %} ## Sample Data Content ### Service: Online Shop The central piece of the sample data is a pre-configured [service](/use-steadybit/services) named **Online Shop**. The service allows a single view into your reliability work, bundling your service's targets, defines healthiness of your service, and all experiments validating different reliability scenarios. The **Online Shop** service is configured with: * [**Target Scope**](/use-steadybit/services#target-scope) — Kubernetes resources of the services `gateway`, `hot-deals`, and `toys-bestseller` in the Sample environment, resolved via the query `k8s.service.name IN ("gateway", "hot-deals", "toys-bestseller")`. See [targets below](#targets) * [**Validations**](/use-steadybit/services#validations) — a periodic HTTP check that verifies the shop's endpoint returns `2xx` while experiments run, defining what 'healthy' means for the **Online Shop** service * [**Provided Experiments**](/use-steadybit/services#provided-experiments) — ready-to-run [provided experiments](/use-steadybit/services#provided-experiments) automatically generated from the linked [service profile](/install-and-configure/manage-service-profiles), covering Redundancy, Dependency, and Scalability categories * [**Custom Experiments**](/use-steadybit/services#custom-experiments) — linked [custom experiments](/use-steadybit/services#custom-experiments) demonstrating advice-based and manually-designed reliability scenarios * [**Advice**](/use-steadybit/services#advice) — reliability recommendations surfaced for the targets within the service's scope Based on this information, Steadybit calculates the associated reliability [risk](/use-steadybit/services#risk) to help you prioritize and communicate your reliability work. #### Provided Experiments Provided experiments are automatically generated from the service profile and scoped to Online Shop's targets and validations: | Experiment | Category | Description | | --------------------------------------- | ----------- | ---------------------------------------------------------------------------------------- | | **Pod Redundancy** | Redundancy | Gradually reduces the redundancy of the service to verify it still provides its services | | **Unavailable Downstream Dependency** | Dependency | Blocks a downstream dependency of the service's container to verify graceful degradation | | **Fill Container Memory Progressively** | Scalability | Gradually fills the memory of the service's containers to verify graceful handling | | **Stress Container CPU Progressively** | Scalability | Gradually stresses the CPU of the service's containers to verify graceful handling | #### Custom Experiments Custom experiments are linked to the service to keep all relevant reliability work in one place. Most of these are generated from [reliability advice](/use-steadybit/explorer/advice), which automatically creates validation experiments based on your infrastructure configuration. | Experiment | Category | Related Advice / Template | Description | | ---------------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Gateway survives unavailability of third-party service** | Dependency | [Third-Party Service Is Unavailable for a Kubernetes Deployment](https://hub.steadybit.com/template/kubernetes-deployment.third-party-service-unavailable?freeText=third-party+service) | Tests whether the gateway deployment continues to function when a dependent service (`toys-bestseller`) becomes unavailable by blocking container network traffic | | **Zone Outage of eu-central-1a for gateway** | Redundancy | [Schedule Pods Across Zones](https://hub.steadybit.com/advice/com.steadybit.extension_kubernetes.advice.single-zone) | Simulates an availability zone outage to verify that traffic is properly routed to healthy pods in other zones and pods recover within 60 seconds | | **Zone Outage of eu-central-1a for toys-bestseller** | Redundancy | [Schedule Pods Across Zones](https://hub.steadybit.com/advice/com.steadybit.extension_kubernetes.advice.single-zone) | Same zone outage scenario targeting the `toys-bestseller` service to verify zone redundancy | | **Memory Overload of toys-bestseller** | Scalability | [Limit Memory Resources](https://hub.steadybit.com/advice/com.steadybit.extension_kubernetes.advice.k8s-memory-limit) | Fills container memory to 80% capacity to verify the application handles memory pressure gracefully, including proper OOM handling and recovery | | **Unhealthiness of toys-bestseller is detected** | Redundancy | [Probes Configured](https://hub.steadybit.com/advice/com.steadybit.extension_kubernetes.advice.k8s-probes) | Verifies that Kubernetes detects unhealthy containers via health probes, restarts them, and routes traffic appropriately during recovery | ### Experiment Runs The sample data includes a history of experiment runs showing: * **Completed runs**: Experiments that passed all validations, demonstrating reliability (e.g. **Zone Outage of eu-central-1a for toys-bestseller**) * **Failed runs**: Experiments that detected issues (e.g., "Check failure"), showing how Steadybit identifies reliability problems (e.g. **Gateway survives unavailability of third-party service**) This allows you to understand what is going on in your system and analyze turbulent conditions. ### Advice Sample data includes advice definitions that help identify reliability improvements for your Kubernetes workloads. Each advice provides actionable guidance and can generate validation experiments: | Advice | Description | | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | [**Limit Memory Resources**](https://hub.steadybit.com/advice/com.steadybit.extension_kubernetes.advice.k8s-memory-limit) | Identifies containers without memory limits configured, which could affect other pods on the same node | | [**Probes Configured**](https://hub.steadybit.com/advice/com.steadybit.extension_kubernetes.advice.k8s-probes) | Checks whether readiness and liveness probes are properly configured to enable Kubernetes health management | | [**Schedule Pods Across Zones**](https://hub.steadybit.com/advice/com.steadybit.extension_kubernetes.advice.single-zone) | Identifies workloads running in a single availability zone that could be affected by zone outages | Browse all available advice in the [Steadybit Reliability Hub](https://hub.steadybit.com/advice). ### Targets Underpinning the Online Shop service, the sample data includes a simulated Kubernetes environment representing an e-commerce "shop" application running on the `shop-sample` cluster in the `shop` namespace. The targets include: * **Containers**: Application containers running services like `gateway` and `toys-bestseller` * **Kubernetes Deployments**: Workload definitions with pod specifications * **Kubernetes Pods**: Running instances of the deployments * **Kubernetes Cluster**: The overall cluster target Each target comes with rich attributes including: * Kubernetes labels (topology zones, service tiers, managed-by tags) * Container metadata (image tags, engine versions) * Host information (hostname, domain) * Datadog and Steadybit-specific tags Once you install an agent and the extension, they will [automatically discover targets](/concepts/discovery) like these. ## Working with Sample Data When opening up Steadybit, you're welcomed by the dashboard showing you a summary of the most-important activities in your tenant. ![Dashboard showing sample data](/files/pYRMa6tqt3anDKmCxNgT) ### Reviewing Service and Service's Risk You can explore the **Online Shop** service by jumping from the dashboard's service widget to the **Online Shop** service. Review its target scope, validations, and properties in the header bar and browse the **Provided Experiments**, **Custom Experiments**, and **Advice** tabs to understand how reliability work is bundled for this application. ![Service Detail](/files/cw0EuL4A9MpSYV4YsaOr) [Learn more about services](/use-steadybit/services). ### Checking Experiment Designs Check out the * linked provided and custom experiment designs in the service **Online Shop** or * navigate to **Experiments** in the Steadybit UI. Open any experiment to explore its design, including: * The hypothesis being tested * The attack steps and their configuration * Target selection using the query UI or query language * Validation checks that determine success or failure ![Overview of sample experiments](/files/xWe3xllJpKnBStLayPHY) ![Sample experiment design](/files/FZnFa7V9GK9P2yaGZq5v) ### Viewing Experiment Runs 1. Open a sample experiment 2. Click on the **Run** tab 3. Review past runs, including: * Run status (completed/failed) * Timeline of each step * Logs and metrics captured during the run ![Sample experiment run](/files/8y3KjpHtOCowlRRZ9gBB) ### Reviewing Advice Review advice * linked in the service **Online Shop** or * navigate to **Explorer** and activate **Show Advice** in the landscape or * go to the **Advice**-tab * Review the reliability recommendations for sample workloads ![Service showing sample advice for simulated Kubernetes environment](/files/Wf1yew6c5QDy7wdaCAwO) ![Explorer showing sample advice for simulated Kubernetes environment](/files/Dlqd53wpyTZIKdMK1wRm) ### Exploring Targets Navigate via * **Services** > **Explore Services** to the **Explorer** to drill-down your targets or * check out the **Explorer**'s **Kubernetes Cluster** view to browse the simulated Kubernetes resources and their attributes ![Explorer showing a simulated Kubernetes environment](/files/qQkJSYSgioIdtQkygS0x) ## Integrating with Your Environment Once you're ready to start chaos engineering with your own infrastructure, simply [install the Steadybit agent and extensions](/quick-start/set-up-agents) in your environment. The sample data targets will automatically be removed once you have real agents connected. Sample experiment designs and run history remain available and can be manually deleted when no longer needed # Compatibility Steadybit offers a wide variety of fault injections, checks and integrations (called [actions](/concepts/actions) for Chaos experiments which are provided by Steadybit's open-source [extensions](/integrate-with-steadybit/extensions).\ These actions can be combined in a timeline-based editor to build your chaos engineering experiment. This page serves you as an overview of the supported technologies.\ A detailed list is available on the [Steadybit Reliability Hub](https://hub.steadybit.com). ## Containers The following capabilities are available when targeting containers, running standalone or in a container orchestration platform like Kubernetes: ### Network Attacks | | Block DNS | Block Traffic | Corrupt Outgoing Packages | Delay Outgoing Traffic | Drop Outgoing Traffic | Limit Outgoing Bandwidth | DNS Error Injection | TCP Reset | | ---------------------------------------------- | --------- | ------------- | ------------------------- | ---------------------- | --------------------- | ------------------------ | ------------------- | --------- | | Docker | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | CRI-O | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | containerd | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Kubernetes | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Red Hat OpenShift | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Elastic Kubernetes Service (EKS) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Elastic Container Service (ECS) on EC2[^1] | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Elastic Container Service (ECS) on Fargate | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | | Google Kubernetes Engine (GKE) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Google Kubernetes Engine (GKE, Autopilot[^2]) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | Azure Kubernetes Service (AKS) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | minikube | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ### Resource Attacks | | Fill Disk | Fill Memory | Stress CPU | Stress IO | Stress Memory | | ---------------------------------------------- | --------- | ----------- | ---------- | --------- | ------------- | | Docker | ✅ | ✅ | ✅ | ✅ | ✅ | | CRI-O | ✅ | ✅ | ✅ | ✅ | ✅ | | containerd | ✅ | ✅ | ✅ | ✅ | ✅ | | Kubernetes | ✅ | ✅ | ✅ | ✅ | ✅ | | Red Hat OpenShift | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Elastic Kubernetes Service (EKS) | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Elastic Container Service (ECS) on EC2[^1] | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Elastic Container Service (ECS) on Fargate | ✅ | ❌ | ✅ | ✅ | ✅ | | Google Kubernetes Engine (GKE) | ✅ | ✅ | ✅ | ✅ | ✅ | | Google Kubernetes Engine (GKE, Autopilot[^2]) | ✅ | ✅ | ✅ | ✅ | ✅ | | Azure Kubernetes Service (AKS) | ✅ | ✅ | ✅ | ✅ | ✅ | | minikube | ✅ | ✅ | ✅ | ✅ | ✅ | ### State Attacks | | Pause Container | Stop Container | | ---------------------------------------------- | --------------- | -------------- | | Docker | ✅ | ✅ | | CRI-O | ✅ | ✅ | | containerd | ✅ | ✅ | | Kubernetes | ✅ | ✅ | | Red Hat OpenShift | ✅ | ✅ | | AWS Elastic Kubernetes Service (EKS) | ✅ | ✅ | | AWS Elastic Container Service (ECS) on EC2[^1] | ✅ | ✅ | | AWS Elastic Container Service (ECS) on Fargate | ❌ | ✅ | | Google Kubernetes Engine (GKE) | ✅ | ✅ | | Google Kubernetes Engine (GKE, Autopilot[^2]) | ✅ | ✅ | | Azure Kubernetes Service (AKS) | ✅ | ✅ | | minikube | ✅ | ✅ | ## Kubernetes Steadybit's capabilities for [containers](#containers) and [physical and virtual hosts](#physical-and-virtual-hosts) also work within Kubernetes clusters: {% content-ref url="#containers" %} [#containers](#containers) {% endcontent-ref %} {% content-ref url="#physical-and-virtual-hosts" %} [#physical-and-virtual-hosts](#physical-and-virtual-hosts) {% endcontent-ref %} On top, Steadybit supports attacks based on the Kubernetes API: ### Attacks | | Cause Crash Loop | Delete Pod | Drain Node | Rollout Restart Deployment | Rollout Restart Argo | Scale Deployment | Scale ReplicaSet | Scale StatefulSet | Set Image | Taint Node | | ----------------------------------------- | ---------------- | ---------- | ---------- | -------------------------- | -------------------- | ---------------- | ---------------- | ----------------- | --------- | ---------- | | Kubernetes | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Red Hat OpenShift | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Elastic Kubernetes Service (EKS) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Google Kubernetes Engine (GKE) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Google Kubernetes Engine (GKE, Autopilot) | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | | Azure Kubernetes Service (AKS) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | minikube | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ### Checks | | DaemonSet Pod Count | Deployment Pod Count | Deployment Rollout Status | Node Count | ReplicaSet Pod Count | StatefulSet Pod Count | | ----------------------------------------- | ------------------- | -------------------- | ------------------------- | ---------- | -------------------- | --------------------- | | Kubernetes | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Red Hat OpenShift | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Elastic Kubernetes Service (EKS) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Google Kubernetes Engine (GKE) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Google Kubernetes Engine (GKE, Autopilot) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Azure Kubernetes Service (AKS) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | minikube | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ### Other Actions | | Display Pod Count Metrics | Display Kubernetes Event Logs | | ----------------------------------------- | ------------------------- | ----------------------------- | | Kubernetes | ✅ | ✅ | | Red Hat OpenShift | ✅ | ✅ | | AWS Elastic Kubernetes Service (EKS) | ✅ | ✅ | | Google Kubernetes Engine (GKE) | ✅ | ✅ | | Google Kubernetes Engine (GKE, Autopilot) | ✅ | ✅ | | Azure Kubernetes Service (AKS) | ✅ | ✅ | | minikube | ✅ | ✅ | ### Ingress Controller Attacks Steadybit can interfere with traffic routed through ingress controllers in the cluster: | | HAProxy Block Traffic | HAProxy Delay Traffic | Nginx Block Traffic | Nginx Delay Traffic | | ----------------------------------------- | --------------------- | --------------------- | ------------------- | ------------------- | | Kubernetes | ✅ | ✅ | ✅ | ✅ | | Red Hat OpenShift | ✅ | ✅ | ✅ | ✅ | | AWS Elastic Kubernetes Service (EKS) | ✅ | ✅ | ✅ | ✅ | | Google Kubernetes Engine (GKE) | ✅ | ✅ | ✅ | ✅ | | Google Kubernetes Engine (GKE, Autopilot) | ✅ | ✅ | ✅ | ✅ | | Azure Kubernetes Service (AKS) | ✅ | ✅ | ✅ | ✅ | | minikube | ✅ | ✅ | ✅ | ✅ | ## Physical and Virtual Hosts The following attacks are available when working with physical hosts and virtual machines (both 64-bit PC and 64-bit ARM architectures). ### Network Attacks | | Block DNS | Block Traffic | Corrupt Outgoing Packages | Delay Outgoing Traffic | Drop Outgoing Traffic | Limit Outgoing Bandwidth | DNS Error Injection | TCP Reset | | ------------------------- | --------- | ------------- | ------------------------- | ---------------------- | --------------------- | ------------------------ | ------------------- | --------- | | Ubuntu 20.04 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Ubuntu 22.04 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Ubuntu 24.04 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Fedora Latest | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Debian Bookworm | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Debian Bullseye | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Amazon Linux 2 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Amazon Linux 2023 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Windows 11 (x64) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | Windows Server 2022 (x64) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | {% hint style="info" %} Other .exe, .deb and .rpm-based distributions will mostly likely work, too, but aren't explicitly tested on. {% endhint %} ### Resource Attacks | | Fill Disk | Fill Memory | Stress CPU | Stress IO | Stress Memory | Limit CPU | | ------------------------- | --------- | ----------- | ---------- | --------- | ------------- | --------- | | Ubuntu 20.04 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Ubuntu 22.04 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Ubuntu 24.04 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Fedora Latest | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Debian Bookworm | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Debian Bullseye | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Amazon Linux 2 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Amazon Linux 2023 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Windows 11 (x64) | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | Windows Server 2022 (x64) | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | {% hint style="info" %} Other .deb and .rpm-based distributions will mostly likely work, too, but aren't explicitly tested on. {% endhint %} ### State Attacks | | Shutdown Host | Stop Process | Time Travel | | ------------------------- | ------------- | ------------ | ----------- | | Ubuntu 20.04 | ✅ | ✅ | ✅ | | Ubuntu 22.04 | ✅ | ✅ | ✅ | | Ubuntu 24.04 | ✅ | ✅ | ✅ | | Fedora Latest | ✅ | ✅ | ✅ | | Debian Bookworm | ✅ | ✅ | ✅ | | Debian Bullseye | ✅ | ✅ | ✅ | | Amazon Linux 2 | ✅ | ✅ | ✅ | | Amazon Linux 2023 | ✅ | ✅ | ✅ | | Windows 11 (x64) | ✅ | ✅ | ✅ | | Windows Server 2022 (x64) | ✅ | ✅ | ✅ | {% hint style="info" %} Other .deb and .rpm-based distributions will mostly likely work, too, but aren't explicitly tested on. {% endhint %} ## Cloud Providers Steadybit supports fault injection on all major cloud providers and an ever-growing list of managed services. ### AWS #### AWS EC2 Steadybit's capabilities for [physical and virtual hosts](#physical-and-virtual-hosts) also work for AWS EC2 environments: {% content-ref url="#physical-and-virtual-hosts" %} [#physical-and-virtual-hosts](#physical-and-virtual-hosts) {% endcontent-ref %} On top, Steadybit supports attacks based on the AWS EC2 API: | | Hibernate | Reboot | Start | Stop | Terminate | | ------------ | --------- | ------ | ----- | ---- | --------- | | EC2 Instance | ✅ | ✅ | ✅ | ✅ | ✅ | #### AWS ECS on EC2 Steadybit's capabilities for [AWS EC2s](#aws-ec2), and [containers](#containers) also work for AWS ECS on EC2 environments: {% content-ref url="#aws-ec2" %} [#aws-ec2](#aws-ec2) {% endcontent-ref %} {% content-ref url="#containers" %} [#containers](#containers) {% endcontent-ref %} On top, Steadybit provides capabilities based on the AWS ECS API: | | Scale Service | Service Event Log | Service Task Count | Stop Process (in Task) | | ---------------------------------------------- | ------------- | ----------------- | ------------------ | ---------------------- | | AWS Elastic Container Service (ECS) on EC2 | ✅ | ✅ | ✅ | ✅ | | AWS Elastic Container Service (ECS) on Fargate | ✅ | ✅ | ✅ | ✅ | #### AWS ECS on Fargate Steadybit's capabilities for [containers](#containers) also work for AWS ECS on Fargate environments: {% content-ref url="#containers" %} [#containers](#containers) {% endcontent-ref %} On top, Steadybit provides capabilities based on the AWS ECS API: | | Scale Service | Service Event Log | Service Task Count | Stop Process (in Task) | | ---------------------------------------------- | ------------- | ----------------- | ------------------ | ---------------------- | | AWS Elastic Container Service (ECS) on EC2 | ✅ | ✅ | ✅ | ✅ | | AWS Elastic Container Service (ECS) on Fargate | ✅ | ✅ | ✅ | ✅ | #### AWS EKS (Elastic Kubernetes Service) Steadybit's capabilities for [containers](#containers), [Kubernetes](#kubernetes) and [AWS EC2s](#aws-ec2) also work for AWS EKS environments: {% content-ref url="#containers" %} [#containers](#containers) {% endcontent-ref %} {% content-ref url="#kubernetes" %} [#kubernetes](#kubernetes) {% endcontent-ref %} {% content-ref url="#aws-ec2" %} [#aws-ec2](#aws-ec2) {% endcontent-ref %} #### AWS ELB (Elastic Load Balancing) Steadybit supports attacks based on the AWS ALB API: | | Return Static Response | | --- | ---------------------- | | ALB | ✅ | #### AWS Elasticache Steadybit supports attacks based on the AWS Elasticache API: | | Node Group Failover | | ----------- | ------------------- | | Elasticache | ✅ | #### AWS Fault Injection Service (FIS) Steadybit integrates with AWS FIS experiment templates, which makes it easy to inject faults into additional managed services. See the [AWS extension](https://hub.steadybit.com/target/com.steadybit.extension_aws.fis-experiment-template) for more details. | | Start Experiment Template | | --- | ------------------------- | | FIS | ✅ | #### AWS Lambda Steadybit supports attacks based on the failure injection wrapper [failure-lambda](https://github.com/steadybit/failure-lambda): | | Block TCP Connections | Fill Diskspace | Inject Exception | Inject Latency | Inject Status Code | | ------ | --------------------- | -------------- | ---------------- | -------------- | ------------------ | | Lambda | ✅ | ✅ | ✅ | ✅ | ✅ | #### AWS Managed Streaming for Kafka (MSK) Steadybit's capabilities for [Kafka](#kafka) also work for AWS MSK environments: {% content-ref url="#kafka" %} [#kafka](#kafka) {% endcontent-ref %} On top, Steadybit provides capabilities based on the AWS MSK API: | | Broker Reboot | | --- | ------------- | | MSK | ✅ | #### AWS Relational Database Service (RDS) Steadybit provides capabilities based on the AWS RDS API for RDS clusters and instances. **RDS Clusters** | | Cluster Failover | | ----------- | ---------------- | | RDS Cluster | ✅ | **RDS Instance** | | Reboot | Stop | | ------------ | ------ | ---- | | RDS Instance | ✅ | ✅ | #### Other AWS Services Steadybit has cross-services capabilities like simulating a full [availability zone outage](https://hub.steadybit.com/target/com.steadybit.extension_aws.zone) or [partial zone outage via subnets](https://hub.steadybit.com/target/com.steadybit.extension_aws.ec2-subnet). Additional managed services are covered via [AWS FIS](#aws-fault-injection-service-fis), or will be added to Steadybit natively in the future. ### Azure #### Azure Kubernetes Service (AKS) Steadybit's capabilities for [containers](#containers), [Kubernetes](#kubernetes) and [Azure VMs](#azure-virtual-machines-vms) also work for Azure AKS environments: {% content-ref url="#containers" %} [#containers](#containers) {% endcontent-ref %} {% content-ref url="#kubernetes" %} [#kubernetes](#kubernetes) {% endcontent-ref %} {% content-ref url="#azure-virtual-machines-vms" %} [#azure-virtual-machines-vms](#azure-virtual-machines-vms) {% endcontent-ref %} #### Azure Event Hub Steadybit's capabilities for [Kafka](#kafka) also work for Azure Event Hub leveraging Kafka-compatibility: {% content-ref url="#kafka" %} [#kafka](#kafka) {% endcontent-ref %} #### Azure Virtual Machines (VMs) Steadybit's capabilities for [physical and virtual hosts](#physical-and-virtual-hosts) also work for Azure Virtual Machines (VMs): {% content-ref url="#physical-and-virtual-hosts" %} [#physical-and-virtual-hosts](#physical-and-virtual-hosts) {% endcontent-ref %} On top, Steadybit supports attacks based on the Azure VMs API: **Virtual Machine** | | Reboot | Delete | Stop | Deallocate | | --------------- | ------ | ------ | ---- | ---------- | | Virtual Machine | ✅ | ✅ | ✅ | ✅ | **Virtual Machine Scale Set Instances** | | Reboot | Delete | Stop | Deallocate | | ------------------ | ------ | ------ | ---- | ---------- | | Scale Set Instance | ✅ | ✅ | ✅ | ✅ | #### Azure Functions Steadybit supports application-level fault injections for Azure Functions: | | Inject Exception | Fill Disk | Inject Latency | Inject Status Code | | -------------- | ---------------- | --------- | -------------- | ------------------ | | Azure Function | ✅ | ✅ | ✅ | ✅ | #### Azure Container Apps Steadybit supports application-level fault injections for Azure Container Apps: | | Inject Exception | Fill Disk | Inject Latency | Inject Status Code | | ------------------- | ---------------- | --------- | -------------- | ------------------ | | Azure Container App | ✅ | ✅ | ✅ | ✅ | #### Azure Network Security Groups Steadybit supports network attacks via Azure NSGs: | | Block Hosts | | ---------------------- | ----------- | | Network Security Group | ✅ | #### Other Azure Services More Azure services will be added to the [Azure](https://hub.steadybit.com/extension/com.steadybit.extension_azure) extension in the future. ### GCP #### Google Kubernetes Engine (GKE) Steadybit's capabilities for [containers](#containers), [Kubernetes](#kubernetes) and [GCP VMs](#gcp-virtual-machines-vms) also work for Google GKE: {% content-ref url="#containers" %} [#containers](#containers) {% endcontent-ref %} {% content-ref url="#kubernetes" %} [#kubernetes](#kubernetes) {% endcontent-ref %} {% content-ref url="#gcp-virtual-machines-vms" %} [#gcp-virtual-machines-vms](#gcp-virtual-machines-vms) {% endcontent-ref %} #### GCP Virtual machines (VMs) Steadybit's capabilities for [physical and virtual hosts](#physical-and-virtual-hosts) also work for Azure Virtual Machines (VMs): {% content-ref url="#physical-and-virtual-hosts" %} [#physical-and-virtual-hosts](#physical-and-virtual-hosts) {% endcontent-ref %} On top, Steadybit supports attacks based on the GCP VMs API: | | Reset | Delete | Stop | Suspend | | --------------- | ----- | ------ | ---- | ------- | | Virtual Machine | ✅ | ✅ | ✅ | ✅ | #### Other GCP Services More GCP services will be added to the [GCP](https://hub.steadybit.com/extension/com.steadybit.extension_gcp) extension in the future. ## Cloud Foundry Steadybit supports the following capabilities for Cloud Foundry applications: | | Check App State | Restart App | Stop App | | ----------- | --------------- | ----------- | -------- | | Application | ✅ | ✅ | ✅ | ## Service Mesh & API Gateway Steadybit provides the following fault injections for service meshes and api gateways. | | GRPC Abort | HTTP Abort | HTTP Delay | | ----- | ---------- | ---------- | ---------- | | Istio | ✅ | ✅ | ✅ | | Kong | ❌ | ✅ | ❌ | ## Kafka Steadybit offers comprehensive support for chaos experiments on Kafka infrastructure. ### Broker | | Check Broker State | Limit Connection Creation Rate | Limit IO Thread | Limit Network Threads | Reduce Message Batch Size | | ------------------------------------- | ------------------ | ------------------------------ | --------------- | --------------------- | ------------------------- | | Unmanaged Kafka | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Managed Streaming for Kafka (MSK) | ✅ | ✅ | ✅ | ✅ | ✅ | | Azure Event Hub (Kafka) | ✅ | ✅ | ✅ | ✅ | ✅ | ### Consumer | | Check Consumer State | Check Topic Lag | Deny Access | | ------------------------------------- | -------------------- | --------------- | ----------- | | Unmanaged Kafka | ✅ | ✅ | ✅ | | AWS Managed Streaming for Kafka (MSK) | ✅ | ✅ | ✅ | | Azure Event Hub (Kafka) | ✅ | ✅ | ✅ | ### Topics | | Check Partitions | Produce Records | Delete Records | Partition Leader Election | | ------------------------------------- | ---------------- | --------------- | -------------- | ------------------------- | | Unmanaged Kafka | ✅ | ✅ | ✅ | ✅ | | AWS Managed Streaming for Kafka (MSK) | ✅ | ✅ | ✅ | ✅ | | Azure Event Hub (Kafka) | ✅ | ✅ | ✅ | ✅ | ## Redis Steadybit offers attacks and checks for Redis databases and instances. ### Database | | Cache Expiration | | -------- | ---------------- | | Database | ✅ | ### Instance | | Check Connections | Check Latency | Check Memory | Check Replication | Client Pause | Connection Exhaustion | Maxmemory Limit | Sentinel Stop | | -------- | ----------------- | ------------- | ------------ | ----------------- | ------------ | --------------------- | --------------- | ------------- | | Instance | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ## RabbitMQ Steadybit offers attacks and checks for RabbitMQ infrastructure. ### Node | | Check Node | | ---- | ---------- | | Node | ✅ | ### Queue | | Alter Max Length | Check Backlog | Publish Records | | ----- | ---------------- | ------------- | --------------- | | Queue | ✅ | ✅ | ✅ | ## CI/CD Steadybit integrates with the following CI/CD systems for triggering jobs from experiments: | | Run Job | | ------- | ------- | | Jenkins | ✅ | Triggering a Steadybit experiment from a CI/CD pipeline works with all products by using the [CLI](/integrate-with-steadybit/cli), [API](/integrate-with-steadybit/api). ## Java Virtual Machine (JVM) / Spring Applications Steadybit supports the following application-level faults for JVM- and Spring-based applications. These don't require any dependency at work at runtime via bytecode manipulation. | | Spring Controller Delay | Spring Controller Exception | HTTP Client Delay | HTTP Client Status | Method Delay | Method Exception | JDBC Template Delay | JDBC Template Exception | | --------------- | ----------------------- | --------------------------- | ----------------- | ------------------ | ------------ | ---------------- | ------------------- | ----------------------- | | JVM application | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ## Observability Steadybit supports the following observability-related experiment actions: | | [Check for Alerts](#user-content-fn-3)[^3] | [Mute Alerts](#user-content-fn-3)[^3] | [Send Events](#user-content-fn-4)[^4] | | -------------------------- | ------------------------------------------ | ------------------------------------- | ------------------------------------- | | AppDynamics | ✅ | ✅ | ❌ | | Datadog | ✅ | ✅ | ✅ | | Dynatrace | ✅ | ✅ | ✅ | | Grafana | ✅ | ❌ | ✅ | | Instana | ✅ | ✅ | ❌ | | New Relic[^5] | ✅ | ✅ | ✅ | | Prometheus | ✅ | ❌ | ❌ | | Splunk Observability Cloud | ✅ | ❌ | ✅ | | Splunk Platform | ✅ | ❌ | ❌ | | StackState | ✅ | ❌ | ❌ | ## Load and API Testing Steadybit integrates with the following load- and API-testing solutions: | | Run Test from Experiment | Run Experiment from Test | | ----------------------------------------------------- | ------------------------ | ------------------------ | | Micro Focus LoadRunner Professional | ❌ | ✅ | | Micro Focus LoadRunner Enterprise | ❌ | ✅ | | Gatling | ✅ | ✅ | | JMeter | ✅ | ✅ | | K6 | ✅ | ✅ | | K6 Cloud | ✅ | ✅ | | Postman | ✅ | ✅ | | Custom HTTP Check (fixed amount, periodic, bandwidth) | ✅ | ✅ | [^1]: extension-host and extension-container needs to run in privileged mode and network mode host is required for the extensions. [^2]: Allow-listing Steadybit is required for container-level attacks in Autopilot-managed GKE clusters. Container attacks in the following namespaces are\ disabled: `kube-system`, `gke-gmp-system`, `composer-system`, `gke-managed-*` [^3]: Different observability integrations us different names for these action. Please check the [Steadybit Reliability Hub](https://hub.steadybit.com) for more\ details. [^4]: Synthetic events will be sent to the observability solution to mark the beginning and end of every experiment action to facilitate root cause analysis. [^5]: Steadybit integrates with New Relic's Workload and Incident alerting. # Install Agent and Extensions This guide helps you with the initial setup of Steadybit by installing an agent and useful extensions. In case something is not working out for you, [let us help you](https://www.steadybit.com/contact)! ## Prerequisite * You have already signed up for an account [on our website](https://signup.steadybit.com/) * You are able to log in to the [Steadybit SaaS platform](https://platform.steadybit.com/) ## Step 1 - Install Agent and Extensions Our agent is the communication channel into your environment. The agent is connected to various extensions that continuously discover and deliver target data to the platform and help you create targeted experiments based on data that is always up-to-date. Therefore, it is necessary that you deploy the agents and extensions to the locations in which you want to perform Chaos Engineering experiments. {% hint style="info" %} Without agents, there is no connection to your environment. And without extensions, there is no discovery data and thus no experiments are possible. {% endhint %} We offer our agents and extensions for different platforms and have made the installation as easy as possible for you. You can find the different installation scripts directly in our platform after logging in. Simply copy the script matching your technology and execute it next to your system. ![Steadybit Onboarding - Setup Scripts](/files/SaEw046NYOv7rPh6E2bw) > If you need a demo application to play around, you can [deploy our Shopping Demo into a Minikube or AWS EKS cluster](/quick-start/deploy-example-application). Once an agent is rolled out, it connects to the platform and appears at the bottom of the page. This may take a few minutes. Afterwards, you are able to continue to the next step. ## Step 2 - Let Steadybit discover your system Now, the agent and extensions are busy discovering your system to give you an overview what has been found. ![Steadybit Onboarding - System Discovery](/files/L4FVbuHccSbLFJSRvlkm) ## Step 3 - Install More Extensions (Optional) You miss support for further technologies? That's why Steadybit supports extensions! Extensions enable you to make Steadybit your own – either by using an official or open source extension or by writing your own! Learn more about extensions in our [dedicated documentation](/integrate-with-steadybit/extensions). ![Steadybit Onboarding - Add Extensions](/files/ZgcgbzTyraaTPDTqrHek) ## Step 4 - Start using Steadybit! That's it! You are welcomed by our Dashboard, and you are ready to take off by [designing and running your first Chaos Engineering experiment](/quick-start/run-experiment). ![Steadybit Dashboard](/files/w1C9SNrLVzoR0SWVhDyZ) # Run an Experiment We will now use Steadybit to design and run our first experiment in just 4 simple steps. This guide focuses on our example shopping demo application, which you can [deploy by follow this guide](/quick-start/deploy-example-application). However, you can also adapt the steps to your application. ## Prerequisite * You have already signed up for an account [on our website](https://signup.steadybit.com) * You can log in to the [Steadybit SaaS platform](https://platform.steadybit.com) * You have already [installed an agent and extensions](/quick-start/set-up-agents) ## Step 1 - Define your Scenario The first step is to think about the scenario you want to test. Good inspiration for scenarios is, e.g., a past incident, dependencies of your architecture, or common pitfalls. In our case, we want to identify how our shopping demo behaves when one of the product backend services is unavailable. We can simply use one of the existing templates to create this experiment step-by-step. In the Steadybit platform, go to 'Experiments' -> 'New Experiment' -> 'From Template' and search for a template with the tag `Shopping Demo Quick Start`. ![Run Experiment - Step 1: Use Template](/files/flVnSx32vZRnq7CPuuhX) You can see the overall experiment structure in the template details or continue this tutorial by choosing 'Use This Template'. > If you miss this experiment template in your Steadybit platform, you can [download the resulting experiment here](https://github.com/steadybit/docs-public/blob/main/quick-start/run-experiment/experiment.json) and import it via 'Experiments' -> 'New Experiment' -> 'Upload'. Continue to [Step 3 - Experiment Design](#step-3-experiment-design). In the future, the template will be available also in your tenant. ## Step 2 - Define experiment Now, you need to define step-by-step template-specific experiment details to validate the scenario. ### (1) Environment The first option is always to select the environment where you want to run the experiment. The environment limits the set of attackable targets and thus prevents you from affecting the wrong stage or interfering with other teams. Check out [Manage Environments](/install-and-configure/manage-environments) later. If you haven't set up environments yet, you can continue with the `Global` environment, which contains everything Steadybit discovered. ![Run Experiment - Template Wizard Step 1: Environment](/files/TE4qsPFl9KHEJHvQSPWp) ### (2) Cluster Name, (3) Namespace, and (4) Deployment Based on the selected template, the experiment needs to know the name of your Kubernetes cluster (`minikube`), the Kubernetes namespace (`steadybit-demo`), and the deployment (`hot-deals`). Just choose the values from the drop-down. ![Run Experiment - Template Wizard Step 2: Kubernetes Cluster Name](/files/fY2YdydGCvdgjWus2zUL) ### (5) HTTP Upstream Endpoint, and (6) Success Rate The next step is to define the upstream HTTP endpoint dependent on our downstream deployment `hot deals`. This is the `/products` endpoint of `gateway`, which crawls all products from `hot-deals`, `toys-bestseller`, and `fashion-bestseller`. The resulting products are shown on our shop's landing page. In Steadybit, we can also use Kubernetes-internal URLs, like `http://gateway.steadybit-demo.svc.cluster.local/products`. ![Run Experiment - Template Wizard Step 5: HTTP Upstream Endpoint](/files/4RYf14y05dhAE3u6SlgM) In the next step, we specify the expected HTTP success rate. Since the shop's landing page uses this endpoint, we aim for `100%` successful responses. ### (7) Recovery Time The last step is to define the recovery time. How long do we expect it to take until all pods of `hot-deals` start up again and are ready to serve traffic? In our example, we expect 60 seconds to be sufficient. Finalize the template's wizard by clicking 'Create Experiment'. ## Step 3 - Experiment Design Woohoo! There is our first experiment design! 🎉 You're now in the timeline-based experiment editor that you can always use to design an experiment from scratch via drag-and-drop. ![Run Experiment - Experiment Design](/files/W9DuEfa0Uy2GcI3mJx0y) Our experiment was already designed by using the template, so we can save and run it immediately to learn whether the shop survives an outage of the downstream deployment `hot-deals`. ## Step 4 - Run experiment When hitting the 'Run Experiment'-button you see the Steadybit run view. As soon as the agent connected, the experiment starts to validate the HTTP endpoint `/products`, and the amount of ready pods for the deployment `hot-deals`. When isolating the deployment's containers, we start noticing faults in the HTTP responses as `hot-deals`' products can't be requested anymore. This is undesirable as there are other products of `fashion-bestseller` and `toys-bestseller` which could have been shown at the shop's landing page. You can improve this behavior by adding appropriate fallbacks or scaling the services. ![Run Experiment - Experiment Run View](/files/lDrJtLaXywGJ1JuXMtLs) Continuing to run the experiment, we see that eventually, Kubernetes restarts the deployment's pods, resulting in missing pods. Even so, after the attack, all pods become ready within the expected 60 seconds. However, the overall experiment run failed, as the HTTP success rate of `100%` was not achieved. ![Run Experiment - Experiment Run View](/files/NPExLBSDZIeB6cUfeoWb) ## Conclusion You have now successfully run an experiment with Steadybit in a Kubernetes environment. You have discovered the impact of an unavailable downstream service on the upstream service. ### What are the next steps? Check how the shop's behavior differs when using [different implementation of the products endpoint](https://github.com/steadybit/shopping-demo?tab=readme-ov-file#products-rest-endpoint). Alternatively, explore your next experiment using the [explorer landscape](/use-steadybit/explorer/landscape) and get [advice](/use-steadybit/explorer/advice) to learn how to improve your system's reliability. Eventually, before rolling it out to more users, make sure to [set up proper environments](/install-and-configure/manage-environments) and [create teams](/install-and-configure/manage-teams-and-users) to benefit from Steadybit's safety in Chaos Engineering rollouts. # Deploy Example Application When trying out Steadybit you may want to start easy instead of directly using your fully fledged system. Therefore, we have a small example application called [Shopping Demo](https://github.com/steadybit/shopping-demo) which you can easily deploy on a local minikube or AWS EKS using this guide. Simply follow these two steps: * [Step 1 - Check out example application](#step-1-have-a-look-a-the-example-application) * [Step 2 - Deploy the example application](#step-2-deploy-the-example-application) * [a) on Minikube](#step-2a-deploy-on-minikube) * [b) on AWS EKS](#step-2b-deploy-on-aws-eks) ## Prerequisites * You have already signed up for an account [on our website](https://signup.steadybit.com/) * You are able to log in to the [Steadybit SaaS platform](https://platform.steadybit.com/) * You have [Kubernetes kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) installed * You have [Helm - The package manager for Kubernetes](https://helm.sh/) installed ## Step 1 - Have a look a the example application In order to give you a quick and easy start, we have developed a small demo application. Our shopping demo is a small product catalog provided by seven distributed backend services and a simple UI. ![shopping-demo-app](/files/8RsTW0eAVuq5OghNLwic) If you want to learn more about our demo, please take a look at our GitHub repository: ## Step 2 - Deploy the example application The example application is already pre-configured to be deployed into a Kubernetes cluster. You can choose whether to deploy it into [a) local minikube installation](#step-2a-deploy-on-minikube) or [b) AWS EKS](#step-2b-deploy-on-aws-eks). > [Kubernetes](https://kubernetes.io/), also known as k8s, is an open source system for automating the deployment, scaling, and management of containerized applications. You can use [minikube](https://minikube.sigs.k8s.io/docs/) to set up a local Kubernetes cluster on macOS, Linux or Windows. As an alternative choose [AWS EKS](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html) to set up a Kubernetes cluster in the cloud. Make sure to [install the agents](/quick-start/set-up-agents) afterwards into the same environment. ### Step 2a) Deploy on Minikube **Prerequisite** * You have a running [minikube](https://minikube.sigs.k8s.io/docs/start/) installation **Start your minikube cluster** From a terminal, run: ```bash minikube start ``` You can access your cluster with: ```bash kubectl get po -A ``` **Deploy the example application** Now we use helm to deploy the demo by running the following command: ```bash helm repo add steadybit-shopping-demo https://steadybit.github.io/shopping-demo helm repo update helm upgrade steadybit-shopping-demo \ --install \ --wait \ --timeout 5m0s \ --create-namespace \ --namespace steadybit-shopping-demo \ --set gateway.service.type=ClusterIP \ steadybit-shopping-demo/steadybit-shopping-demo ``` Verify that all Shopping Demo pods are running: ```bash kubectl get pods --namespace steadybit-shopping-demo ``` You will see the following result, all pods are ready if you can see the status `Running`: ```bash NAME READY STATUS RESTARTS AGE fashion-bestseller-79b9698f88-557vt 1/1 Running 0 11s gateway-7fc74f7f9b-tshzg 1/1 Running 0 11s hot-deals-75cb898ff7-wrnxc 1/1 Running 0 10s postgres-68f9db56cc-wxxth 1/1 Running 0 10s toys-bestseller-6df5bd864f-kzrt9 1/1 Running 0 11s orders-dcf644b8-g277b 1/1 Pending 0 10s inventory-7895d47cb7-sfdqb 1/1 Pending 0 10s activemq-6dd55b4b7-wqmk6 1/1 Running 0 11s ``` You can do a local port forward to your minikube to open the `gateway` service via your browser: ```bash kubectl port-forward deployment/gateway 8080:8080 --namespace steadybit-shopping-demo ``` Visit `http://127.0.01:8080/products` in your browser to retrieve the aggregated list of all products or just use `curl`: ```bash curl http://127.0.01:8080/products ``` The result is an aggregated list of all products of the services `toys`, `hot-deals` and `fashion`: ```bash { "fashionResponse": { "responseType": "REMOTE_SERVICE", "products": [ { "id": "e9f0bec4-989c-4b9f-8bf9-334622e915ad", "name": "Bob Mailor Slim Jeans", "category": "FASHION" }, { "id": "b110185b-d808-4104-b605-08a90b1248ce", "name": "Lewi's Jeanshose 511 Slim Fit", "category": "FASHION" }, { "id": "222d7084-3cc7-43c3-890f-4598aa44eb2f", "name": "Urban Classics Shirt Shaped Long Tee", "category": "FASHION" } ] }, "toysResponse": { "responseType": "REMOTE_SERVICE", "products": [ ... ] }, "hotDealsResponse": { "responseType": "REMOTE_SERVICE", "products": [ ... ] }, "duration": 112, "statusFashion": "REMOTE_SERVICE", "statusToys": "REMOTE_SERVICE", "statusHotDeals": "REMOTE_SERVICE" } ``` ### Step 2b) Deploy on AWS EKS **Prerequisites** * [AWS account](https://aws.amazon.com/de/account/) * [AWS Command Line Interface](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) * [AWS eksctl](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html) **Create your AWS Elastic Kubernetes Service (AWS EKS) cluster and nodes** Verfiy your AWS CLI configuration by running: ```bash aws --version ``` Your output should be similiar to: ```bash aws-cli/2.0.44 Python/3.8.5 Darwin/19.6.0 source/x86_64 ``` Create your Amazon EKS cluster and containing 2 nodes by running the following command. More details are available at [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html) From a terminal, run: ```bash eksctl create cluster \ --name steadybit-demo-cluster \ --region us-west-2 ``` You can access your cluster with: ```bash kubectl get nodes ``` Your output should be look like: ```bash NAME STATUS ROLES AGE VERSION ip-192-168-53-195.us-west-2.compute.internal Ready 113s v1.17.12-eks-7684af ip-192-168-68-23.us-west-2.compute.internal Ready 118s v1.17.12-eks-7684af ``` **Deploy the Shopping Demo** Now we use kubectl to deploy the demo by running the following command: ```bash helm repo add steadybit-shopping-demo https://steadybit.github.io/shopping-demo helm repo update helm upgrade steadybit-shopping-demo \ --install \ --wait \ --timeout 5m0s \ --create-namespace \ --namespace steadybit-shopping-demo \ -f \ steadybit-shopping-demo/steadybit-shopping-demo ``` Maybe you need to edit some ingress hosts names in your own `values.yaml` file. Verify that all Shopping Demo pods are running: ```bash kubectl get pods --namespace steadybit-shopping-demo ``` You will see the following result, all pods are ready if you can see the status `Running`: ```bash NAME READY STATUS RESTARTS AGE fashion-bestseller-79b9698f88-557vt 1/1 Running 0 11s gateway-7fc74f7f9b-tshzg 1/1 Running 0 11s hot-deals-75cb898ff7-wrnxc 1/1 Running 0 10s postgres-68f9db56cc-wxxth 1/1 Running 0 10s toys-bestseller-6df5bd864f-kzrt9 1/1 Running 0 11s orders-dcf644b8-g277b 1/1 Pending 0 10s inventory-7895d47cb7-sfdqb 1/1 Pending 0 10s activemq-6dd55b4b7-wqmk6 1/1 Running 0 11s ``` With the following command you can now determine the external IP and port to access the `gateway` service: ```bash kubectl get svc -n steadybit-shopping-demo ``` Example response: ```bash NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE fashion-bestseller NodePort ------------- ----:-----/--- --- gateway LoadBalancer 10.98.173.27 127.0.0.1 80:30131/TCP 3h15m hot-deals NodePort ------------- ----:-----/--- --- product-db NodePort ------------- ----:-----/--- --- toys-bestseller NodePort ------------- ----:-----/--- --- orders NodePort ------------- ----:-----/--- --- inventory NodePort ------------- ----:-----/--- --- activemq NodePort ------------- ----:-----/--- --- ``` Visit `http://{EXTERNAL-IP}:{PORT}/products` in your browser to retrieve the aggregated list of all products or just use `curl`: ```bash curl http://{EXTERNAL-IP}:{PORT}/products ``` The result is an aggregated list of all products of the services `toys`, `hot-deals` and `fashion`: ```bash { "fashionResponse": { "responseType": "REMOTE_SERVICE", "products": [ { "id": "e9f0bec4-989c-4b9f-8bf9-334622e915ad", "name": "Bob Mailor Slim Jeans", "category": "FASHION" }, { "id": "b110185b-d808-4104-b605-08a90b1248ce", "name": "Lewi's Jeanshose 511 Slim Fit", "category": "FASHION" }, { "id": "222d7084-3cc7-43c3-890f-4598aa44eb2f", "name": "Urban Classics Shirt Shaped Long Tee", "category": "FASHION" } ] }, "toysResponse": { "responseType": "REMOTE_SERVICE", "products": [ ... ] }, "hotDealsResponse": { "responseType": "REMOTE_SERVICE", "products": [ ... ] }, "duration": 112, "statusFashion": "REMOTE_SERVICE", "statusToys": "REMOTE_SERVICE", "statusHotDeals": "REMOTE_SERVICE" } ``` ## Next Steps Now, make sure to [install the agents](/quick-start/set-up-agents) into the same environment (minikube or AWS EKS). # Actions Actions are the building blocks of Chaos Engineering experiments and allow you to attack, check, or load test your systems during an experiment run. For instance, a simple experiment may consist of a single attack, e.g., to increase CPU usage within a container. In more advanced cases, you may compose various actions to, e.g., inject traffic, increase memory usage, and check whether your Observability monitor notices anything. Check out [Steadybit's Templates](https://hub.steadybit.com/templates) to learn more about how to combine actions into valuable experiments. {% hint style="info" %} An action is always coming from a Steadybit extension and is submitted via the Steadybit Agent to the platform. If you haven't installed any extensions, only one action is available: a simple wait action. {% endhint %} ## Kinds of Actions The following sections will explain the supported kinds of actions in more detail. ### Attacks Attacks, as the name implies, have a deliberate negative impact on your system. For example, you may decide to: * Kill a set of containers or simulate a Kubernetes deployment rollout (state attack). * Increase the memory or CPU usage on a host (resource attack). * Artificially increase network latency or reconfigure AWS VPC configurations to simulate an availability zone outage (network attack). In the end, an attack execution always has the same intended effect: Let me understand how a system behaves under adverse/turbulent conditions. Attacks introduce adverse/turbulent conditions for as long as necessary, e.g., as long as necessary to observe the effect. Once done, the attack is rolled back (whenever possible) to move the system back into an operational state. You can learn about all supported attacks within our [Reliability Hub](https://hub.steadybit.com/actions?kind=attack). ### Checks Checks (sometimes called probes) act as verifications or assertions within experiments. Once you progress past one-off executions of attacks, you should verify your expectations automatically. Not only for easier execution but also consistency and knowledge sharing with team members. On top of this, automatic verification is faster, thus reducing the time attacks need to be executed and reducing the time your system will be affected by attacks. You can incorporate various checks into your experiments. Some examples: * Ensure that no Kubernetes rollout is in progress before starting an attack (see [Kubernetes Deployment Rollout Status check in Reliability Hub](https://hub.steadybit.com/action/com.steadybit.extension_kubernetes.rollout-status) * Verify that your Datadog monitors are reporting issues when there are issues (injected by attacks, see [Datadog Monitor Status check in Reliability Hub](https://hub.steadybit.com/action/com.steadybit.extension_datadog.monitor.status_check)) * Run Prometheus PromQL queries to check internal system metrics (see [Prometheus Metrics check in Reliability Hub](https://hub.steadybit.com/action/com.steadybit.extension_prometheus.instance.metrics)) * Leverage your Postman API tests to check that your API is still working as intended while a part of your system is under attack (see [Postman Collection check in Reliability Hub](https://hub.steadybit.com/action/com.steadybit.extension_postman.collection.run)) Most checks will immediately abort the experiment run and mark it as *failed*. You can learn about all supported checks within our [Reliability Hub](https://hub.steadybit.com/actions?kind=check). ### Load Tests Load tests, within the chaos engineering experiment context, are helpful to introduce a baseline of expected traffic on your system – primarily when operating in non-production environments. Without any load on the system, some effects of attacks may not be observable, e.g., your observability solution might not identify problems for end-users because there were no end-users. You can learn about all supported load tests within our [Reliability Hub](https://hub.steadybit.com/actions?kind=load_test). ## Supported Actions Check out our [Reliability Hub](https://hub.steadybit.com/actions) to learn about actions you can leverage with Steadybit. ## Missing an action? Extend Steadybit! Are you missing support for a particular action? No problem, Steadybit is extensible, allowing you to use open- and closed-source extensions to enhance its capabilities. Learn more within our [extension documentation](/integrate-with-steadybit/extensions)! # Discovery Steadybit automatically discovers your system and makes it available as potential Chaos Engineering targets. Each target is associated with attributes you can use to specify the exact set of targets when designing or running a Chaos Engineering experiment. What kind of targets are discovered (e.g., containers, hosts, Kubernetes deployments, or JVM applications), and what kind of attributes are associated (e.g., `k8s.cluster-name`, `aws.zone` or `application.name`) depends on the installed extensions. {% hint style="info" %} Target discovery is always performed by a Steadybit extension. The discovered metadata is submitted via the Steadybit Agent to the platform. If you haven't installed any extensions, you aren't seeing any targets and can't make use of Steadybit's Chaos Engineering platform. {% endhint %} ## Supported Targets Check out our [Reliability Hub](https://hub.steadybit.com/targets) to learn about which targets are discovered by a Steadybit extension. ## Missing a target? Extend Steadybit! Are you missing support for a particular technology? No problem, Steadybit is extensible, allowing you to use open- and closed-source extensions to enhance its capabilities. Learn more within our [extension documentation](/integrate-with-steadybit/extensions)! # Query Language ### What is the Query Language? There are some use cases, where you want to restrict the targets discovered by Steadybit. One use case can be that you want to [design an experiment](/use-steadybit/experiments/design#basic-elements) and make sure, that there are only targets of a specific Kubernetes cluster affected by the experiment. Another use case is that you want to restrict the available targets when [configuring an environment](/install-and-configure/manage-environments#define-your-own-environment). Boiling down to a set of targets can result in complex statements. For instance, you want to make sure that the targets are matching some sets of key-value pairs but also not in your production cluster. Expressions like these can now easily be written in Steadybits Query Language. The Query Language is a textual representation of the Query UI but with a more advanced feature set. It allows you to build semantic expression blocks, combining them with other expressions or negating them. The Query UI and the Query Language always come together, so it is up to you to choose the style. ### How to switch between the Query UI and the Query Langauge

The same query can be expressed with the Query UI on the left and the Query Language on the right

When introducing the Query Language, we added a new tab to the already known query UI. Clicking it will switch the edit mode to the Query Language. The already configured query will be translated into the textual form and rendered within the text editor. You can just go ahead writing your own query expressions in there. Like the Query UI, the editor will provide you with auto-completion on available keys and values. Please note that the Query UI is limited in regard to the queries you write. For instance, one cannot express a query like `(a="b" AND d="e") OR f="g"` with the Query UI. The interface will tell you when the query can only be edited with the language editor.

Complex queries can only be edited in the Query language editor.

### Query Examples #### Key-value comparison Keys and values can be compared using `=`, `!=`, `~`, `!~`, `=*`, `!=*`, `~*`, `!~*`, `IN ()` and `NOT IN ()` ``` // Simple equals check to get all targets of Kubernetes Cluster 'prod' k8s.cluster-name="prod" // Get all Daemonsets/Deployments/Statefulsets (needs platform >= 2.3.7) target.type IN ("com.steadybit.extension_kubernetes.kubernetes-daemonset", "com.steadybit.extension_kubernetes.kubernetes-deployment", "com.steadybit.extension_kubernetes.kubernetes-statefulset") ``` ``` // Not equals check to get all targets not running in the Kuberneters Cluster 'prod' k8s.cluster-name!="prod" // Get all targets that are not a Daemonset/Deployment/Statefulset (needs platform >= 2.3.7) target.type NOT IN ("com.steadybit.extension_kubernetes.kubernetes-daemonset", "com.steadybit.extension_kubernetes.kubernetes-deployment", "com.steadybit.extension_kubernetes.kubernetes-statefulset") ``` ``` // Get all targets with a container name that contains "hot-deals" k8s.container.name~"hot-deals" ``` ``` // Query for all container targets whose maintainer does not contain Jane container.label.maintainer!~"Jane" ``` ``` // Simple equals ignore case check to get all targets of Kubernetes Cluster 'PrOd' ignore casing k8s.cluster-name=*"PrOd" ``` ``` // Not equals ignore case check to get all targets not running in the Kuberneters Cluster 'PrOd' ignore casing k8s.cluster-name!=*"PrOd" ``` ``` // Get all targets with a container name that contains "Hot-DeAls" ignore casing k8s.container.name~*"Hot-DeAls" ``` ``` // Query for all container targets whose maintainer does not contain "jAne" ignore casing container.label.maintainer!~*"jAne" ``` You can also check for the presence or absence of a certain key using `IS PRESENT` and `IS NOT PRESENT`. ``` // Cluster-name is set, so getting all targets running in a Kubernetes cluster k8s.cluster-name IS PRESENT ``` ``` // Kubernetes label service-tier isn't set on a target k8s.label.service-tier IS NOT PRESENT ``` #### Using variables and placeholders You can use an experiment [variable](/use-steadybit/experiments/variables) (`{{...}}`) or a template placeholder (`[[...]]`) as a value in a query. Markers may be written with or without quotes: ``` // Reference a variable as the value k8s.deployment={{deployment}} // Equivalent, quoted form k8s.deployment="{{deployment}}" ``` When a variable holds several values (a [list of fixed values](/use-steadybit/experiments/variables#list-of-fixed-values) or a multi-value [dynamic value](/use-steadybit/experiments/variables#dynamic-value)), using it inside `IN (...)` matches *any* of its values: ``` // Matches every deployment the variable resolves to k8s.deployment IN ({{deployment}}) ``` {% hint style="info" %} A `=` (or `!=`) comparison against a multi-value variable also works: it matches when the attribute equals **any** of the values (and `!=` when it equals **none**), i.e. it is equivalent to `IN (...)` (`NOT IN (...)`). The Query UI displays it in the `IN (...)` form. {% endhint %} #### Aggregations To aggregate a key's value, you can use the `COUNT` function to check for the number of distinct values with numeric operators like `<`,`<=`,`=`,`>=` and `>`. ``` // Only find targets running in at least two AWS availability zones COUNT(aws.zone) >= 2 ``` ``` // Only find targets running just a single pod replica COUNT(k8s.pod.name) = 1 ``` #### Expression Concatenation Simple expressions can be chained with AND & OR. ``` // Get all targets of Kubernetes cluster "prod" or "staging" k8s.cluster-name="prod" OR k8s.cluster-name="staging" ``` ``` // Get all targets of Kubernetes cluster "prod" running on the host "ip-1-2-3-4" k8s.cluster-name="prod" AND host.hostname="ip-1-2-3-4" ``` #### Expression Negation You can negate a specific key-value expression using NOT. ``` // Matches all, but not prod NOT k8s.cluster-name="prod" ``` #### Parenthesis Expression blocks can be encapsulated using parenthesis. ``` (k8s.cluster-name="prod" OR k8s.cluster-name="staging") AND aws.zone="eu-central-1b" ``` #### Quoting Special Characters Keys containing special characters like `:` and `/` needs to be quoted to work properly. ``` // Quoting keys with special characters is necessary "label.aws:ec2launchtemplate/version"="some value" ``` # Install Agent and Extensions ### Overview Before installing the agent, we recommend to make yourself familiar with the general [Agent Architecture](/install-and-configure/install-agent/agent-architecture). ### Prerequisites To install the agent, you must be able to access the following URLs via HTTPS (443) on your target environment: * (Platform) * (.deb and .rpm packages; Linux installation only) * (Setup Scripts; one-liner for Docker or Linux installation only) * (Kubernetes installation via helm only) * and and https\://\*.docker.io (Container Images; Kubernetes and Docker installation only) {% hint style="info" %} If you want to check if your environment is properly configured to install and run an agent, you run our `env-check`tool on the target machine: ```bash curl -sfL https://get.steadybit.com/env-check.sh | sh -s ``` {% endhint %} {% hint style="warning" %} **Container image retention.** Steadybit publishes container images for the agent, extensions on `ghcr.io/steadybit/...` and retains each image for **1 year** after publication. Older images — including tagged releases — are pruned automatically. If you need to keep specific versions available beyond that window, mirror them into your own registry before they expire. {% endhint %} ### Setup of Agents We offer the following deployment options: * [Docker](/install-and-configure/install-agent/install-as-docker-container) * [Kubernetes](/install-and-configure/install-agent/install-on-kubernetes) * [Host](/install-and-configure/install-agent/install-on-linux-hosts) * [Windows](/install-and-configure/install-agent/install-on-windows-hosts) More configuration options can be found in [Advanced Configuration](/install-and-configure/install-agent/advanced-configuration) # Architecture The core of the Steadybit architecture is the agent-based approach consisting of * a central platform, being the center of control for you * one agent per network boundary, deployed somewhere in your system and works as a channel between the platform and multiple extensions * extensions, deployed wherever you need them, to discover running infrastructure components (hosts, containers, Kubernetes resources, etc.) and run actions (like, e.g., chaos engineering attacks or checks). The following diagram shows this in an example of three hosts, one with the agent installed to communicate with the installed extensions and platform. * The agent is deployed only once, as everything runs within one network. * Also, the AWS and Kubernetes extensions only run once as they work with the respective API. * The extensions for host- and container-based chaos engineering need to run per host to reach the underlying resources (e.g., network, CPU, memory) ![Steadybit agent and extension architecture](/files/Zh0pGthzXmfKkJ5dDjL5) The Steadybit agent periodically polls its registered extensions for discovery data using HTTP. For each call, only the delta of the discovery data is sent to the platform.\ If an experiment is to be executed for the agent, the agent connects via a websocket to the platform and receives the actions to be executed. The agent will then control the action execution and calls the respective HTTP endpoints of the extension. If the connection between the platform and agent or agent and extension is interrupted, the agent immediately stops and rollbacks any active action. {% hint style="info" %} The Steadybit platform never connects to the agent or any extensions:\ \&#xNAN;*all connections are initiated by the agent*, regardless of the deployment model. {% endhint %} Check our [reliability hub](https://hub.steadybit.com/extensions) for an up-to-date list of extensions. Our extensions are Open Source and available on [github.com](https://github.com/steadybit). They are written in Go to ensure the best possible resource usage. # Install on Kubernetes This method will install the Steadybit Agent on your Kubernetes Cluster using [Helm](https://helm.sh). So you need to have helm installed. ## Prerequisites You must be able to access the following URLs via HTTPS (443) on your target environment: * (Platform) * (Kubernetes helm repository) * and (Container Images) ## Installation To deploy the agent to Kubernetes, you can copy the installation script from the [setup page](https://platform.steadybit.com/settings/agents/setup) in the SaaS platform. Alternatively, you can update the script below with the name of your Kubernetes cluster and your agent key, which you find in the platform's [setup page](https://platform.steadybit.com/settings/agents/setup). Just run the script while connected to your Kubernetes cluster. ```shell helm repo add steadybit https://steadybit.github.io/helm-charts helm repo update helm install steadybit-agent --namespace steadybit-agent \ --create-namespace \ --set agent.key= \ --set global.clusterName= \ steadybit/steadybit-agent ``` To configure the installation, specify the values on the command line using the `--set` flag, or provide a YAML file with your values using the `-f` flag. For more configuration options have a look at our [steadybit/helm-charts repository](https://github.com/steadybit/helm-charts/tree/main/charts/steadybit-agent). ### Configure Container Runtime By default, the agent assumes that your cluster uses the `containerd` runtime. If this is not the case, you need to add`--set extension-container.container.engine=docker` or `cri-o`. #### Determine Container Runtime on a Node In case of a managed Kubernetes service, there might be a vendor-specific method to check the container runtime. In case you have access on the `kubectl`, you can get the container runtime via ```shell kubectl get nodes -o wide ``` The column `CONTAINER-RUNTIME` shows you the runtime you are using. You can find further in the [official Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/). ### Configure custom On-Prem Platform If you are running on-prem, make sure to set the URL of the platform: `--set agent.registerUrl=` ### Extensions The agent's purpose is to establish a communication from the platform to your systems. However, the discovery of your systems and providing Chaos Engineering to a technology works via extensions. By default, the official helm chart already installs the extensions [extension-container](https://hub.steadybit.com/extension/com.steadybit.extension_container), [extension-host](https://hub.steadybit.com/extension/com.steadybit.extension_host), [extension-http](https://hub.steadybit.com/extension/com.steadybit.extension_http) and [extension-kubernetes](https://hub.steadybit.com/extension/com.steadybit.extension_kubernetes). In order to discover all extensions in the Kubernetes custer the agent needs to access the Kubernetes API. For more details, please take a look at the [Kubernetes Auto Registration](/install-and-configure/install-agent/extension-registration#kubernetes-auto-registration) documentation. #### Add more Extensions Further extensions can be enabled via helm values of steadybit-agent. For example, [`extension-postman`](https://github.com/steadybit/extension-postman) can be enabled by setting`extension-postman.enabled=true`. See our [GitHub Repository](https://github.com/steadybit/helm-charts/tree/main/charts/steadybit-agent) for a detailed list of all the configuration parameters. Alternatively, you can also install extensions independently with their own helm charts. The installation instructions are listed in the Github repositories of the extension and can be browsed via the [Reliability Hub](https://hub.steadybit.com/). #### Disable Default Extensions If you want to disable some, or all, of the default extensions, please set the following values in helm: * [extension-container](https://hub.steadybit.com/extension/com.steadybit.extension_container): `--set extension-container.enabled=false` * [extension-host](https://hub.steadybit.com/extension/com.steadybit.extension_host): `--set extension-host.enabled=false` * [extension-http](https://hub.steadybit.com/extension/com.steadybit.extension_http): `--set extension-http.enabled=false` * [extension-kubernetes](https://hub.steadybit.com/extension/com.steadybit.extension_kubernetes): `--set extension-kubernetes.enabled=false` ### Alternative: OpenShift installation (< 4.18) The SecurityContextConstraints for OpenShift are included in our helm chart. You need to configure the CRI-O container runtime and we're good to go. ```bash helm repo add steadybit https://steadybit.github.io/helm-charts helm repo update helm template steadybit-agent --namespace steadybit-agent \ --create-namespace \ --set agent.key= \ --set global.clusterName= \ --set extension-container.container.engine=cri-o \ steadybit/steadybit-agent ``` ### Alternative: OpenShift installation (>= 4.18) The SecurityContextConstraints for OpenShift are included in our helm chart. You need to configure the CRI-O container runtime and to use the crun container engine and we're good to go. ```bash helm repo add steadybit https://steadybit.github.io/helm-charts helm repo update helm template steadybit-agent --namespace steadybit-agent \ --create-namespace \ --set agent.key= \ --set global.clusterName= \ --set extension-container.container.engine=cri-o \ --set extension-container.containerEngines.cri-o.ociRuntime.path=crun \ --set extension-container.containerEngines.cri-o.ociRuntime.root=/run/crun \ steadybit/steadybit-agent ``` ### Alternative: GKE Autopilot installation You can install the agent and extensions on Google Kubernetes Engine Autopilot clusters (1.32.1-gke.1729000 or later). Due to restrictions by GKE host attacks won't be available. For the container extension to work, you first need to apply a workload allow list: ```bash kubectl apply -f - <<'EOF' apiVersion: auto.gke.io/v1 kind: AllowlistSynchronizer metadata: name: steadybit-synchronizer spec: allowlistPaths: - Steadybit/extension-container/* EOF kubectl wait --for=condition=Ready allowlistsynchronizer/steadybit-synchronizer --timeout=60s ``` After this, you're ready to deploy the agent while specifying the cluster name and agent key. ``` helm repo add steadybit https://steadybit.github.io/helm-charts helm repo update helm upgrade --install steadybit-agent --namespace steadybit-agent \ --create-namespace \ --set agent.key= \ --set global.clusterName= \ --set extension-container.container.engine=containerd \ --set extension-container.platform=gke-autopilot \ --set extension-host.enabled=false \ --set agent.registerUrl=https://platform.steadybit.com \ steadybit/steadybit-agent ``` ### Alternative: Generate Kubernetes Manifests We currently don't provide a static Kubernetes manifest, but you can generate it from the helm chart. We recommend to use the helm chart, as it is easier to update the agent and extensions. ```shell helm repo add steadybit https://steadybit.github.io/helm-charts helm repo update helm template steadybit-agent --namespace steadybit-agent \ --create-namespace \ --set agent.key= \ --set global.clusterName= \ steadybit/steadybit-agent ``` ## Resource limits Keep an eye on OOMing / crash looping agents and extensions after installation. The memory usage highly depends on the number of discovered targets. We try to provide reasonable defaults, but you might need to adapt the resource limits to your use case. For example, to increase the memory limits for the agent: ```shell --set agent.resources.limits.memory=1Gi ``` ## Update To update the agent and extensions, you can use the following command: ```shell helm repo update helm upgrade --reuse-values steadybit-agent --namespace steadybit-agent \ steadybit/steadybit-agent ``` # Install on Linux Hosts Using the respective package managers, this method will install the Steadybit agent on your Linux machine. By default, it will install the agent as well as the extensions [extension-http](https://hub.steadybit.com/extension/com.steadybit.extension_http), [extension-container](https://hub.steadybit.com/extension/com.steadybit.extension_container), and [extension-host](https://hub.steadybit.com/extension/com.steadybit.extension_host). ## Supported Package Managers * apt * dnf * yum ## Prerequisites To install the agent, you must be able to access the following URLs via HTTPS (443) on your target environment: * (Platform) * with redirect to and europe-west1-apt.pkg.dev (.deb and .rpm packages) * (Setup Scripts) ## Agent Installation To install the agent on your Linux system, copy the installation script from the SaaS platform's [setup page](https://platform.steadybit.com/settings/agents/setup). Alternatively, you can update and run the script below with your agent key, which you find on the platform's [setup page](https://platform.steadybit.com/settings/agents/setup): ```shell wget https://get.steadybit.com/agent-linux.sh chmod a+x agent-linux.sh ./agent-linux.sh --key ``` | Parameter | Description | Default | | ---------------- | --------------------------------------------------------- | --------------------------------------------------------------------------------- | | `--key` | The API key the agent uses | | | `--extensions` | The extensions to install | `steadybit-extension-host,steadybit-extension-container,steadybit-extension-http` | | `--platform-url` | If running on-prem, the URL of your platform installation | `https://platform.steadybit.com` | | `--version` | Override the used package version | | The package repositories have a pull limit. If you're installing the packages on a large number of servers, please consider using a mirror repository. ### Configure HTTP Proxy Server The Steadybit Agent uses HTTP and WebSockets to communicate with the platform and to download updates. To simplify the agent deployment, consider allowing direct communication to our platform. If you require a single entry into and out of your network, you can configure the agent to use a proxy: 1. Edit `/etc/steadybit/agent` and set the values for these variables: ```shell STEADYBIT_AGENT_PROXY_HOST="" STEADYBIT_AGENT_PROXY_PORT="" STEADYBIT_AGENT_PROXY_PROTOCOL="" STEADYBIT_AGENT_PROXY_USER="" STEADYBIT_AGENT_PROXY_PASSWORD="" ``` 2. Restart the service When using **systemd** ```shell systemctl daemon-reload && systemctl restart steadybit-agent ``` When using **InitV** ```shell service steadybit-agent restart ``` ### Configure container runtime For Linux installations, the docker runtime is auto-configured by the extension by looking at the system paths. If this is incorrectly auto-configured, it can be set in `/etc/steadybit/extension-container` using `STEADYBIT_EXTENSION_CONTAINER_SOCKET` and `STEADYBIT_EXTENSION_CONTAINER_RUNTIME`. If you have installed docker just before, etc., please restart your host or the needed services. ## Managing the Agent and Extension ### Logs The logs for agent and extensions are located in `/var/log/steadybit-*.log` ### Configuration The configurations for agent and extensions are located in `/etc/steadybit` ### Start/Stop When using **systemd** ```shell systemctl start steadybit-agent systemctl stop steadybit-agent systemctl restart steadybit-agent ``` When using **InitV** ```shell service steadybit-agent start service steadybit-agent stop service steadybit-agent restart ``` The same applies to extensions. The services are named `steadybit-extension-*`. ## Removing the Agent Remove the packages using the package manager: ```shell apt-get remove --purge steadybit-agent \ steadybit-extension-http \ steadybit-extension-container \ steadybit-extension-host ``` Or when using yum: ```shell yum remove steadybit-agent \ steadybit-extension-http \ steadybit-extension-container \ steadybit-extension-host ``` ## Additional Extensions If you want to use additional extensions (e.g. [extension-jvm](https://hub.steadybit.com/extension/com.steadybit.extension_jvm) for attacking Java applications), you can apply the `--extensions` parameter. 
./agent-linux.sh --key <agent-key> --extensions steadybit-extension-host,steadybit-extension-container,steadybit-extension-http,steadybit-extension-jvm
## Using a single agent across multiple hosts By default, the `agent-linux.sh` installs the agent and extension on a single host. When installed on multiple hosts, each runs an agent instance, allocating many resources. It is possible to install the agent on a dedicated host and only the container and host extension on each host. #### Non-Agent Hosts * Install only container and host extensions by using `--no-agent` flag ```bash ./agent-linux.sh --key --no-agent --extensions steadybit-extension-host,steadybit-extension-container ``` * Reconfigure the extension to listen on http by editing `/etc/steadybit/extension-container` and `/etc/steadybit/extension-host` and removing the `STEADYBIT_EXTENSION_UNIX_SOCKET=...` line.\ If you want to override the default port, set `STEADYBIT_EXTENSION_PORT`. * Restart the extensions ```bash systemctl restart steadybit-extension-container systemctl restart steadybit-extension-host ``` #### Agent Host * Install the Agent as usual * For each extension running on a different host, create a `/etc/steadybit/extensions.d/extension-*.yaml` to configure the remote-located extensions {% code title="/etc/steadybit/extensions.d/extension-container-172-31-16-248.yaml" %} ```yaml url: http://172.31.16.248:8085 # replace with your host types: - ACTION - DISCOVERY ``` {% endcode %} # Install on Windows Hosts Steadybit Agent is packaged as a Windows *.msi* installer. Unlike the Linux installation, the [extension-host-windows](https://hub.steadybit.com/target/com.steadybit.extension_host_windows.host) is not bundled and has to be installed separately. ## Agent installation To install the agent on your Windows system, download the latest [installer](https://windows-registry.steadybit.com/steadybit-agent/windows-agent-latest.zip) archive and extract the installer into a folder of your choice. You may also download the [SHA-256 checksum](https://windows-registry.steadybit.com/steadybit-agent/windows-agent-latest.sha256) and verify it matches the hash of the downloaded archive. Looking for a specific version? Find it in the [Windows Registry](https://windows-registry.steadybit.com). ### Regular installation (GUI) Double-click the downloaded *.msi* installer and follow the installation instructions. During the installation process you must specify the *Agent Key* and *Registration URL*, and are also able to configure proxy settings in a dedicated dialog. ![Windows Agent Installer](/files/pYtrr9KdhUchcL2XnMMy) ### Headless installation (No GUI) Run the following command from the **administrator** powershell: ```pwsh msiexec /i SteadybitAgentInstaller.msi \` STEADYBIT_AGENT_KEY="" \` STEADYBIT_AGENT_REGISTER_URL="" \` /qn ``` ### Configure HTTP Proxy Server The Steadybit Agent uses HTTP and WebSockets to communicate with the platform. To simplify the agent deployment, consider allowing direct communication to our platform. If you require a single entry into and out of your network, you can configure the agent to use a proxy server. Enter your proxy server settings during installation in the GUI or add the following command line flags to the terminal installation command: ```pwsh STEADYBIT_AGENT_PROXY_HOST="" STEADYBIT_AGENT_PROXY_PORT="" STEADYBIT_AGENT_PROXY_PROTOCOL="" STEADYBIT_AGENT_PROXY_USER="" STEADYBIT_AGENT_PROXY_PASSWORD="" ``` You may also add these entries as environment variables and restart the *SteadybitAgent* service. ## Start/Stop {% hint style="info" %} The Windows Service runs as the Local System account and inherits its privileges. {% endhint %} During the agent installation, an application service is automatically initialized for you. If you want to manually control whether the agent is running or not, use the following commands: ```pwsh Start-Service SteadybitAgent ``` ```pwsh Restart-Service SteadybitAgent ``` ```pwsh Stop-Service SteadybitAgent ``` ## Logs ### File System Navigate to `C:\ProgramData\Steadybit GmbH\Steadybit Agent\steadybit-agent.log` ### Event Viewer If you need to diagnose issues or review the activity of the agent and its extensions, their logs are recorded in the **Windows Application Event Log**. Here's how to access them: 1. **Open Event Viewer:** * Press `WINKEY + R` to open the Run dialog. * Type `eventvwr.msc` and press Enter. * Alternatively, search for "Event Viewer" in the Windows Start Menu. 2. **Navigate to Application Logs:** * In the Event Viewer window, on the left-hand pane, expand **Windows Logs**. * Select **Application**. 3. **Find Relevant Entries:** * The central pane will now display a list of events. Look for entries where the **Source** column might indicate the agent or its extensions (e.g., "Steadybit Agent," or a specific extension name). * You can sort by "Date and Time" to find recent events or use the "Filter Current Log..." option in the right-hand pane to narrow down your search (e.g., by Event level like "Error" or "Warning", or by specific Event sources)." ## Java Virtual Machine Options Additional JVM options — for example to tune the heap size — go into the agent's JVM options file. Edit it and restart the *SteadybitAgent* service to apply your changes. Its location depends on the installer version: * Recent installers (≥ 2.4.0): `C:\Program Files\Steadybit GmbH\Steadybit Agent\app\agent.cfg`, with one option per line, each prefixed with `java-options=`: ``` java-options=-Xms16m java-options=-Xmx1024m ``` * Older installers (< 2.4.0): `C:\Program Files\Steadybit GmbH\Steadybit Agent\Core\agent.l4j.ini`, with one option per line: ``` -Xms16m -Xmx1024m ``` ## Certificates By default, the Steadybit Agent (version 2.2.2 and later) will pick up the Certificates trusted by the system. If you don't want this, remove the option `-Djavax.net.ssl.trustStoreType=Windows-ROOT` from the agent's [JVM options file](#java-virtual-machine-options) and restart the *SteadybitAgent* service. It will then use the bundled Java runtime's keystore, which can be managed using `keytool`. The runtime is located at `C:\Program Files\Steadybit GmbH\Steadybit Agent\runtime` (≥ 2.4.0) or `C:\Program Files\Steadybit GmbH\Steadybit Agent\Core\jre` (< 2.4.0). ## Uninstalling the Steadybit Agent Here are two methods to remove the Steadybit Agent from your Windows computer: ### Method 1: Using Windows Settings This method uses the modern Windows Settings panel, which is common in Windows 10 and 11. 1. **Open Windows Settings:** * Click the **Start** button (the Windows icon, usually in the bottom-left corner). * Click the **Settings** cogwheel icon ⚙️. 2. **Navigate to Apps:** * In the Settings window, click on **Apps**. 3. **Find Installed Apps:** * In the Apps section, click on **Installed apps** (or it might be called **Apps & features** on some older Windows 10 versions). 4. **Locate Steadybit Agent:** * You'll see a list of all applications installed on your computer. Scroll through this list or use the search bar at the top of the list to find "**Steadybit Agent**". 5. **Initiate Uninstall:** * Once you find "Steadybit Agent": * Click the three vertical dots (`...`) next to "Steadybit Agent" and select **Uninstall**. 6. **Follow Uninstaller Prompts:** * A confirmation pop-up might appear asking if you're sure. Click **Uninstall** again if prompted. * The Steadybit Agent's own uninstaller will then launch. Follow any instructions it provides to complete the removal. ### Method 2: Using Control Panel This method uses the traditional Control Panel interface, which has been around for many Windows versions. 1. **Open the Run Dialog:** * Press the **`WINKEY`** (the key with the Windows logo) + **`R`** key on your keyboard at the same time. This will open the "Run" dialog box. * Alternatively, you can search for "Run" in the Start menu. 2. **Open Programs and Features:** * In the Run dialog box, type **`appwiz.cpl`** and press **Enter** or click **OK**. * This will directly open the "Programs and Features" window (sometimes titled "Uninstall or change a program"). 3. **Locate Steadybit Agent:** * You'll see a list of installed programs. Scroll through this list to find "**Steadybit Agent**". You can also click on the "Name" column to sort the list alphabetically, which might make it easier to find. 4. **Initiate Uninstall:** * Select "**Steadybit Agent**" from the list by clicking on it once. * An **Uninstall** button (or sometimes "Uninstall/Change") will appear at the top of the list or when you right-click the entry. Click this button. 5. **Follow Uninstaller Prompts:** * A confirmation pop-up might appear. Confirm that you want to uninstall. * The Steadybit Agent's own uninstaller will then launch. Follow any instructions it provides to complete the removal. *** # Install using Docker Compose This method will install the Steadybit Agent on your machine using Docker compose. It will run the agent as well as the [extension-http](https://hub.steadybit.com/extension/com.steadybit.extension_http), [extension-container](https://hub.steadybit.com/extension/com.steadybit.extension_container), and [extension-host](https://hub.steadybit.com/extension/com.steadybit.extension_host). ## Prerequisites To install the agent, you must be able to access the following URLs via HTTPS (443) on your target environment: * (Platform) * (Setup Scripts) * and (Container Images) ## Supported Platforms: * Linux (needs the Docker and Docker compose plugin installed) * Windows (using Docker Desktop using the WSL2 engine) * MacOS (using Docker Desktop) {% hint style="warning" %} **Docker Desktop:** You're only able to attack containers and processes running on Docker **Windows Subsystem for Linux:** With the default Kernel you won't be able to execute network attacks. {% endhint %} ## Agent Installation To deploy the agent to Docker, you can copy the installation script from the [setup page](https://platform.steadybit.com/settings/agents/setup) in the SaaS platform. Alternatively, you can update and run the script below with your agent key, which you find in the platform's [setup page](https://platform.steadybit.com/settings/agents/setup): ```bash wget https://get.steadybit.com/agent.sh chmod a+x agent.sh ./agent.sh --key ``` | Parameter | Description | Default | | ---------------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | `` | The action to take. Either one of `up`, `down`, `restart`, `config` | `up` | | `--key` | The agent key. Can be found on your platform's \[ page). | | | `--platform-url` | If running on-prem, the url of your platform installation to use | `https://platform.steadybit.com` | | `--image` | The Agent Docker image to use. | `steadybit/agent:latest` | {% hint style="info" %} In case you don't want to deploy to Docker directly using the script or you need to modify it, you can use the `config` command to print the Docker compose configuration and apply it yourself. {% endhint %} ### Configure HTTP Proxy Server The Steadybit Agent uses HTTP and websockets to communicate with the platform. To simplify the agent deployment, consider to allow direct communication to our platform. If a single entry into and out of your network is required, you can configure the agent to use an HTTP proxy. Export the following environment variables before running the script or edit the Docker compose configuration printed by the `config` command. ```bash export STEADYBIT_AGENT_PROXY_HOST="" export STEADYBIT_AGENT_PROXY_PORT="" export STEADYBIT_AGENT_PROXY_PROTOCOL="" export STEADYBIT_AGENT_PROXY_USER="" export STEADYBIT_AGENT_PROXY_PASSWORD="" ./agent.sh --key ``` ## Additional Extensions To use additional extensions (e.g. [extension-jvm](https://hub.steadybit.com/extension/com.steadybit.extension_jvm) for attacking Java applications), you need to edit the Docker compose file generated using the `config` command: Add the extension as an additional service in the Docker compose and register it via environment variables to the agent. See [extension installation](/integrate-with-steadybit/extensions/extension-installation) to learn more about extension registration. # Install on Amazon ECS The Steadybit Agent can be installed on **Amazon Elastic Container Service**. ## Amazon ECS with EC2 ### Compatibility [extension-host](https://hub.steadybit.com/extension/com.steadybit.extension_host) and [extension-container](https://hub.steadybit.com/extension/com.steadybit.extension_container) needs to run in privileged mode and network mode `host` is required for the extensions. ### Installation #### Pre-requisites * You need to have an ECS cluster running with at least one EC2 instance. * You need to know the (private) Subnet-IDs where you want to place the agent and extension tasks. * The security group used by the ec2 instances need to allow inbound traffic to the `extension-host` and `extension-container` (ports 8085 and 8086) as they are running as daemon service using the host network. #### Step-By-Step Guide 1. Copy the required Files * [steadybit-agent.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-agent.json) * [steadybit-agent-role-trust-policy.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-agent-role-trust-policy.json) * [steadybit-agent-role-permissions.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-agent-role-permissions.json) * [steadybit-extension-host.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-extension-host.json) * [steadybit-extension-container.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-extension-container.json) * [steadybit-extension-http.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-extension-http.json) * [steadybit-extension-aws.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-extension-aws.json) * [steadybit-extension-aws-role-permissions.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-extension-aws-role-permissions.json) 2. The agent needs some permissions to be able to look up extensions running in the cluster. Create a IAM role for the agent task with the following permissions: ```bash aws iam create-role --role-name steadybit-agent-task-role --assume-role-policy-document file://steadybit-agent-role-trust-policy.json aws iam put-role-policy --role-name steadybit-agent-task-role --policy-name steadybit-agent-extension-lookup --policy-document file://steadybit-agent-role-permissions.json ``` 3. If you like to install the `extension-aws` you need to create a new IAM role with the following permissions. Please have a look at the [extension documentation](https://github.com/steadybit/extension-aws?tab=readme-ov-file#required-permissions-policies) for the latest list of required permissions. ```bash aws iam create-role --role-name steadybit-extension-aws-task-role --assume-role-policy-document file://steadybit-agent-role-trust-policy.json aws iam put-role-policy --role-name steadybit-extension-aws-task-role --policy-name steadybit-extension-aws --policy-document file://steadybit-extension-aws-role-permissions.json ``` 4. Replace all placeholders in the JSON files with your values. All placeholders are prefixed with `MY-`. Take care, the placeholders are used multiple times in the JSON files. * `MY-AGENT-KEY`: Your agent key * `MY-CLUSTER-NAME`: The name of your ECS cluster * `MY-PLATFORM-URL`: The URL of your Steadybit platform, for SaaS use `https://platform.steadybit.com` * `MY-REGION`: The AWS region where your ECS cluster is running * `MY-ACCOUNT`: The AWS account ID 5. Register the Task Definitions ```bash aws ecs register-task-definition --cli-input-json file://steadybit-agent.json aws ecs register-task-definition --cli-input-json file://steadybit-extension-host.json aws ecs register-task-definition --cli-input-json file://steadybit-extension-container.json aws ecs register-task-definition --cli-input-json file://steadybit-extension-http.json aws ecs register-task-definition --cli-input-json file://steadybit-extension-aws.json ``` 6. Create the Services * **Agent** - please replace the cluster-name, subnet-ids, and security-group-id with your values. The security group needs to allow outbound traffic to the Steadybit platform and to the extensions (ports 8080-8099). ```bash aws ecs create-service \ --cluster MY-CLUSTER \ --service-name steadybit-agent \ --task-definition steadybit-agent \ --propagate-tags TASK_DEFINITION \ --desired-count 1 \ --deployment-configuration maximumPercent=101,minimumHealthyPercent=0 \ --tags key=steadybit.com/discovery-disabled,value=true \ --network-configuration '{"awsvpcConfiguration": {"subnets": ["MY-SUBNET-1", "MY-SUBNET-2", "MY-SUBNET-3"], "securityGroups": ["MY-SECURITY-GROUP-ID"], "assignPublicIp": "DISABLED"}}' ``` * **Extension Host** - please replace the cluster-name. The extension will use the host network strategy and use the security groupd and subnets of your ec2 instances. ```bash aws ecs create-service \ --cluster MY-CLUSTER \ --service-name steadybit-extension-host \ --task-definition steadybit-extension-host \ --propagate-tags TASK_DEFINITION \ --launch-type EC2 \ --tags key=steadybit.com/discovery-disabled,value=true \ --scheduling-strategy DAEMON ``` * **Extension Container** - please replace the cluster-name. The extension will use the host network strategy and use the security groupd and subnets of your ec2 instances. ```bash aws ecs create-service \ --cluster MY-CLUSTER \ --service-name steadybit-extension-container \ --task-definition steadybit-extension-container \ --propagate-tags TASK_DEFINITION \ --launch-type EC2 \ --tags key=steadybit.com/discovery-disabled,value=true \ --scheduling-strategy DAEMON ``` * **Extension HTTP** - please replace the cluster-name, subnet-ids, and security-group-id with your values. The security group needs to allow inbound traffic to the extension (port 8085) and outbound traffic to all ports/destination you want to reach out with the http checks implemented in the extension. ```bash aws ecs create-service \ --cluster MY-CLUSTER \ --service-name steadybit-extension-http \ --task-definition steadybit-extension-http \ --propagate-tags TASK_DEFINITION \ --desired-count 1 \ --deployment-configuration maximumPercent=101,minimumHealthyPercent=0 \ --tags key=steadybit.com/discovery-disabled,value=true \ --network-configuration '{"awsvpcConfiguration": {"subnets": ["MY-SUBNET-1", "MY-SUBNET-2", "MY-SUBNET-3"], "securityGroups": ["MY-SECURITY-GROUP-ID"], "assignPublicIp": "DISABLED"}}' ``` * **Extension AWS** - please replace the cluster-name, subnet-ids, and security-group-id with your values. The security group needs to allow inbound traffic to the extension (port 8085) ```bash aws ecs create-service \ --cluster MY-CLUSTER \ --service-name steadybit-extension-aws \ --task-definition steadybit-extension-aws \ --propagate-tags TASK_DEFINITION \ --desired-count 1 \ --deployment-configuration maximumPercent=101,minimumHealthyPercent=0 \ --tags key=steadybit.com/discovery-disabled,value=true \ --network-configuration '{"awsvpcConfiguration": {"subnets": ["MY-SUBNET-1", "MY-SUBNET-2", "MY-SUBNET-3"], "securityGroups": ["MY-SECURITY-GROUP-ID"], "assignPublicIp": "DISABLED"}}' ``` ## Amazon ECS with Fargate ### Compatibility The agent and most of the extensions can be run as an ECS service in Fargate. However, [extension-host](https://hub.steadybit.com/extension/com.steadybit.extension_host) and [extension-container](https://hub.steadybit.com/extension/com.steadybit.extension_container) are not compatible with AWS Fargate because they require access to the underlying compute instance which is not possible with Fargate. [extension-aws](https://hub.steadybit.com/extension/com.steadybit.extension_aws) can also be used with Fargate and offers some alternative actions to discover and attack ECS resources. ### Installation 1. Copy the required Files * [steadybit-agent-fargate.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-agent-fargate.json) * [steadybit-agent-role-trust-policy.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-agent-role-trust-policy.json) * [steadybit-agent-role-permissions.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-agent-role-permissions.json) * [steadybit-extension-http-fargate.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-extension-http-fargate.json) * [steadybit-extension-aws-fargate.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-extension-aws-fargate.json) * [steadybit-extension-aws-role-permissions.json](https://github.com/steadybit/docs-public/blob/main/install-and-configure/install-agent/ecs/steadybit-extension-aws-role-permissions.json) 2. The agent needs some permissions to be able to look up extensions running in the cluster. Create a IAM role for the agent task with the following permissions: ```bash aws iam create-role --role-name steadybit-agent-task-role --assume-role-policy-document file://steadybit-agent-role-trust-policy.json aws iam put-role-policy --role-name steadybit-agent-task-role --policy-name steadybit-agent-extension-lookup --policy-document file://steadybit-agent-role-permissions.json ``` 3. Fargate tasks needs a task execution role to be able to write logs. If you don't already have an existing role, you can create one via: ```bash aws iam create-role --role-name steadybit-agent-task-execution-role --assume-role-policy-document file://steadybit-agent-role-trust-policy.json aws iam attach-role-policy --role-name steadybit-agent-task-execution-role --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy aws iam attach-role-policy --role-name steadybit-agent-task-execution-role --policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess ``` 4. If you like to install the `extension-aws` you need to create a new IAM role with the following permissions. Please have a look at the [extension documentation](https://github.com/steadybit/extension-aws?tab=readme-ov-file#required-permissions-policies) for the latest list of required permissions. ```bash aws iam create-role --role-name steadybit-extension-aws-task-role --assume-role-policy-document file://steadybit-agent-role-trust-policy.json aws iam put-role-policy --role-name steadybit-extension-aws-task-role --policy-name steadybit-extension-aws --policy-document file://steadybit-extension-aws-role-permissions.json ``` 5. Replace all placeholders in the JSON files with your values. All placeholders are prefixed with `MY-`. Take care, the placeholders are used multiple times in the JSON files. * `MY-AGENT-KEY`: Your agent key * `MY-CLUSTER-NAME`: The name of your ECS cluster * `MY-PLATFORM-URL`: The URL of your Steadybit platform, for SaaS use `https://platform.steadybit.com` * `MY-REGION`: The AWS region where your ECS cluster is running * `MY-ACCOUNT`: The AWS account ID 6. Register the Task Definitions ```bash aws ecs register-task-definition --cli-input-json file://steadybit-agent-fargate.json aws ecs register-task-definition --cli-input-json file://steadybit-extension-http-fargate.json aws ecs register-task-definition --cli-input-json file://steadybit-extension-aws-fargate.json ``` 7. Create the Services * **Agent** - please replace the cluster-name, subnet-ids, and security-group-id with your values. The security group needs to allow outbound traffic to the Steadybit platform and to the extensions (ports 8080-8099). ```bash aws ecs create-service \ --cluster MY-CLUSTER \ --service-name steadybit-agent \ --task-definition steadybit-agent-fargate \ --propagate-tags TASK_DEFINITION \ --launch-type FARGATE \ --desired-count 1 \ --deployment-configuration maximumPercent=101,minimumHealthyPercent=0 \ --tags key=steadybit.com/discovery-disabled,value=true \ --network-configuration '{"awsvpcConfiguration": {"subnets": ["MY-SUBNET-1", "MY-SUBNET-2", "MY-SUBNET-3"], "securityGroups": ["MY-SECURITY-GROUP-ID"], "assignPublicIp": "DISABLED"}}' ``` * **Extension HTTP** - please replace the cluster-name, subnet-ids, and security-group-id with your values. The security group needs to allow inbound traffic to the extension (port 8085) and outbound traffic to all ports/destination you want to reach out with the http checks implemented in the extension. ```bash aws ecs create-service \ --cluster MY-CLUSTER \ --service-name steadybit-extension-http \ --task-definition steadybit-extension-http \ --propagate-tags TASK_DEFINITION \ --launch-type FARGATE \ --desired-count 1 \ --deployment-configuration maximumPercent=101,minimumHealthyPercent=0 \ --tags key=steadybit.com/discovery-disabled,value=true \ --network-configuration '{"awsvpcConfiguration": {"subnets": ["MY-SUBNET-1", "MY-SUBNET-2", "MY-SUBNET-3"], "securityGroups": ["MY-SECURITY-GROUP-ID"], "assignPublicIp": "DISABLED"}}' ``` * **Extension AWS** - please replace the cluster-name, subnet-ids, and security-group-id with your values. The security group needs to allow inbound traffic to the extension (port 8085) ```bash aws ecs create-service \ --cluster MY-CLUSTER \ --service-name steadybit-extension-aws \ --task-definition steadybit-extension-aws \ --propagate-tags TASK_DEFINITION \ --launch-type FARGATE \ --desired-count 1 \ --deployment-configuration maximumPercent=101,minimumHealthyPercent=0 \ --tags key=steadybit.com/discovery-disabled,value=true \ --network-configuration '{"awsvpcConfiguration": {"subnets": ["MY-SUBNET-1", "MY-SUBNET-2", "MY-SUBNET-3"], "securityGroups": ["MY-SECURITY-GROUP-ID"], "assignPublicIp": "DISABLED"}}' ``` ## FAQ * **Q:** How can I update the agent/extensions and force pulling a new image version when using `latest`? * **A:** `aws ecs update-service --cluster --service --force-new-deployment` * **Q:** Can I shell into the agent/extension tasks? * **A:** Yes, with ECS Exec, details can be found [here](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html), short summary below: * You need to allow update your service to allow it, e.g.: `aws ecs update-service --service steadybit-agent --cluster --enable-execute-command` * The task role needs the following permissions: * `ssmmessages:CreateControlChannel` * `ssmmessages:CreateDataChannel` * `ssmmessages:OpenControlChannel` * `ssmmessages:OpenDataChannel` * The task definition needs to include `initProcessEnabled` in the `linuxParameters`, e.g: ```json { "containerDefinitions": [ { "name": "steadybit-agent", ... "linuxParameters": { "initProcessEnabled": true } } ] } ``` * After that, you can shell into the agent task with `aws ecs execute-command --cluster --task --container --interactive --command "/bin/bash"` # Extension Registration ## Overview Extensions needs to be registered with the agent. There are multiple ways to do that: * [Kubernetes Auto Registration](#kubernetes-auto-registration) (default behavior when using Kubernetes with our helm charts) * [Environment Variables](#using-environment-variables) * [Configuration Files](#using-configuration-files) (default behavior when using Linux packages) * [Agent API](#using-the-agent-api) ### Kubernetes Auto Registration The agent is looking for services and pods with an annotation `steadybit.com/extension-auto-registration`. These annotations are already added to our extensions if you use our official helm charts. #### Permissions In order to automatically discover and register extensions present in the Kubernetes cluster, the agent needs access pod and service definitions via the Kubernetes API. Appropriate permissions are already configured in the official helm charts by creating a (cluster) role granting `get`, `list` and `watch` on `pods` and `services` and a corresponding (cluster) role binding to a service account used for the agent pod. The used `rules` property is listed below. ```yaml rules: - apiGroups: [ "" ] resources: - "services" - "pods" verbs: [ "get", "list", "watch" ] ``` #### Kubernetes Auto Registration fine tuning If you want to fine tune the auto registration, you can configure the mechanism to: * include only extension pods matching a given label selector * exclude extensions pod matching a given label selector * include only extension from a specific namespace {% tabs %} {% tab title="using Helm Chart" %} ```yaml agent: extensions: autoregistration: matchLabelsInclude: custom/extension-i-want-to-register: true matchLabelsExclude: app.kubernetes.io/name: extension-host namespace: my-namespace ``` {% endtab %} {% tab title="using Environment Variables" %} ``` STEADYBIT_AGENT_EXTENSIONS_AUTOREGISTRATION_MATCHLABELS_0_KEY=custom/extension-i-want-to-discover STEADYBIT_AGENT_EXTENSIONS_AUTOREGISTRATION_MATCHLABELS_0_VALUE=true STEADYBIT_AGENT_EXTENSIONS_AUTOREGISTRATION_MATCHLABELSEXCLUDE_0_KEY=app.kubernetes.io/name STEADYBIT_AGENT_EXTENSIONS_AUTOREGISTRATION_MATCHLABELSEXCLUDE_0_VALUE=extension-host STEADYBIT_AGENT_EXTENSIONS_AUTOREGISTRATION_NAMESPACE=my-namespace ``` {% endtab %} {% endtabs %} ### Using Environment Variables You can specify environment Variables via `agent.env` files or directly via the command line. Please note that these environment variables are index-based (referred to as `n`) to register multiple extension instances. Valid Types are: * `DISCOVERIES` referring to a [index response of a discovery](https://github.com/steadybit/discovery-kit/blob/main/docs/discovery-api.md#index-response). * `ACTIONS` referring to a [list of actions](https://github.com/steadybit/action-kit/blob/main/docs/action-api.md#action-list). * `EVENTS` referring to a [list of event listeners](https://github.com/steadybit/event-kit/blob/main/docs/event-api.md#event-listeners-list). * `ADVICE` referring to a [list of advices](https://github.com/steadybit/advice-kit/blob/main/docs/advice-api.md#index-response). {% tabs %} {% tab title="with specified type" %} |

Environment Variable
(n refers to the index of the extension's instance)
(type refers to the type of the extension's endpoint)

| Required | Description | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------- | | `STEADYBIT_AGENT_type_EXTENSIONS_n_URL` | yes | Fully-qualified URL of the endpoint, e.g., `http://my-extension.steadybit-extension.svc.cluster.local:8080/actions` | | `STEADYBIT_AGENT_type_EXTENSIONS_n_METHOD` | | Optional HTTP method to use. Default: `GET` | | `STEADYBIT_AGENT_type_EXTENSIONS_n_BASIC_USERNAME` | | Optional basic authentication username to use within HTTP requests. | | `STEADYBIT_AGENT_type_EXTENSIONS_n_BASIC_PASSWORD` | | Optional basic authentication password to use within HTTP requests. | **Example:** To register, e.g., two ACTION extensions, where the second one requires basic authentication, you use * `STEADYBIT_AGENT_ACTIONS_EXTENSIONS_0_URL`, * `STEADYBIT_AGENT_ACTIONS_EXTENSIONS_1_URL`, * `STEADYBIT_AGENT_ACTIONS_EXTENSIONS_1_BASIC_USERNAME` and * `STEADYBIT_AGENT_ACTIONS_EXTENSIONS_1_BASIC_PASSWORD`. {% endtab %} {% tab title="without specifying a type" %} You can also register extension without specifying the type, e.g., `STEADYBIT_AGENT_EXTENSIONS_REGISTRATIONS_0_URL`. In this case, the agent will try all known Types. |

Environment Variable
(n refers to the index of the extension's instance)

| Required | Description | | ----------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------- | | `STEADYBIT_AGENT_EXTENSIONS_REGISTRATIONS_n_URL` | yes | Fully-qualified URL of the endpoint, e.g., `http://my-extension.steadybit-extension.svc.cluster.local:8080/actions` | | `STEADYBIT_AGENT_EXTENSIONS_REGISTRATIONS_n_METHOD` | | Optional HTTP method to use. Default: `GET` | | `STEADYBIT_AGENT_EXTENSIONS_REGISTRATIONS_n_BASIC_USERNAME` | | Optional basic authentication username to use within HTTP requests. | | `STEADYBIT_AGENT_EXTENSIONS_REGISTRATIONS_n_BASIC_PASSWORD` | | Optional basic authentication password to use within HTTP requests. | | {% endtab %} | | | | {% endtabs %} | | | ### Using Configuration Files Linux packages installations are using this approach by default. The package installer of the extensions is writing configuration files to `/etc/steadybit/extensions.d/extension-*.yaml` which are read by the agent. The content of each file is a YAML document with the following structure: ```yaml url: http://123.45.67.890:8085 types: - ACTION - DISCOVERY ``` ### Using the Agent API You can also register extensions via the [Agent API](/install-and-configure/install-agent/agent-api). Extension registrations are persisted using the configured persistence provider. With each agent restart, the agent will re-register these manual extensions registrations. You can find detailed information about the agent API in the [Agent API](/install-and-configure/install-agent/agent-api) documentation. Example: **POST** `http://localhost:42899/extensions` ```json { "url": "http://123.45.67.890:8085", "types": [ "ACTION", "DISCOVERY" ] } ``` ## Verify registered extensions To check which extensions are registered in the agent, you need to take a look at the agent's logs. Example output: ``` Jan 16 07:50:49 steadybit-agent-0 steadybit-agent INFO Extension-registry change: ADD Extension{source='KubernetesExtensionRegistrator', httpEndpointRef=GET http://steadybit-agent-extension-jmeter.steadybit-agent.svc.cluster.local:8087} Jan 16 07:50:49 steadybit-agent-0 steadybit-agent INFO Extension-registry change: ADD Extension{source='KubernetesExtensionRegistrator', httpEndpointRef=GET http://steadybit-extension-loadtest-9005.steadybit-agent.svc.cluster.local:9005} Jan 16 07:50:49 steadybit-agent-0 steadybit-agent INFO Extension-registry change: ADD Extension{source='KubernetesExtensionRegistrator', httpEndpointRef=GET http://steadybit-agent-extension-gatling.steadybit-agent.svc.cluster.local:8087} Jan 16 07:50:49 steadybit-agent-0 steadybit-agent INFO Extension-registry change: ADD Extension{source='KubernetesExtensionRegistrator', httpEndpointRef=GET http://steadybit-agent-extension-k6.steadybit-agent.svc.cluster.local:8087} ``` # Using Mutual TLS for Extensions For stronger security, the communication between the agent and extensions can be secured using mutual TLS. You need to configure a client certificate at the agent that will be used to communicate with the extensions. And also a server certificate and client CAs at the extension to verify the client certificate. ## Configuring a Client Certificate for Agent {% tabs %} {% tab title="using Helm Chart" %} #### Using a Secret In case the client certificate is stored in a secret, you can directly reference it: `agent.extensions.tls.clientCertificate.fromSecret=some-client-secret` #### Using a Container path If you put the client certificate into the container some other way, use this option to tell the agent where to find it: `agent.extensions.clientCertificate.path=/some/client.crt`\ `agent.extensions.clientCertificate.key.path=/some/client.key` #### Password Protected Private Keys In case the private key is password protected, you can specify it as well: **Using a Value** `agent.extensions.tls.clientCertificate.key.password.value=password123` **Using a Secret** `agent.extensions.tls.clientCertificate.key.password.valueFrom.secretKeyRef.name=some-secret`\ `agent.extensions.tls.clientCertificate.key.password.valueFrom.secretKeyRef.key=some-key` #### Additional Certificate Authorities In case the extension uses custom CAs that are not known to the agent, you can add them as well: **Using a Volume Mount** The volume in this sample refers to a config map, but any other volume config would work as well. `agent.extraCertificates.fromVolume=extra-certs`\ `agent.extraVolumes[0].name=extra-certs`\ `agent.extraVolumes[0].configMap.name=self-signed-ca` **Using Container Path** If you put the extra CAs into the container another way, you can specify the path to it: `agent.extraCertificates.fromVolume=extra-certs` {% endtab %} {% tab title="using Environment Variables" %} #### Client Certificate `STEADYBIT_AGENT_EXTENSIONS_CLIENT_CERT_CHAIN_FILE=/some/client.crt`\ `STEADYBIT_AGENT_EXTENSIONS_CLIENT_CERT_KEY_FILE=/some/client.key` #### Password Protected Private Keys In case the private key is password protected, you can specify it as well: `STEADYBIT_AGENT_EXTENSIONS_CLIENT_CERT_PASSWORD=password123` #### Additional Certificate Authorities In case the extension uses a custom CA that is not known to the agent you can add them as well. `STEADYBIT_AGENT_EXTRA_CERTS_PATH=/some/ca-certs/` {% endtab %} {% endtabs %} If you have successfully configured the client certificate for the agent, the log should contain similar output to this: {% code overflow="wrap" %} ``` 2023-07-29 06:27:05.836 INFO 1 --- [ parallel-1] c.s.k.c.service.ExtensionHttpClient : Configuring extension http client using mutual TLS (keyFile=/opt/steadybit/agent/etc/extensions/client/tls.key, chainFile=/opt/steadybit/agent/etc/extensions/client/tls.crt) ``` {% endcode %} ## Requiring Client Certificates in Extensions {% tabs %} {% tab title="Using Helm Chart" %} The `steadybit-agent` helm charts include all extensions provided by Steadybit using a `extension-*` prefix. This sample is for `extension-container`, the options shown here apply to the other extensions as well. And in case you use the extensions helm chart directly, you need to strip the prefix from the examples. For the extension to require client certificates, you need to configure both the server certificate and allowed client certificates. If you only specify a server certificate, TLS is used, but client certificates are not mandatory. #### Server Certificate **Using a Secret** In case the server certificate is stored in a secret, you can directly reference it: `extension-container.tls.server.certificate.fromSecret=some-server-secret` **Using a Container Path** If you put the server certificate into the container some other way, use this option to tell the agent where to find it: `extension-container.tls.server.certificate.path=/some/server.crt`\ `extension-container.tls.server.certificate.key.path=/some/server.key` #### Client Certificates **Using Secrets** In case the client certificates are stored in secrets, you can directly reference them: `extension-container.tls.client.certificates.fromSecrets[0]=some-secret-1`\ `extension-container.tls.client.certificates.fromSecrets[1]=some-secret-2` **using Container Path** If you put the client certificates into the container some other way, use this option to tell the agent where to find them: `extension-container.tls.client.certificates.paths[0]=/some/client.crt`\ `extension-container.tls.client.certificates.paths[1]=/some/clients/` {% endtab %} {% tab title="Using Environment Variables" %} #### Server Certificate `STEADYBIT_EXTENSION_TLS_SERVER_CERT=/some/server.crt`\ `STEADYBIT_EXTENSION_TLS_SERVER_KEY=/some/server.key` #### Client Certificates `STEADYBIT_EXTENSION_TLS_CLIENT_CAS=/some/client.crt,/some/clients/` {% endtab %} {% endtabs %} If you have successfully configured the server certificate and the allowed client certificates for an extension, the log should contain similar output to this: ``` 2023-07-29 06:26:51.600 INF Starting extension server on port 9093 (TLS: true) ``` # Configuration Options There are plenty of options for the agent that can be controlled using Environment variables. | Environment Variable | Description | | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `STEADYBIT_AGENT_KEY` |

The API key the agent uses
Example: foobar

| | `STEADYBIT_AGENT_REGISTER_URL` |

The baseUrl where the agent registers.
Default:

| | `STEADYBIT_LOG_LEVEL` |

Sets the loglevel for the com.steadybit logger
Default: INFO

| | `LOG_LEVEL` |

Sets the loglevel for the root logger
Default: INFO

| | `STEADYBIT_LOG_FORMAT` |

Sets the log format for the console logger (json or text)
Default: text

| | `STEADYBIT_AGENT_REGISTER_TIMEOUT` |

Timeout for the registration request.
Default: 20s

| | `STEADYBIT_AGENT_REGISTER_INTERVAL` |

The interval how often the agent registers at the platform.
Default: 10s

| | `STEADYBIT_AGENT_EXPERIMENT_TIMEOUT` |

Timeout for the request to connect to an experiment.
Default: 10s

| | `STEADYBIT_AGENT_EXPERIMENT_KEEP_ALIVE_INTERVAL` |

Interval how often a transport keep-alive is sent during an experiment.
Default: 20s

| | `STEADYBIT_AGENT_EXPERIMENT_KEEP_ALIVE_TIMEOUT` |

Timeout for a transport keep-alive. Kept above the websocket pong timeout.
Default: 90s

| | `STEADYBIT_AGENT_WEBSOCKET_PING_INTERVAL` |

Interval of the heartbeat for the websocket connection.
Default: 5s

| | `STEADYBIT_AGENT_EXTENSIONS_TIMEOUT_CONNECT` |

Connect timeout for extension communication
Default: 10s

| | `STEADYBIT_AGENT_EXTENSIONS_TIMEOUT_READ` |

Read timeout for extension communication
Default: 30s

| | `STEADYBIT_AGENT_EXTENSIONS_TIMEOUT_CALL` |

Call timeout for extension communication
Default: 30s

| | `STEADYBIT_AGENT_EXTENSIONS_AUTOREGISTRATION_NAMESPACE` |

Filter extension auto registration in Kubernetes to a given namespace.
Default: null

| | `SERVER_PORT` |

HTTP endpoint port
Default: 42899

| | `STEADYBIT_AGENT_PROXY_HOST` |

Hostname of a proxy to access Steadybit platform

| | `STEADYBIT_AGENT_PROXY_PORT` |

Port of a proxy to access Steadybit platform

| | `STEADYBIT_AGENT_PROXY_PROTOCOL` |

Protocol of a proxy to access Steadybit platform
Valid values: HTTP, SOCKS4, SOCKS5
Default: http

| | `STEADYBIT_AGENT_PROXY_USER` |

Username of a proxy to access Steadybit platform

| | `STEADYBIT_AGENT_PROXY_PASSWORD` |

Password of a proxy to access Steadybit platform

| | `STEADYBIT_AGENT_IDENTIFIER` |

The identifier which will be used to register the agent at the platform
Default: The agent will use the hostname as identifier

| | `STEADYBIT_AGENT_AUTH_PROVIDER` |

The auth mechanism to use - OAUTH2 or AGENT-KEY

Default: AGENT-KEY

| | `STEADYBIT_AGENT_AUTH_OAUTH2_CLIENT_ID` | The public identifier of your OAuth 2.0 Client | | `STEADYBIT_AGENT_AUTH_OAUTH2_CLIENT_SECRET` | The client secret | | `STEADYBIT_AGENT_AUTH_OAUTH2_ISSUER_URI` | The issuer uri of the identity provider | | `STEADYBIT_AGENT_AUTH_OAUTH2_TOKEN_URI` | The token uri of the identity provider to directly specify the URL to retrieve the access tokens from | | `STEADYBIT_AGENT_AUTH_OAUTH2_AUDIENCE` | Optional - Some provider needs the audience parameter to authenticate the client. | | `STEADYBIT_AGENT_AUTH_OAUTH2_AUTHORIZATION_GRAN_TYPE` |

The grant type to use either client\_credentials or password.
Default: client\_credentials

| | `STEADYBIT_AGENT_AUTH_OAUTH_USERNAME` | The username to use for the username flow | | `STEADYBIT_AGENT_AUTH_OAUTH_PASSWORD` | The username to use when using the password flow | | `STEADYBIT_AGENT_AUTH_OAUTH_CLIENT_CERT_CHAIN_FILE` | Optional - the x.509 client certificate to use for mutual TLS with the identity provider. | | `STEADYBIT_AGENT_AUTH_OAUTH_CLIENT_CERT_KEY_FILE` | Optional - the PKCS#8 encoded private key to use for mutual TLS with the identity provider | | `STEADYBIT_AGENT_AUTH_OAUTH_CLIENT_CERT_PASSWORD` | Optional - Password for the the PKCS#8 encoded private key to use for mutual TLS with the identity provider | | `RESILIENCE4J.RETRY_INSTANCES_HTTPDISCOVERY_MAXATTEMPTS` | Optional - Resilience4j: The maximum number of attempts (including the initial call as the first attempt) for DiscoveryKit resources | | `RESILIENCE4J.RETRY_INSTANCES_HTTPDISCOVERY_WAITDURATION` | Optional - Resilience4j: A fixed wait duration between retry attempts for DiscoveryKit resources | | `RESILIENCE4J.RETRY_INSTANCES_HTTPDISCOVERY_ENABLEEXPONENTIALBACKOFF` | Optional - Resilience4j: Enable or disable exponential backoff for DiscoveryKit resources | | `RESILIENCE4J.RETRY_INSTANCES_HTTPDISCOVERY_EXPONENTIALBACKOFFMULTIPLIER` | Optional - Resilience4j: The multiplier for exponential backoff for DiscoveryKit resources | | `STEADYBIT_AGENT_HTTP_DISCOVERY_USE_RETRY` | Optional - Resilience4j: Enable/Disable the retry mechanism. Default is true / enabled | | `STEADYBIT_AGENT_WEBCLIENT_USE_POOL` | Optional - Be able to disable the netty connection pooling by setting environment variable STEADYBIT\_AGENT\_WEBCLIENT\_USE\_POOL=false. Default is true. | # Agent State The agent keeps most of its state in memory. However, some information is persisted to ensure that the agent can recover from a restart. ## Configuration The default state provider is a file-based provider that stores the agent's state by default in the directory `/var/lib/steadybit-agent`. The path can be configured via the `STEADYBIT_AGENT_STATE_PATH` environment variable. The helm chart is, by default, creating a statefulset with a persistent volume claim to store the agent's state. Another option is to use [Redis](https://redis.io/) as a state provider. The following set of environment variables can be used to configure this. | Environment Variable | Required | Description | | ----------------------------------------- | -------- | -------------------------------------------------------------------------------------------- | | `STEADYBIT_AGENT_IDENTIFIER` | yes | The unique agent identifiert, required if `redis` is used. Should be stable across restarts. | | `STEADYBIT_AGENT_STATE_PROVIDER` | yes | Needs to be set to `redis` | | `STEADYBIT_AGENT_STATE_REDIS_HOST` | yes | The redis host | | `STEADYBIT_AGENT_STATE_REDIS_PORT` | no | The redis port, defaults to `6379` | | `STEADYBIT_AGENT_STATE_REDIS_USERNAME` | no | The redis username | | `STEADYBIT_AGENT_STATE_REDIS_PASSWORD` | no | The redis password | | `STEADYBIT_AGENT_STATE_REDIS_DB` | no | The redis database, defaults to `0` | | `STEADYBIT_AGENT_STATE_REDIS_SSL_ENABLED` | no | Should SSL be used | ## Extension Registrations The agent persists extension registrations to ensure they are not lost after a restart. Extension registrations can be maintained via the agent API, see [Extension Registration](/install-and-configure/install-agent/extension-registration). If redis is used as the state provider, existing file-based extension registrations are ignored. Auto discovered extensions from Kubernetes are independently registered and not persisted via the configured persistence provider. They are also not visible or manageable via the agent API. ## Execution State The agent keeps track of an experiment's current execution state. This state is persisted so that actions can be rolled back in case of an agent restart. # Agent API The Agent provides some HTTP endpoints to interact with the agent itself. It's only needed for advanced use cases, like debugging, troubleshooting or for example manual extension registrations. The agent listens on port `42899` by default. ## Swagger UI & Spec The Spec is available at `http(s)://:42899/spec`. Swagger UI is available at `http(s)://:42899/swagger-ui/index.html`. ## Authentication Modifying endpoints require basic authentication. The default username is `_` with the agent-key as password. Credentials might be changed using the properties `steadybit.agent.api.auth.username` and `steadybit.agent.api.auth.password` in the agent configuration. # Install On-Prem Platform {% hint style="info" %} This part of the documentation is only intended in the context of a supported PoC (Proof of Concept) together with the Steadybit team. Please, [book an appointment](https://www.steadybit.com/book-demo) to scope your PoC before continuing to evaluate the on-prem solution. If you just want to try out Steadybit, we recommend you [sign up for our SaaS platform](https://signup.steadybit.com). {% endhint %} This guide helps you with the initial installation and deployment of our Steadybit platform On-Premise. It is explicitly of use for spaces where it is not yet possible to take advantage of our Software as a Service (SaaS) platform and the need to host our platform yourself. After completing this getting started, continue by [setting up Steadybit](/install-and-configure/install-agent). Installing and deploying the Steadybit platform on your servers is simple: * [Step 0 - Check Prerequisites](#step-0-check-prerequisites) * [Step 1 - Get your keys](#step-1-get-your-keys) * [Step 2 - Deploy Platform](#step-2-deploy-platform) If you need help, please [reach out to us](https://www.steadybit.com/contact). ## Step 0 - Check Prerequisites Before continuing, make sure that access to the following URLs via HTTPS (443) is possible from your location of installation: * and (Docker Image) * (Agent Linux Installation) * (Setup Scripts) * (when using Kubernetes and helm) The platform itself exposes the following ports: * Port `8080`: Application port for UI/API/agents * Port `7878`: Websocket port for agents ## Step 1 - Get your keys To install the platform on-premise and connect the agents against it you need an agent key and a valid license.\ Get in touch with us, and we will provide you an on-prem license key and an agent key. The agent key is also used to authenticate against our Docker registry `docker.steadybit.io` hosting the platform images. The username is `_` and the password is the agent key. ## Step 2 - Deploy Platform It is our goal to make the installation as easy as possible for you, that's why we recommend you to use our Helm chart that takes a lot of the work out of it and only requires a few parameters from you. If you are not familiar with Helm and would like to learn more about it, check out [helm's QuickStart](https://helm.sh/docs/intro/quickstart/) {% hint style="info" %} In case you can't use Helm or Kubernetes at all, get in touch with us and we'll find the best solution. The platform can be deployed without Helm and also on plain Docker hosts. When pulling the platform images manually, e.g., on plain Docker hosts or to mirror them into an internal registry, log in to our Docker registry first using your agent key from [Step 1 - Get your keys](#step-1-get-your-keys): ```bash docker login docker.steadybit.io --username _ --password ``` {% endhint %} Please replace the placeholders `replace-with-agent-key` and `replace-with-license-key` with your agent key and license key of [Step 1 - Get your keys](#step-1-get-your-keys): ```bash helm repo add steadybit https://steadybit.github.io/helm-charts helm repo update helm install steadybit-platform \ --create-namespace \ --namespace steadybit-platform \ --set platform.tenant.agentKey= \ --set platform.tenant.license= \ steadybit/steadybit-platform ``` The Helm chart automatically creates the Kubernetes image pull secret for `docker.steadybit.io` from your agent key, so no manual `docker login` is required. To make it convenient for you, we have a default for everything. That's also why we include the necessary Postgres database and set up everything for you automatically. Nevertheless, feel free to adjust parameters after having a look on the helm chart in our public [GitHub repository](https://github.com/steadybit/helm-charts/tree/master/charts/steadybit-platform). > **BE AWARE:** The database is not backed up and if the pod is deleted the data is lost! > > For productive usage, we strongly recommend to setup your own Postgres database and configure the Steadybit platform as described [here](/install-and-configure/install-on-prem-platform/advanced-configuration). ## What's next? Done, the platform is running. The default login for the on-prem platform is * username: `admin` * password: `admin` Now you can set it up by connecting the first agents to it. We'll show you how to do that in our getting started [Set up Platform & Agents](/quick-start/set-up-agents), at step 1. ### Productive Usage As mentioned above, this getting started helped to set up quickly a Steadybit platform. Before using Steadybit * configure your own Postgres database as [described here](/install-and-configure/install-on-prem-platform/advanced-configuration#database-configuration). * we recommend to use your internal authorization services such as LDAP or OIDC provider as [described here](/install-and-configure/install-on-prem-platform/advanced-configuration#ldap-authentication). ### Advanced Configuration More configuration options can be found in [Advanced Configuration](/install-and-configure/install-on-prem-platform/advanced-configuration) ### Troubleshooting If you have any problems, check our [Troubleshooting](/troubleshooting/common-fixes/on-prem-platform) page. Or please [reach out to us](https://www.steadybit.com/contact). # Install on Minikube {% hint style="info" %} This part of the documentation is only intended in the context of a supported PoC (Proof of Concept) together with the Steadybit team. Please, [book an appointment](https://www.steadybit.com/book-demo) to scope your PoC before continuing to evaluate the on-prem solution. If you just want to try out Steadybit, we recommend you [sign up for our SaaS platform](https://signup.steadybit.com). {% endhint %} This guide helps you with the initial installation and deployment of our Steadybit platform On-Premise on a Minikube with NGINX ingress. It is explicitly of use for spaces where you just want to try out the platform and can't take advantage of our Software as a Service (SaaS) platform. After completing this getting started, continue by [setting up Steadybit](/install-and-configure/install-agent). Installing and deploying the Steadybit platform on your machine is simple: * [Step 0 - Check Prerequisites](#step-0-check-prerequisites) * [Step 1 - Get your keys](#step-1-get-your-keys) * [Step 2 - Install Minikube](#step-2-install-minikube) * [Step 3 - Deploy Platform](#step-3-deploy-platform-on-minikube) ## Step 0 - Check Prerequisites Before continuing, make sure that access to the following URLs via HTTPS (443) is possible from your location of installation: * and (Docker Image) * (Agent Linux Installation) * (Setup Scripts) * (when using Kubernetes and helm) The platform itself exposes the following ports: * Port `8080`: Application port for UI/API * Port `7878`: Acceptor port for agents ## Step 1 - Get your keys To install the platform on-premise and connect the agents against it you need an agent key and a valid license.\ Get in touch with us and we will provide you an on-prem license key and an agent key. The agent key is also used to authenticate against our Docker registry `docker.steadybit.io` hosting the platform images. The username is `_` and the password is the agent key. The Helm chart used below automatically creates the Kubernetes image pull secret from your agent key, so no manual `docker login` is required. ## Step 2 - Install Minikube First install Minikube to run on your system: Start the cluster and enable the ingress resource: ```bash minikube start minikube addons enable ingress ``` Verify the ingress is running ```bash kubectl get pods -n ingress-nginx ``` ## Step 3 - Deploy Platform on Minikube Add repo and update contents: ```bash helm repo add steadybit https://steadybit.github.io/helm-charts helm repo update ``` Install the platform using a parameterized helm script to make the platform available on port 80. You also need to replace the placeholders `replace-with-agent-key` and `replace-with-license-key` with your agent key and license key of [Step 1 - Get your keys](#step-1-get-your-keys): ```bash helm upgrade --install steadybit-platform \ --create-namespace \ --namespace steadybit-platform \ --set platform.tenant.agentKey= \ --set platform.tenant.license= \ --set platform.publicWebsocketPort=80 \ --set platform.service.type=ClusterIP \ --set-string "ingress.annotations.nginx\.ingress\.kubernetes\.io/proxy-read-timeout=3600" \ --set-string "ingress.annotations.nginx\.ingress\.kubernetes\.io/proxy-send-timeout=3600" \ steadybit/steadybit-platform ``` Make the ingress accessible ```bash minikube tunnel ``` Now you should be able to access the platform in your browser on # Advanced Agent Authentication ## Agent authentication using agent-key By default, the agents are authenticating using the configured agent-key.\ This key is global and all the same for all agents of your organization. {% hint style="warning" %} In case your agent key got compromised and you need to rotate the agent key, contact our [support](https://steadybit.com/contact). {% endhint %} ## Agent Authentication using [OpenID Connect](https://openid.net/connect/) If you want to have more security controls, you can use OIDC for the agent authentication instead of the agent key. This requires a OIDC identity provider (e.g. Keycloak). For each request to the platform the agent will use an access token which was issued by the identity provider. The platform verifies the token. ### Platform Configuration To instruct the platform to validate all incoming agent requests against a specific identity provider, set `STEADYBIT_AUTH_AGENT_PROVIDER=OAUTH2` and the `STEADYBIT_AUTH_AGENT_OAUTH2_ISSUER_URI` to the issuer URI of your identity provider. All options are described in [Platform Configuration Options](/install-and-configure/install-on-prem-platform/advanced-configuration#openid-connect-authentication). ### Agent Configuration To tell the agent to use OIDC, set `STEADYBIT_AGENT_AUTH_PROVIDER=OAUTH2`, `STEADYBIT_AGENT_AUTH_OAUTH2_ISSUER_URI` to the issuer URI of your identity provider and the `STEADYBIT_AGENT_AUTH_OAUTH2_CLIENT_ID` to the client id to use.\ Alternatively instead of using the issuer URI you can directly configure the `STEADYBIT_AGENT_AUTH_OAUTH2_TOKEN_URI` to tell the agents where to get the access tokens from, this won't try to read the OIDC discovery endpoint. All options are described in [Agent Configuration Options](/install-and-configure/install-agent/advanced-configuration). #### Using client credentials flow To use the client credentials flow you need to set the `STEADYBIT_AGENT_AUTH_OAUTH2_CLIENT_SECRET` to the client secret. #### Using password flow (with credentials or mutual TLS) For using username and password set `STEADYBIT_AGENT_AUTH_OAUTH2_AUTHORIZATION_GRANT_TYPE=password` and provide a username and password with `STEADYBIT_AGENT_AUTH_OAUTH2_USERNAME` and `STEADYBIT_AGENT_AUTH_OAUTH2_PASSWORD`. For using mutual TLS specify PEM-files containing a X.509 certificate and a PKCS#8 private key by setting `STEADYBIT_AGENT_AUTH_OAUTH2_CLIENT_CERT_KEY_FILE` and `STEADYBIT_AGENT_AUTH_OAUTH2_CLIENT_CERT_CHAIN_FILE`. In case the key is encrypted you can specify the password using `STEADYBIT_AGENT_AUTH_OAUTH2_CLIENT_CERT_PASSWORD`.\ The agent will re-load the certificates when the file changes automatically, but it might take up to ten seconds. # Configuration Options ### Machine Requirements The machine you are installing Steadybit onto, must have **at least** 4 CPUs and 8 GB available memory. | Environment Variable | Required | Description | | ------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------ | | `JVM_MAX_RAM_PERCENTAGE` | |

Define the MaxRAMPercentage of the platform JVM
Default: 75.0

| ### Debug Docker Images The platform container image doesn't contain a shell by default. However, in case you need to exec into the container using a shell for debugging purposes, we provide an additional debug variant with the `platform-debug` tag. ### Database Configuration Steadybit requires a PostgreSQL 15 database. The platform relies heavily on PostgreSQL-specific features and cannot run on other RDBMS. **Note**: Ensure the database and platform clocks are in sync, as time-based operations might be impacted otherwise. | Environment Variable | Required | Description | | ---------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------- | | `SPRING_DATASOURCE_URL` | yes |

JDBC Url for the database connection
Example: jdbc:postgresql://postgres:5432/steadybitdb

| | `SPRING_DATASOURCE_USERNAME` | yes |

Database Username
Example: postgres

| | `SPRING_DATASOURCE_PASSWORD` | yes |

Database Password
Example: postgres

| | `STEADYBIT_DB_WEB_ENABLED` | |

Enable Http Endpoint for Database export
Default: true

| #### Database Maintenance The platform performs periodic database maintenance (VACUUM, ANALYZE) on configurable tables to optimize performance. | Environment Variable | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `STEADYBIT_DB_MAINTENANCE_ENABLED` | |

Enable automatic database maintenance
Default: true

| | `STEADYBIT_DB_MAINTENANCE_CRON` | |

Cron expression for when maintenance runs
Default: 0 0 0 ? \* SAT \* (midnight on Saturdays)

| | `STEADYBIT_DB_MAINTENANCE_TABLES` | |

Comma-separated list of tables to maintain
Default: target,target\_stats,target\_submission\_tracking,audit\_log,experiment\_execution,execution\_log\_event,execution\_metric\_event,execution\_artifact,execution\_spans,license\_usage,file

| #### Database Permissions The database user requires the following permissions: | Command | Description | | ---------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `GRANT CONNECT ON DATABASE TO ;` | Allow the application user (e.g. `postgres`) to connect | | `GRANT CREATE ON DATABASE TO ;` | Allow the application user (e.g. `postgres`) to create schemas | | `GRANT CREATE ON SCHEMA public TO ;` | Extensions (`pg_trgm`, `uuid-ossp`, `btree-gin`, `pg_stat_statements`) are created by the platform | | `GRANT USAGE ON SCHEMA public TO ;` | Access to PostgreSQL extension functions | #### Target Index Advisor Steadybit runs a nightly job that inspects which target attribute keys are actually used as target enrichment rule selectors, samples their real-world cardinality, and suggests — or, by default, creates — dedicated PostgreSQL indexes for the keys that make enrichment-rule matching fast. Index removal is never performed automatically in any mode; a no-longer-justified index is always a logged suggestion for an operator to act on by hand. Suggestions and the DDL to run manually are logged under `[TargetIndexAdvisor]`. | Environment Variable | Required | Description | | --------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `STEADYBIT_TARGETS_INDEX_ADVISOR_MODE` | |

Controls what the nightly pass does. OFF — does nothing at all (no classification, no sampling queries, no catalog reads). REPORT\_ONLY — classifies keys and logs suggestions, but takes no DB action. AUTO\_CREATE — additionally creates a missing index the first night a key classifies as needle-like.
Default: AUTO\_CREATE

| | `STEADYBIT_TARGETS_INDEX_ADVISOR_CRON` | |

Cron expression for when the nightly per-tenant pass runs.
Default: 0 15 2 ? \* \* \* (02:15)

| | `STEADYBIT_TARGETS_INDEX_ADVISOR_CRON_JITTER` | |

Maximum per-tenant random offset added on top of the cron schedule, so tenants don't all sample the target table at the same instant. Deterministically derived from the tenant key, so it stays stable across restarts.
Default: 20m

| | `STEADYBIT_TARGETS_INDEX_ADVISOR_NEEDLE_ROW_THRESHOLD` | |

A key classifies as needle-like (index it) once the average number of rows matched per distinct value drops to or below this threshold.
Default: 100

| | `STEADYBIT_TARGETS_INDEX_ADVISOR_BULK_ROW_THRESHOLD` | |

A key classifies as bulk-like (safe to remove an existing index) once the average number of rows matched per distinct value reaches or exceeds this threshold. Deliberately a 10x gap from the needle threshold so ordinary night-to-night sampling noise can't flip a key between the two.
Default: 1000

| | `STEADYBIT_TARGETS_INDEX_ADVISOR_POPULATION_FLOOR` | |

Skip classifying/indexing a key if its target type's total population is smaller than this — a full scan of a small type is already fast regardless of selectivity.
Default: 5000

| | `STEADYBIT_TARGETS_INDEX_ADVISOR_SAMPLE_CONFIDENCE_FLOOR` | |

Below this many matched rows, the adaptive sampling estimate isn't trusted and the classifier falls back to a full scan for that key.
Default: 2000

| #### RDS Machine Requirements The workload is bound by the database CPU on peaks. If you have \~100k targets simultaneously in the platform, we recommend a burstable instance with four vCPU (e.g., db.t4g.xlarge). Regarding disk size, 20 GB should be enough capacity for the start (as extending on AWS should not be a problem). If you choose a smaller instance for cost savings, the target ingestion will be slower, so it will take a bit longer until the target data in the platform is consistent. #### AWS RDS IAM Authentication Steadybit supports AWS RDS IAM authentication using the [AWS Advanced JDBC Wrapper](https://github.com/aws/aws-advanced-jdbc-wrapper). This allows you to authenticate to your RDS database using IAM credentials instead of a database password. **Prerequisites** Before configuring Steadybit, you need to set up IAM database authentication on your RDS instance. Follow the [AWS documentation on IAM database authentication for MariaDB, MySQL, and PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html) to: 1. Enable IAM authentication on your RDS instance 2. Create a database user that uses IAM authentication 3. Configure the required IAM policy with `rds-db:connect` permission **Configuration** To enable IAM authentication, configure the following environment variables: | Environment Variable | Required | Description | | ---------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SPRING_DATASOURCE_URL` | yes |

JDBC URL using the AWS wrapper prefix
Example: jdbc:aws-wrapper:postgresql://your-rds-endpoint:5432/steadybitdb

| | `SPRING_DATASOURCE_USERNAME` | yes |

The IAM database user
Example: steadybit\_iam

| | `spring.datasource.hikari.data-source-properties.wrapperPlugins` | yes |

Comma-separated list of AWS JDBC Wrapper plugins. Include iam for IAM authentication.
Example: iam,initialConnection,auroraConnectionTracker,failover2,efm2

| | `spring.datasource.hikari.data-source-properties.wrapperDialect` | | The database dialect for the AWS JDBC Wrapper. Required for some plugins to work correctly, e.g. when using aurora. | **Note:** When using IAM authentication, you do not need to set `SPRING_DATASOURCE_PASSWORD` as the AWS SDK will generate authentication tokens automatically using the configured IAM credentials. **Example Configuration** ```yaml env: - name: SPRING_DATASOURCE_URL value: "jdbc:aws-wrapper:postgresql://my-rds-instance.abc123.us-east-1.rds.amazonaws.com:5432/steadybitdb" - name: SPRING_DATASOURCE_USERNAME value: "steadybit_iam" - name: spring.datasource.hikari.data-source-properties.wrapperPlugins value: "iam,efm2" ``` **Available Wrapper Plugins** You can also configure additional wrapper plugins for aurora (e.g. `initialConnection`, `auroraConnectionTracker`) or clusters (`failover2`). For a complete list of available plugins and their configuration options, see the [AWS Advanced JDBC Wrapper documentation](https://github.com/aws/aws-advanced-jdbc-wrapper/blob/main/docs/using-the-jdbc-driver/UsingTheJdbcDriver.md#list-of-available-plugins). ### Message Broker Configuration A Redis message broker is required to run the platform with multiple instances. | Environment Variable | Required | Description | | ----------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------ | | `SPRING_REDIS_HOST` | yes |

Redis server host
Example: redis

| | `SPRING_REDIS_PORT` | |

Redis server port
Default: 6379

| | `SPRING_REDIS_USERNAME` | | Redis Username | | `SPRING_REDIS_PASSWORD` | | Redis Password | | platform <= 1.0.96 `SPRING_REDIS_SSL`, platform > 1.0.96 `SPRING_REDIS_SSL_ENABLED` | |

Wether to enable ssl support.
Default: false

| | `SPRING_REDIS_CLIENT_NAME` | | Client name to be set on connections with CLIENT SETNAME. | ### Tenant Configuration | Environment Variable | Required | Description | | --------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `STEADYBIT_TENANT_AGENTKEY` | yes |

Agent key for the tenant assigned to you. Treat it as sensitive information.
Example: foobar

| | `STEADYBIT_TENANT_LICENSE` | yes |

License key for the tenant assigned to you. Treat it as sensitive information.
Example: secret-license

| ### Web Configuration | Environment Variable | Required | Description | | -------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `STEADYBIT_WEB_PUBLIC_URL` | | URL to point to your Steadybit installation. Use this if your platform runs behind a reverse proxy doing path rewriting. Also, it is used for the links in notifications. | | `STEADYBIT_WEB_PUBLIC_EXPERIMENT_PORT` | | By default, the Websocket connections are advertised to the agents on port 7878. If the public port differs (e.g. because of a proxy) use this property to advertise a different port. | | `STEADYBIT_WEB_PUBLIC_EXPERIMENT_URL` | | By default, the Websocket connections are advertised using the same URL name as the agents register to. You can override the advertised URL if you run a separate load balancer for the websockets. | ### Log Configuration | Environment Variable | Required | Description | | -------------------- | -------- | ---------------------------------------------------------------------------------------------- | | `LOGGING_FORMAT` | | By default, Steadybit uses `text` format. Set this to `json` to switch the log format to JSON. | ### Static-Authentication You can use a static username/password to authenticate as an admin user. | Environment Variable | Required | Description | | ---------------------------------- | -------- | ------------------------------------------------------------------------------------------------------- | | `STEADYBIT_AUTH_PROVIDER` | yes |

Use STATIC for static authentication
Example: LDAP

| | `STEADYBIT_AUTH_STATIC_0_USERNAME` | yes |

Username
Example: admin

| | `STEADYBIT_AUTH_STATIC_0_PASSWORD` | yes |

Password
Example: {noop}admin

| ### LDAP-Authentication You can use an LDAP server for [authentication and synchronization](/install-and-configure/install-on-prem-platform/ldap-integration). | Environment Variable | Required | Description | | ---------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `STEADYBIT_AUTH_PROVIDER` | yes |

Use LDAP for LDAP-Authentication
Example: LDAP

| | `STEADYBIT_AUTH_LDAP_URL` | yes |

LDAP-Server URL
Example: ldap\://openldap:389/dc=steadybit,dc=com

| | `STEADYBIT_AUTH_LDAP_MANAGER_DN` | | Username (DN) of the "manager" user identity is used to authenticate to an LDAP server. If omitted, anonymous access will be used. **Example:** `uid=admin,ou=system` | | `STEADYBIT_AUTH_LDAP_MANAGER_PASSWORD` | | The password for the manager DN. This is required if the manager-dn is specified. | | `STEADYBIT_AUTH_LDAP_USER_SEARCH_BASE` | | the base DN for searching users in the LDAP directory | | `STEADYBIT_AUTH_LDAP_USER_SEARCH_FILTER` | |

the filter for searching users in the LDAP directory
Default: (&(objectClass=inetOrgPerson)(uid={0}))

| | `STEADYBIT_AUTH_LDAP_METHOD` | |

The method to authenticate the user. Either bind or password-compare.
Default: bind

| | `STEADYBIT_AUTH_LDAP_USERNAME_ATTRIBUTE` | |

Name of the attribute that contains the username
Default: uid

| | `STEADYBIT_AUTH_LDAP_PASSWORD_ATTRIBUTE` | |

Name of the attribute that contains the user password, used if using password-compare
Default: userPassword

| | `STEADYBIT_AUTH_LDAP_SYNC_ADMIN_GROUP_DN` | yes |

The DN for the groupOfNames/groupOfUniqueNames for the Admin users
Example: cn=steadybit\_admin,ou=groups,dc=steadybit,dc=com

| | `STEADYBIT_AUTH_LDAP_SYNC_USER_GROUP_DN` | yes |

The DN for the groupOfNames/groupOfUniqueNames for the User users
Example: cn=steadybit\_user,ou=groups,dc=steadybit,dc=com

| | `STEADYBIT_AUTH_LDAP_SYNC_TEAM_SEARCH_FILTER` | |

The filter for the groupOfNames/groupOfUniqueNames for the teams
Example: ou=teams,ou=groups,dc=steadybit,dc=com

| | `STEADYBIT_AUTH_LDAP_SYNC_TEAM_KEY_ATTRIBUTE` | |

The attribute to use as Team key
Example: cn=steadybit\_admin,ou=groups,dc=steadybit,dc=com

| | `STEADYBIT_AUTH_LDAP_SYNC_TEAM_NAME_ATTRIBUTE` | |

The attribute to use as Team name
Example: cn=steadybit\_admin,ou=groups,dc=steadybit,dc=com

| | `STEADYBIT_AUTH_LDAP_SYNC_CRON` | |

Cron Expression which defines the periods for the LDAP synchronization
Default: 0 0 \_/2 ? \_ \* \*

| ### OpenID Connect Authentication You can use OpenID Connect compatible authentication provider for [authentication and synchronization](/install-and-configure/install-on-prem-platform/oidc-integration). | Environment Variable | Required | Description | | ------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `STEADYBIT_AUTH_PROVIDER` | yes |

Use OAUTH2 for OIDC-Authentication
Example: OAUTH2

| | `STEADYBIT_AUTH_OAUTH2_ISSUER_URI` | yes |

URI for the OpenID Connect discovery endpoint
Example:

| | `STEADYBIT_AUTH_OAUTH2_CLIENT_ID` | yes |

The client ID to use for the OIDC registration
Example: steadybit

| | `STEADYBIT_AUTH_OAUTH2_CLIENT_SECRET` | yes |

The client secret to use for the OIDC registration
Example: ijhdfpjdf80wiphubfqwd113342r

| | `STEADYBIT_AUTH_OAUTH2_SCOPE` | |

The OAUTH2 scope to use for the OIDC registration
Default: openid,profile,email

| | `STEADYBIT_AUTH_OAUTH2_USER_NAME_ATTRIBUTE` | |

Name of the OidcIdToken attribute that will be used to identify the user
Default: sub

| | `STEADYBIT_AUTH_OAUTH2_FULL_NAME_ATTRIBUTE` | |

Name of the OidcIdToken attribute that will be used to pick the full name of the user
Default: name

| | `STEADYBIT_AUTH_OAUTH2_CLAIMS_TEAM_NAME_ATTRIBUTE_NAME` | |

Name of the OidcIdToken claims attribute that will be used to pick up the assigned team names from. Steadybit automatically creates the specified teams in the platform and assigns the user to them.
Default: groups
Example value in OIDC provider for single team: team1
Example value in OIDC provider for multiple teams: \["team1","team2"]

| | `STEADYBIT_AUTH_OAUTH2_HOSTED_DOMAIN` | |

Restrict the login to users with a specific email domain. If set, only users with an email address from this domain will be allowed to log in. Can be used with Google Workspace OIDC.
Example: example.com

| ### Using SSL/TLS Encryption SSL can be configured by setting the various `SERVER_SSL_*` properties and requires a java keystore (typically PKCS12). | Environment Variable | Required | Description | | ------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SERVER_PORT` | |

Port to use
Default: 8080

| | `SERVER_SSL_KEY_STORE` | |

Path to the key store that holds the SSL certificate (typically a .jks or .p12 file).
Example: file:/keystores/steadybit.p12

| | `SERVER_SSL_KEY_STORE_TYPE` | |

Type of the keystore
Example: PKCS12

| | `SERVER_SSL_KEY_STORE_PASSWORD` | | Password used to access the key store | | `SERVER_SSL_KEY_ALIAS` | | Alias that identifies the key in the keystore to be used | | `SERVER_SSL_KEY_PASSWORD` | | Password used to access the key in the key store. | ### Audit-Log Export Audit logs can be exported to an AWS S3 Bucket. | Environment Variable | Required | Description | | ------------------------------------------ | -------- | ------------------------------------------------------------------ | | `STEADYBIT_AUDITLOG_EXPORT_CRON` | | Cron Expression which defines the periods for the audit log export | | `STEADYBIT_AUDITLOG_EXPORT_S3_BUCKET_NAME` | | AWS S3 Bucket Name | | `STEADYBIT_AUDITLOG_EXPORT_SUBFOLDER` | | Subfolder in the S3 Bucket | | `STEADYBIT_AUDITLOG_EXPORT_REGION` | | AWS Region | ### Advanced Agent Authentication OpenID Connect can be used to [authenticate the agents to the platform](/install-and-configure/install-on-prem-platform/advanced-agent-authentication). | Environment Variable | Required | Description | | ---------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------- | | `STEADYBIT_AUTH_AGENT_PROVIDER` | |

Set to OAUTH2 to use the OIDC.
Default: AGENT\_KEY

| | `STEADYBIT_AUTH_AGENT_OAUTH2_ISSUER_URI` | yes | The issuer URI of your identity provider | ### Proxy Settings Steadybit will use these proxy settings if the platform needs to connect to other services (for example, your OIDC identity provider). | Environment Variable | Required | Description | | -------------------------- | -------- | ---------------------------------------------------------------------------- | | `STEADYBIT_PROXY_HOST` | | Hostname of your proxy | | `STEADYBIT_PROXY_PORT` | | Port of your proxy | | `STEADYBIT_PROXY_PROTOCOL` | |

Protocol of your proxy
Default: http

| | `STEADYBIT_PROXY_USER` | | Username of your proxy | | `STEADYBIT_PROXY_PASSWORD` | | Password of your proxy | ### Experiment Execution | Environment Variable | Description | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `STEADYBIT_EXPERIMENT_EXECUTION_PARALLEL_EXECUTION_CHECK_SCOPE` | Steadybit will show a warning if experiments are running for the same `TENANT` (default), `TEAM`, or `ENVIRONMENT` | | `STEADYBIT_EXPERIMENT_EXECUTION_MAXIMUM_DURATION` | The maximum total duration of an experiment. Default is `12h` | | `STEADYBIT_EXPERIMENT_EXECUTION_EXPERIMENT_TIMEOUT` | The time after which the experiment execution should time out after the estimated duration is reached. Default is `15m` | | `STEADYBIT_EXPERIMENT_EXECUTION_PREPARATION_TIMEOUT` | The time after which the experiment execution should time out if not all agents are prepared. Default is `60s` | | `STEADYBIT_EXPERIMENT_EXECUTION_STEP_START_TIMEOUT` | The time after which an experiment step should time out if not started after triggering. Default is `180s` | ### Data Retention Settings All retention settings are defined via * `STEADYBIT_X_RETENTION_PERIOD` a duration string, sequence of decimal numbers, and a unit suffix, see [Spring conversion expressions](https://docs.spring.io/spring-boot/reference/features/external-config.html#features.external-config.typesafe-configuration-properties.conversion.durations). * `STEADYBIT_X_RETENTION_CRON` cron string for Quartz Note that `X` links to a specific domain, see below (e.g., targets stats via `TARGETS_STATS`). | Environment Variable | Description | Database Table | Default Value | | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------- | | `STEADYBIT_EXTERNAL_VENDOR_AMPLITUDE_LOCAL_STORAGE_RETENTION_PERIOD` |

Maximum age of Amplitude analytics events.
Only applicable if Amplitude is enabled and the mode is set to LOCAL\_STORAGE.

| `amplitude_local_event` | `365d` | | `STEADYBIT_EXTERNAL_VENDOR_AMPLITUDE_LOCAL_STORAGE_RETENTION_CRON` | Cron String for the cleanup job of Amplitude analytics events. | `amplitude_local_event` |

0 15 5 1/1 \* ? \*
(every day at 5:15)

| | `STEADYBIT_AUDITLOG_RETENTION_PERIOD` | Maximum age of audit logs. | `audit_log` | `90d` | | `STEADYBIT_AUDITLOG_RETENTION_CRON` | Cron String for the cleanup job of audit logs. | `audit_log` |

0 20 5 1/1 \* ? \*
(every day at 5:20)

| | `STEADYBIT_EXPERIMENT_EXECUTION_ARTIFACT_RETENTION_PERIOD` | Maximum age of experiment execution artifacts. | `execution_artifact` | Keep until the execution is deleted | | `STEADYBIT_EXPERIMENT_EXECUTION_ARTIFACT_RETENTION_CRON` | Cron String for the cleanup job of experiment execution artifacts. | `execution_artifact` | | | `STEADYBIT_EXPERIMENT_EXECUTION_LOG_RETENTION_PERIOD` | Maximum age of experiment execution logs. | `execution_log_event` | Keep until the execution is deleted | | `STEADYBIT_EXPERIMENT_EXECUTION_LOG_RETENTION_CRON` | Cron String for the cleanup job of experiment execution logs. | `execution_log_event` | | | `STEADYBIT_EXPERIMENT_EXECUTION_METRIC_RETENTION_PERIOD` | Maximum age of experiment execution metrics. | `execution_metric_event` | Keep until the execution is deleted | | `STEADYBIT_EXPERIMENT_EXECUTION_METRIC_RETENTION_CRON` | Cron String for the cleanup job of experiment execution metrics. | `execution_metric_event` | | | `STEADYBIT_EXPERIMENT_EXECUTION_SPAN_RETENTION_PERIOD` | Maximum age of experiment execution spans. | `execution_spans` | `28d` | | `STEADYBIT_EXPERIMENT_EXECUTION_SPAN_RETENTION_CRON` | Cron String for the cleanup job of experiment execution spans. | `execution_spans` |

0 25 5 1/1 \* ? \*
(every day at 5:25)

| | `STEADYBIT_EXPERIMENT_EXECUTION_RETENTION_PERIOD` | Maximum age of experiment executions. | `experiment_execution` and more | No cleanup by default | | `STEADYBIT_EXPERIMENT_EXECUTION_RETENTION_CRON` | Cron String for the cleanup job of experiment executions. | `experiment_execution` and more | | | `STEADYBIT_METRIC_RETENTION_PERIOD` | Maximum age of metrics. | `metric` | `365d` | | `STEADYBIT_METRIC_RETENTION_CRON` | Cron String for the cleanup job of metrics. | `metric` |

0 35 5 1/1 \* ? \*
(every day at 5:35)

| | `STEADYBIT_TARGETS_STATS_RETENTION_PERIOD` | Maximum age of target stats. | `target_stats` | `7d` | | `STEADYBIT_TARGETS_STATS_RETENTION_CRON` | Cron String for the cleanup job of target stats. | `target_stats` |

0 30 5 1/1 \* ? \*
(every day at 5:30)

| | `STEADYBIT_STEADYBUDDY_RETENTION_CONVERSATIONS_PERIOD` | Maximum age of [SteadyBuddy](#steadybuddy) conversations and their chat memory, state, and history. | `ai_conversation` and more | `30d` | | `STEADYBIT_STEADYBUDDY_RETENTION_CONVERSATIONS_CRON` | Cron String for the cleanup job of SteadyBuddy conversations. | `ai_conversation` and more |

0 35 5 1/1 \* ? \*
(every day at 5:35)

| | `STEADYBIT_STEADYBUDDY_RETENTION_TRACES_PERIOD` | Maximum age of persisted SteadyBuddy LLM debug traces. | `ai_trace` | `7d` | | `STEADYBIT_STEADYBUDDY_RETENTION_TRACES_CRON` | Cron String for the cleanup job of SteadyBuddy traces. | `ai_trace` |

0 40 5 1/1 \* ? \*
(every day at 5:40)

| ### SteadyBuddy [SteadyBuddy](/use-steadybit/steadybuddy) is Steadybit's AI-powered assistant for designing, running, and analyzing experiments via natural language. On-prem, it is **disabled until you configure a model provider** and requires the AI capability to be part of your license. #### Core Settings | Environment Variable | Required | Description | Default Value | | --------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | | `STEADYBIT_STEADYBUDDY_PROVIDER` | yes | Enables SteadyBuddy and selects the model provider. See [AI Provider Settings](#ai-provider-settings) for available options. When unset, the entire AI feature is disabled. | none — AI disabled | | `STEADYBIT_STEADYBUDDY_REQUEST_TIMEOUT` | | Per-request timeout for LLM calls. | `120s` | | `STEADYBIT_STEADYBUDDY_TRACING_ENABLED` | | When `true`, every chat turn and suggestion records a debug trace in the database. The trace exposes the full assembled context (system prompt, action catalog, environments). For support requests, the Steadybit team may ask you to share the trace to improve the provided answers. See [Data Retention Settings](#data-retention-settings) for how long traces are kept. | `false` | #### AI Provider Settings **Amazon Bedrock** | Environment Variable | Required | Description | Default | | -------------------------------------------------- | -------- | ---------------------------------------------- | --------------------------------------------- | | `STEADYBIT_STEADYBUDDY_PROVIDER` | yes | Set to `BEDROCK` to use Amazon Bedrock. | none — AI disabled | | `STEADYBIT_STEADYBUDDY_BEDROCK_REGION` | | AWS region for Bedrock. | `eu-central-1` | | `STEADYBIT_STEADYBUDDY_BEDROCK_REGULAR_MODEL_NAME` | | Inference profile id for the capable model. | `eu.anthropic.claude-sonnet-4-6` | | `STEADYBIT_STEADYBUDDY_BEDROCK_REGULAR_MAX_TOKENS` | | Max output tokens for the capable model. | `4096` | | `STEADYBIT_STEADYBUDDY_BEDROCK_CHEAP_MODEL_NAME` | | Inference profile id for the classifier model. | `eu.anthropic.claude-haiku-4-5-20251001-v1:0` | | `STEADYBIT_STEADYBUDDY_BEDROCK_CHEAP_MAX_TOKENS` | | Max output tokens for the classifier model. | `1024` | **Ollama** | Environment Variable | Required | Description | Default | | ----------------------------------------- | -------- | ------------------------------ | ------------------------ | | `STEADYBIT_STEADYBUDDY_PROVIDER` | yes | Set to `OLLAMA` to use Ollama. | none — AI disabled | | `STEADYBIT_STEADYBUDDY_OLLAMA_BASE_URL` | | Base URL of the Ollama server. | `http://localhost:11434` | | `STEADYBIT_STEADYBUDDY_OLLAMA_MODEL_NAME` | | Model name to use. | `qwen3.6` | **Anthropic** | Environment Variable | Required | Description | Default | | ---------------------------------------------------- | -------- | --------------------------------------------------------- | ------------------- | | `STEADYBIT_STEADYBUDDY_PROVIDER` | yes | Set to `ANTHROPIC` to use Anthropic. | none — AI disabled | | `STEADYBIT_STEADYBUDDY_ANTHROPIC_API_KEY` | yes | API key for Anthropic. Treat it as sensitive information. | — | | `STEADYBIT_STEADYBUDDY_ANTHROPIC_REGULAR_MODEL_NAME` | | Capable model. | `claude-sonnet-4-6` | | `STEADYBIT_STEADYBUDDY_ANTHROPIC_REGULAR_MAX_TOKENS` | | Max output tokens for the capable model. | `8192` | | `STEADYBIT_STEADYBUDDY_ANTHROPIC_CHEAP_MODEL_NAME` | | Classifier model. | `claude-haiku-4-5` | | `STEADYBIT_STEADYBUDDY_ANTHROPIC_CHEAP_MAX_TOKENS` | | Max output tokens for the classifier model. | `1024` | **OpenAI** | Environment Variable | Required | Description | Default | | ------------------------------------------------- | -------- | ------------------------------------------------------ | ------------------ | | `STEADYBIT_STEADYBUDDY_PROVIDER` | yes | Set to `OPENAI` to use OpenAI. | none — AI disabled | | `STEADYBIT_STEADYBUDDY_OPENAI_API_KEY` | yes | API key for OpenAI. Treat it as sensitive information. | — | | `STEADYBIT_STEADYBUDDY_OPENAI_REGULAR_MODEL_NAME` | | Capable model. | `gpt-5.4` | | `STEADYBIT_STEADYBUDDY_OPENAI_REGULAR_MAX_TOKENS` | | Max output tokens for the capable model. | `4096` | | `STEADYBIT_STEADYBUDDY_OPENAI_CHEAP_MODEL_NAME` | | Classifier model. | `gpt-5.4-mini` | | `STEADYBIT_STEADYBUDDY_OPENAI_CHEAP_MAX_TOKENS` | | Max output tokens for the classifier model. | `1024` | ### Endpoint Rate Limits Rate limits protect the UI, API, and Agent endpoints of the Steadybit platform. They can be enabled or disabled by setting the environment variable `steadybit.ratelimit.enabled` to `true` or `false`. Rate limits restrict the number of processable requests in a given timeframe. Every request reduces this capacity and will fail if none is left. After a given time, the capacity is refilled, and requests can be processed again. All rate limits apply to the associated tenant, whereas some endpoints are additionally restricted by a qualifier, specifying the user or agent issuing the request. | Name | Description |

Tenant
(capacity,refill token, refill rate)

|

Qualifier
(capacity,refill token, refill rate)

| | ------------------ | -------------------------------------------------------------------------------- | ----------------------------------------------------- | -------------------------------------------------------- | | API General | All API requests not mentioned below | 100/100/60s | - | | API Experiment | Experiment execution endpoints | 10/10/60s | - | | API Security | Killswitch, cancel experiment, remove schedule, remove team member, delete token | unlimited | - | | UI General | All UI requests not mentioned below |

1000/1000/1s tenant,
500 user

| 500/100/10s | | UI Security | Killswitch, cancel experiment, remove schedule, remove team member, delete token | unlimited | - | | Agent Registration | Agent registration requests | 200/200/10s | - | | Agent Definition | Extension metadata like target types, attributes, enrichment rules or advices | 200/200/10s | 10/10/10s | | Agent Experiment | Experiment execution and metadata like metrics, logs, spans | 1000/1000/1s | 100/100/10s | | Agent Target |

Submitted targets
(based on target count, and not request count)

| 100000/30000/5s | 75000/7500/5s | Environment variables can override the predefined defaults to define stricter or more relaxed restrictions. `name` has to be replaced by the rate limit name in environment variable format (all uppercase, separated by underscore). `capacity` states the initial capacity, `refill-tokens` the amount that should be refilled every `refill-period` in unit `refill-unit`. Furthermore, the tenant and qualifier (user or agent) restrictions can be overridden separately. ```bash steadybit.ratelimit.configurations..perTenant.capacity= steadybit.ratelimit.configurations..perTenant.refill-tokens= steadybit.ratelimit.configurations..perTenant.refill-period= steadybit.ratelimit.configurations..perTenant.refill-unit= steadybit.ratelimit.configurations..perQualifier.capacity= steadybit.ratelimit.configurations..perQualifier.refill-tokens= steadybit.ratelimit.configurations..perQualifier.refill-period= steadybit.ratelimit.configurations..perQualifier.refill-unit= ``` For example, changing the general UI rate limit would look like this: ```bash steadybit.ratelimit.configurations.UI_GENERAL.perTenant.capacity=10000 steadybit.ratelimit.configurations.UI_GENERAL.perTenant.refill-tokens=1000 steadybit.ratelimit.configurations.UI_GENERAL.perTenant.refill-period=5 steadybit.ratelimit.configurations.UI_GENERAL.perTenant.refill-unit=s ``` #### Rate Limit Metrics The Steadybit Platform provides the following rate limit metrics: | Metric | Labels | Value | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | | `ratelimit_tokens_available` | `tenantKey`, `bucketName` (as described above), `qualifier` (username or agent id) | Number of available tokens | | `ratelimit_tokens_total` | `tenantKey`, `bucketName` (as described above), `qualifier` (username or agent id), `status` (`consumed`, `rejected` or `failed`) | Number of requested tokens | #### Hub Connections The platform can connect to multiple hubs. | Environment Variable | Description | Default | | ------------------------------- | ------------------------------------- | ------- | | `STEADYBIT_HUBS_MAX_HUBS_COUNT` | Maximum number of hubs to connect to. | 5 | # Monitoring the Platform {% hint style="info" %} This page applies to **on-premise installations** of the Steadybit platform. If you are using our SaaS at [platform.steadybit.com](https://platform.steadybit.com), monitoring is fully managed by Steadybit. {% endhint %} When you operate the Steadybit platform yourself, observing a small set of metrics is enough to spot the vast majority of issues before they impact your users. This page describes: * the metrics endpoint exposed by the platform, * the **four Golden Signals** to monitor (latency, traffic, errors, saturation), * a downloadable Grafana dashboard — the same one we use internally, * a ready-to-use set of **recommended Prometheus alert rules** with the thresholds we run in production. ## Metrics Endpoint The platform exposes Prometheus-compatible metrics on the management port `9090`: ``` http://:9090/actuator/prometheus ``` Port `9090` is administrative and is not exposed to end users. Scrape it from your Prometheus instance the same way you would any Spring Boot Actuator endpoint. All metric names referenced on this page come from this endpoint. ## The Four Golden Signals ### 1. Latency How long requests take to complete. Two views matter: | Signal | Metric | What it tells you | | ----------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | HTTP request latency | `http_server_requests_seconds_sum` / `http_server_requests_seconds_count` | End-user perceived UI/API responsiveness. | | Message queue lead time | `queue_lead_time` | How far behind the platform is in processing target updates from agents. Spikes here are the earliest symptom of an overloaded platform. | **Recommended thresholds:** | Metric | Warning | Critical | | --------------------------------------------------------------------------------------- | -------- | -------- | | Mean GET response time (2 min window) | > 200 ms | > 500 ms | | Mean POST response time (2 min window) | > 400 ms | > 600 ms | | Mean POST response time, UI endpoints (`/ui/experiments/validate`, `/ui/targets/count`) | > 5 s | > 10 s | | Message queue accumulated lead time | > 10 min | > 30 min | ### 2. Traffic Request volume hitting the platform. Sudden spikes often correlate with too many agents reconnecting or a misbehaving integration. | Metric | Warning | Critical | | ----------------------------------------------- | --------- | ---------- | | `http_server_requests_seconds_count` per second | > 5 req/s | > 10 req/s | These thresholds are conservative; tune them to match your installed scale (number of agents, tenants, targets). ### 3. Errors The percentage of requests returning HTTP 5xx. | Metric | Critical | | ------------------------------------------------------------------------ | -------------------- | | `http_server_requests_seconds_count{status=~"5.."}` / total request rate | > 5 % over 5 minutes | A sustained error rate above 5 % almost always points to a database issue, a broken upstream integration, or a recently failed deployment. ### 4. Saturation How "full" the platform is. The two limits to watch: | Resource | Metric | Warning | Critical | | ------------------------ | -------------------------------------------------------------------------- | ---------------- | ---------------- | | JVM heap memory | `jvm_memory_used_bytes{area="heap"}` / `jvm_memory_max_bytes{area="heap"}` | > 80 % for 5 min | — | | Database connection pool | `hikaricp_connections_active` / `hikaricp_connections_max` | > 60 % for 5 min | > 80 % for 5 min | A connection pool above 80 % is a strong indicator of either long-running transactions on the database or an undersized pool. See the [Database Runbooks](/install-and-configure/install-on-prem-platform/database-runbooks) page for cleanup procedures. ## Grafana Dashboard We publish the same Grafana dashboard we use to operate Steadybit SaaS. It is grouped into four sections — Message Queues, Target Ingestion, Platform Chaos Engineering Activity, and Platform Resource Consumption — each backed by the metrics described above. **Download:** [steadybit-platform-dashboard.json](https://github.com/steadybit/docs-public/blob/main/.gitbook/assets/steadybit-platform-dashboard.json) To install, in Grafana go to **Dashboards → New → Import**, then either upload the file or paste its contents. Select your Prometheus data source when prompted. ## Recommended Prometheus Alert Rules The following `PrometheusRule` is the exact configuration we run in production. Adjust the thresholds to your scale, but keep the structure — each rule maps to a Golden Signal above. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: steadybit-platform-golden-signals labels: release: prometheus spec: groups: - name: GoldenSignalsAlerts rules: # ---------------- Latency ---------------- - alert: SteadybitQueueLeadTimeCritical expr: max by (groupKey) (queue_lead_time) > 1800 for: 10m labels: { severity: critical } annotations: summary: "Message queue lead time critical" description: "Queue lead time has been above 30 minutes for 10 minutes." - alert: SteadybitQueueLeadTimeWarning expr: | max by (groupKey) (queue_lead_time) > 600 and max by (groupKey) (queue_lead_time) < 1800 for: 5m labels: { severity: warning } annotations: summary: "Message queue lead time elevated" - alert: SteadybitGetLatencyCritical expr: | sum(rate(http_server_requests_seconds_sum{method="GET",exception="none",app_kubernetes_io_name="platform"}[2m])) by (uri) / sum(rate(http_server_requests_seconds_count{method="GET",exception="none",app_kubernetes_io_name="platform"}[2m])) by (uri) > 0.5 for: 5m labels: { severity: critical } - alert: SteadybitPostLatencyCritical expr: | sum(rate(http_server_requests_seconds_sum{method="POST",exception="none",app_kubernetes_io_name="platform"}[2m])) by (uri) / sum(rate(http_server_requests_seconds_count{method="POST",exception="none",app_kubernetes_io_name="platform"}[2m])) by (uri) > 0.6 for: 2m labels: { severity: critical } # ---------------- Traffic ---------------- - alert: SteadybitRequestRateCritical expr: irate(http_server_requests_seconds_count{exception="none",app_kubernetes_io_name="platform"}[1m]) > 10 for: 1m labels: { severity: critical } # ---------------- Errors ---------------- - alert: SteadybitErrorRateCritical expr: | sum(rate(http_server_requests_seconds_count{app_kubernetes_io_name="platform",status=~"5.."}[1m])) / sum(rate(http_server_requests_seconds_count{app_kubernetes_io_name="platform"}[1m])) > 0.05 for: 5m labels: { severity: critical } annotations: summary: "Platform 5xx error rate above 5%" # ---------------- Saturation ---------------- - alert: SteadybitJvmHeapWarning expr: | sum(jvm_memory_used_bytes{app_kubernetes_io_instance="platform",area="heap"}) / sum(jvm_memory_max_bytes{app_kubernetes_io_instance="platform",area="heap"}) > 0.8 for: 5m labels: { severity: warning } - alert: SteadybitHikariPoolWarning expr: | (sum(hikaricp_connections_active) by (pool) / sum(hikaricp_connections_max) by (pool)) * 100 >= 60 and (sum(hikaricp_connections_active) by (pool) / sum(hikaricp_connections_max) by (pool)) * 100 < 80 for: 1m labels: { severity: warning } annotations: summary: "HikariCP pool {{ $labels.pool }} > 60 % utilized" - alert: SteadybitHikariPoolCritical expr: | (sum(hikaricp_connections_active) by (pool) / sum(hikaricp_connections_max) by (pool)) * 100 >= 80 for: 1m labels: { severity: critical } annotations: summary: "HikariCP pool {{ $labels.pool }} > 80 % utilized" ``` ## When an Alert Fires | Alert | First thing to check | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | Queue lead time | Database CPU and HikariCP saturation. The post-processing pipeline writes to Postgres on every step. | | GET / POST latency | Database, then JVM heap. | | 5xx error rate | Platform logs (`Platform Log Events` panel) for stack traces. | | JVM heap > 80 % | Capture a heap dump as described in [Troubleshooting › On-prem platform](/troubleshooting/common-fixes/on-prem-platform#create-heap-dump). | | HikariCP pool > 80 % | Long-running or blocked transactions. Follow the [Database Runbooks](/install-and-configure/install-on-prem-platform/database-runbooks). | | Blocked outgoing requests | A user webhook or hub points to a host that is blocked by your egress policy. | ## Related Pages * [Database Runbooks](/install-and-configure/install-on-prem-platform/database-runbooks) — recover from blocking transactions or disk pressure. * [Configuration Options](/install-and-configure/install-on-prem-platform/advanced-configuration) — JVM and database tuning parameters. * [Maintenance & Incident Support](/install-and-configure/install-on-prem-platform/maintenance-and-incident-support) — communicate planned maintenance and incidents to your users. * [Troubleshooting › On-prem platform](/troubleshooting/common-fixes/on-prem-platform) — common fixes for installation issues. # Database Runbooks {% hint style="info" %} This page applies to **on-premise installations** of the Steadybit platform. It assumes you are running a PostgreSQL 15+ database that the platform connects to. See [Database Configuration](/install-and-configure/install-on-prem-platform/advanced-configuration#database-configuration) for connection settings. {% endhint %} The Steadybit platform stores its state in PostgreSQL. This page covers two operational scenarios: 1. [The connection pool is saturated by long-running or blocked transactions.](#blocking-transactions) 2. [The database is running out of disk space.](#running-out-of-disk-space) In both cases, monitoring should give you advance warning — see [Monitoring the Platform](/install-and-configure/install-on-prem-platform/monitoring) for the alerts we recommend. {% hint style="warning" %} The queries below modify production data. Always run a `SELECT` first to confirm the scope, and take a backup before running `DELETE` or `pg_terminate_backend`. If you are unsure, [contact Steadybit support](https://www.steadybit.com/contact) before proceeding. {% endhint %} ## Blocking Transactions **Symptom:** the `SteadybitHikariPoolCritical` alert fires, target updates from agents are not being processed, or the UI feels stuck on operations that read or write the `target` table. **Cause:** PostgreSQL transactions that started but were never committed or rolled back continue to hold row-level locks. New connections from the platform queue up waiting for those locks and the HikariCP connection pool fills up. ### Step 1 — List transactions that hold locks ```sql SELECT a.datname, l.relation::regclass, l.transactionid, l.mode, l.granted, a.usename, a.query, a.query_start, age(now(), a.query_start) AS age, a.pid FROM pg_stat_activity a JOIN pg_locks l ON l.pid = a.pid ORDER BY a.query_start; ``` ### Step 2 — Narrow down to transactions older than 1 hour Most legitimate platform queries finish in under a second. Anything older than an hour is almost certainly stuck. ```sql SELECT a.pid, a.query, a.query_start, age(now(), a.query_start) AS age FROM pg_stat_activity a JOIN pg_locks l ON l.pid = a.pid WHERE a.query_start < now() - INTERVAL '1 hour' ORDER BY a.query_start; ``` ### Step 3 — Terminate the offending backend Replace `` with the process ID returned by the previous query: ```sql SELECT pg_terminate_backend(); ``` `pg_terminate_backend` rolls back the transaction and closes the connection. The platform reconnects automatically; no restart is required. ### Step 4 — Verify recovery * Re-run the query from Step 1 — the count of held locks should drop. * Check the `Datasource connections` panel; `hikaricp_connections_active` should fall back to its baseline. * Confirm that the message queue lead time (`queue_lead_time`) starts decreasing. ## Running Out of Disk Space **Symptom:** Postgres logs report `could not extend file` or `No space left on device`, or your storage monitoring is approaching the volume capacity. **Cause:** historical execution data, target snapshots, or audit log entries have accumulated beyond what the configured retention removes. Bloat in heavily updated tables (notably `target`) can also consume far more space than the live row count would suggest. ### Step 1 — Identify the largest tables ```sql SELECT nspname AS schema, relname AS table, reltuples AS row_estimate, pg_size_pretty(pg_total_relation_size(c.oid)) AS total, pg_size_pretty(pg_indexes_size(c.oid)) AS index, pg_size_pretty(pg_total_relation_size(reltoastrelid)) AS toast, pg_size_pretty(pg_total_relation_size(c.oid) - pg_indexes_size(c.oid) - COALESCE(pg_total_relation_size(reltoastrelid), 0)) AS table_only FROM pg_class c LEFT JOIN pg_namespace n ON n.oid = c.relnamespace WHERE relkind = 'r' ORDER BY pg_total_relation_size(c.oid) DESC LIMIT 20; ``` The most common offenders are: | Table | Why it grows | | ----------------------------------------------------------------------- | ------------------------------------------------------------------------- | | `target` | Snapshot of every target seen by every agent. Heavy update churn → bloat. | | `target_stats`, `target_submission_tracking` | Per-submission counters that the platform vacuums periodically. | | `audit_log` | One row per administrative action. | | `experiment_execution`, `execution_log_event`, `execution_metric_event` | Grow with the number of experiment runs. | ### Step 2 — Reclaim space with VACUUM The platform runs scheduled `VACUUM`/`ANALYZE` (see `STEADYBIT_DB_MAINTENANCE_*` in [Configuration Options](/install-and-configure/install-on-prem-platform/advanced-configuration#database-maintenance)). When you need to reclaim space immediately, run a full vacuum **on a low-traffic window** — `VACUUM FULL` takes an exclusive lock on the affected table: ```sql -- Releases space back to the OS but locks the table. VACUUM FULL; -- Or, target a single table: VACUUM FULL public.target; ``` For a non-blocking alternative on PostgreSQL, install and use [`pg_repack`](https://reorg.github.io/pg_repack/). ### Step 3 — Delete obsolete data (if needed) Only delete data after confirming that the platform's built-in retention is not enough for your situation, and after taking a backup. The data the platform tolerates losing the most are old target snapshots: ```sql -- Replace the cutoff with a date well before your current retention horizon. DELETE FROM public.target WHERE timestamp < '2025-01-01 00:00:00'; ``` After a large delete, run `VACUUM FULL` on the affected table again to release the space. ### Step 4 — Increase the volume size If after vacuuming, the database is still close to full, the safest fix is to increase the underlying volume. On AWS RDS this is a non-disruptive operation; on a self-hosted Postgres, follow your storage provider's resize procedure. Plan ahead by reviewing your [Machine Requirements](/install-and-configure/install-on-prem-platform/advanced-configuration#rds-machine-requirements) — at \~100 k targets we recommend at least 20 GB of database storage. ## Preventive Maintenance The platform performs automatic maintenance on a configurable schedule. The defaults are sensible for most installations; review them if you operate at scale or have observed bloat. | Variable | Default | Description | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------ | | `STEADYBIT_DB_MAINTENANCE_ENABLED` | `true` | Enable automatic VACUUM/ANALYZE. | | `STEADYBIT_DB_MAINTENANCE_CRON` | `0 0 0 ? * SAT *` | Saturday at midnight. | | `STEADYBIT_DB_MAINTENANCE_TABLES` | see [Configuration Options](/install-and-configure/install-on-prem-platform/advanced-configuration#database-maintenance) | Tables included in the maintenance window. | Combined with the alerts described in [Monitoring the Platform](/install-and-configure/install-on-prem-platform/monitoring), this is usually enough to keep the database healthy without manual intervention. ## Related Pages * [Monitoring the Platform](/install-and-configure/install-on-prem-platform/monitoring) — alerts that warn you before these situations become critical. * [Configuration Options › Database Configuration](/install-and-configure/install-on-prem-platform/advanced-configuration#database-configuration) — connection and maintenance settings. * [Troubleshooting › On-prem platform](/troubleshooting/common-fixes/on-prem-platform) — installation-time database issues. # Maintenance & Incident Support During the operation of the on-premise Steadybit platform, you may run into situations that require users to be informed about incidents and planned maintenance that will affect on-premise Steadybit users. To inform these users, you can tell the Steadybit platform about incidents and maintenance windows. Once done, the platform can * show banners in the user interface indicating the specific situation, * require explicit confirmation when executing experiments via the user interface and * optionally disable experiment runs entirely during an ongoing incident. This feature is available in versions > 1.0.2 of the Steadybit platform. ## Learning about Platform Activity You can use the following metrics to learn whether maintenance on the Steadybit platform would be affecting any users: * `platform.experiments.executing` is exposed by the Steadybit platform as a Prometheus metric. It represents the number of currently executing experiments. ## Admin API Endpoints The maintenance and incident API endpoints are only reachable through the admin port `9090` on the platform Kubernetes deployment workload. This port is not exposed to users by default, and you should only use it for administrative purposes. API endpoints on port `9090` do not require authentication. One option to reach this port from your local machine is a port forward, as the following snippet shows: ```bash kubectl port-forward -n steadybit-platform deployment/steadybit-platform 9090 ``` ### Maintenance To configure a maintenance window, you can use the API endpoints under path `/actuator/systemstatusmaintenance`. You can retrieve the maintenance configuration via HTTP `GET` or clear it via HTTP `DELETE`. The following example shows how you could leverage HTTP `POST` to configure a maintenance window. ```bash curl -X POST \ -H "Content-Type: application/json" \ http://localhost:9090/actuator/systemstatusmaintenance -d ' { "title": "Planned maintenance", "message": "We need to perform maintenance on the platform", "statusPage": "https://status.example.com", "start": "2024-06-07T00:00:00Z", "end": "2024-06-08T00:00:00Z" } ' ``` ![System banner appearing at the top of the Steadybit UI presenting the information provided through the maintenance API](/files/baFHRb6MbgBqRHCj7yVu) ### Incidents You can use incidents to inform users of service disruption. In contrast to scheduled maintenance, an incident is never scheduled and has varying severity. To configure an incident, you can use the API endpoints under path `/actuator/systemstatusincident`. You can retrieve the incident configuration via HTTP `GET` or clear it via HTTP `DELETE`. The following example shows how you could leverage HTTP `POST` to configure an incident. ```bash curl -X POST \ -H "Content-Type: application/json" \ http://localhost:9090/actuator/systemstatusincident -d ' { "title": "Platform Overloaded", "message": "We are experiencing high load on the platform", "statusPage": "https://status.example.com", "allowExperimentExecution": true, "severity": "DEGRADED_PERFORMANCE" } ' ``` ![System banner appearing at the top of the Steadybit UI presenting the information provided through the incident API](/files/hhIgZ3OXIRKWIOnFzwzN) The following severities are supported by the API. The user interface will adapt its banner style according to the chosen incident severity. * `UNDER_MAINTENANCE` * `DEGRADED_PERFORMANCE` * `PARTIAL_OUTAGE` * `MAJOR_OUTAGE` # OIDC Integration The Steadybit Platform supports integration with OpenID Connect (OIDC) into a centralized user management. This integration enables OIDC-based authentication and automatic team membership assignment. ## Activation To enable the OIDC integration, set the following environment variable: ``` STEADYBIT_AUTH_PROVIDER=OAUTH2 ``` You can use an OIDC compatible authentication provider for user authentication. | Config | Value | | ------------- | ------------------------------------------ | | Grant type | `authorization_code` | | Redirect uri | `https:///oauth2/login/code/default` | | Login url | `https:///login` | | Response type | `code` | Be aware to configure your ingress / loadbalancer to set the `X-Forwarded-Proto` and `x-forwarded-for` headers. Otherwise, the correct redirect URL will not be generated. ## Authentication {% hint style="info" %} The first user to sign in will be assigned the `admin` role; all others will be assigned the `user` role. {% endhint %} To connect to a compatible Identity Provider (IdP) set the following environment variables: * `STEADYBIT_AUTH_OAUTH2_ISSUER_URI`: URI for the OpenID Connect discovery endpoint * `STEADYBIT_AUTH_OAUTH2_CLIENT_ID`: The client ID to use for the OIDC registration * `STEADYBIT_AUTH_OAUTH2_CLIENT_SECRET`: The client secret to use for the OIDC registration When a user authenticates via OIDC in the Steadybit Platform for the first time, a corresponding Steadybit user object is created. The user has the type `user` (see [Permissions](/install-and-configure/manage-teams-and-users/permissions)). If the ID token contains a groups claim, the user is automatically assigned to the corresponding teams (see [Synchronization](#synchronization)). For detailed OIDC authentication configuration parameters, refer to [OpenID Connect Authentication](/install-and-configure/install-on-prem-platform/advanced-configuration#openid-connect-authentication). ## Synchronization To automate team assignment, the IdP can return a token with a claim (attribute) containing team identifiers (team keys), the user should be assigned to. ### Configuration Configure the name of the OIDC token claim used for team mapping with the following environment variable: ``` STEADYBIT_AUTH_OAUTH2_CLAIMS_TEAM_NAME_ATTRIBUTE_NAME ``` * The claim value must be a JSON array of team keys (for example, `["ADM", "TEAM1", "TEAM2"]`). * By default, the platform expects the claim to be named `groups`. You can change it to any other claim name your IdP provides. * Additional OAuth-related configuration options are listed under [OpenID-Connect authentication](/install-and-configure/install-on-prem-platform/advanced-configuration#openid-connect-authentication). #### Usage example If the IdP returns team names in a claim called `mygroups`, configure: ``` STEADYBIT_AUTH_OAUTH2_CLAIMS_TEAM_NAME_ATTRIBUTE_NAME=mygroups ``` #### Scoping If the claim used for team association is not included by default, request it via the scope parameter in your OIDC configuration. For example, if your IdP sends team information in a claim called `mygroups`, configure: ``` STEADYBIT_AUTH_OAUTH2_SCOPE=openid,profile,email,mygroups ``` The scopes `openid,profile,email` must always be included in the request. ### How it works During authentication, the platform inspects the returned OIDC ID token for the configured claim and expects an array of team keys (for example, `["ADM", "TEAM1", "TEAM2"]`). The process is as follows: 1. Read the configured claim from the ID token and validate that it is an array of strings. 2. For each team key: * The team key is derived from the claim value: the first 16 characters, uppercased (for example, `"my-developers"` becomes `MY-DEVELOPERS`). * Look up the corresponding team in Steadybit. * If the team does not exist, create it. * Assign the authenticated user to the team with the role `member`. 3. Remove the user from any OIDC-managed teams that are no longer present in the claim. If the claim is empty or absent, the user is removed from all OIDC-managed teams. Teams and memberships created through OIDC synchronization are marked as **managed by OIDC**. This has the following effects: * **Team key** of OIDC-managed teams cannot be edited by platform admins — it is controlled by the IdP. * **Team name** of OIDC-managed teams can be edited by platform admins. The IdP-provided name is used when the team is first created; subsequent OIDC syncs do not overwrite the team name, so renames made in Steadybit are preserved. * **OIDC-managed members** cannot be removed from teams via the UI or API by platform admins. Membership changes must be made in the IdP. * **Manually added members** on an OIDC-managed team are preserved and can be managed normally. OIDC synchronization only affects OIDC-managed memberships. * Other team properties (description, allowed environments, allowed actions) remain editable by admins. ### Limitations * Scope of mapping: The dynamic mapping feature applies only to teams. Users are not assigned platform roles (for example, `admin`) or team-specific roles (for example, `team owner`) via this mechanism. Platform and team roles are managed separately (for example, during initial setup or by manual assignment). * ID token requirement: The claim containing the team information must be included in the OIDC ID token. Claims present only in the access token are not processed for team mapping. #### Example OIDC token Below is an example payload from an OIDC ID token that includes team mapping information in the custom attribute `mygroups`: ```json { "header": { "alg": "RS256", "typ": "JWT", "kid": "xxx" }, "payload": { "usernames": ["auth0|1234", "google-oauth2|123"], "mygroups": ["TEAM1", "TEAM2", "ADM"], "nickname": "ansgar", "name": "max musterman", "picture": "https://s.gravatar.com/avatar/xxx?s=480&r=pg&d=identicon", "updated_at": "2025-03-26T16:52:18.722Z", "email": "max@musterman.com", "email_verified": true, "iss": "https://login.example.com/", "aud": "123", "sub": "auth0|1234", "iat": 1234567890, "exp": 1234567890, "sid": "xxxx", "nonce": "xxx" }, "signature": "xxx" } ``` ### Troubleshooting If team synchronization does not work as expected, check the following: * Claim name mismatch: Ensure that the claim name in the OIDC ID token matches the value of `STEADYBIT_AUTH_OAUTH2_CLAIMS_TEAM_NAME_ATTRIBUTE_NAME`. For example, if the IdP sends teams in the claim `groups` but the parameter is set to `mygroups`, the teams will not be mapped. * Claim location: Verify that the team attribute is present in the ID token. Dynamic mapping only processes claims in the ID token, not in the access token. * Token content: Confirm that the claim contains a valid JSON array of team keys (for example, `["ADM", "TEAM1"]`). * Missing groups claim in Okta: Make sure the groups claim is requested as described above and that the [Okta application is configured correctly](https://developer.okta.com/docs/guides/customize-tokens-groups-claim/main/#add-a-groups-claim-for-the-org-authorization-server). # LDAP Integration The Steadybit Platform supports integration with Lightweight Directory Access Protocol (LDAP) servers for centralized user management. This integration enables LDAP-based authentication and automatic team membership assignment based on LDAP group memberships. ## Activation To enable LDAP integration, set the following environment variable: ``` STEADYBIT_AUTH_PROVIDER=LDAP ``` ## Authentication The Steadybit Platform supports LDAP-based user authentication with the following behavior: The platform accesses the LDAP server anonymously by default. To use authenticated access, configure both `STEADYBIT_AUTH_LDAP_MANAGER_DN` and `STEADYBIT_AUTH_LDAP_MANAGER_PASSWORD` environment variables. Users are authenticated using LDAP bind operations with their credentials. To use password comparison instead, set `STEADYBIT_AUTH_LDAP_METHOD` to `password-compare`. For detailed LDAP authentication configuration parameters, refer to [LDAP-Authentication](/install-and-configure/install-on-prem-platform/advanced-configuration#ldap-authentication). ## Synchronization {% hint style="info" %} When LDAP-based user synchronization is enabled, manual user management and team assignment operations are disabled. {% endhint %} The Steadybit Platform can synchronize users and team associations from LDAP at regular intervals. The synchronization process consists of three phases: ### 1. User Synchronization The platform queries the configured LDAP groups and synchronizes user accounts based on group membership: * **Administrator Users**: Retrieved from the LDAP group specified in `STEADYBIT_AUTH_LDAP_SYNC_ADMIN_GROUP_DN` * **Regular Users**: Retrieved from the LDAP group specified in `STEADYBIT_AUTH_LDAP_SYNC_USER_GROUP_DN` LDAP groups must contain `uniqueMember` or `member` attributes that reference user entries. Users are created or updated in the Steadybit Platform based on LDAP data. **Note**: Users removed in LDAP are not automatically removed from the platform. ### 2. Team Synchronization Teams are identified using the LDAP search filter defined in `STEADYBIT_AUTH_LDAP_SYNC_TEAM_SEARCH_FILTER`. Teams that do not exist in the Steadybit Platform are created automatically. Teams that no longer exist in LDAP are automatically deleted from the platform, provided they were originally created by LDAP synchronization. ### 3. Team Member Synchronization Users referenced in LDAP teams are assigned to the corresponding Steadybit teams with the `member` role. LDAP-managed members that are no longer referenced in the LDAP group are removed from the team. Manually added members on an LDAP-managed team are preserved and not affected by the synchronization. Only users that exist in the Steadybit Platform (from User Synchronization) can be assigned to teams. ### Managed by LDAP Teams and memberships created through LDAP synchronization are marked as **managed by LDAP**. This has the following effects: * **Team name and key** of LDAP-managed teams cannot be edited by platform admins. They are controlled by LDAP. * **LDAP-managed members** cannot be removed from teams via the UI or API by platform admins. Membership changes must be made in LDAP. * **Manually added members** on an LDAP-managed team are preserved and can be managed normally. * Other team properties (description, allowed environments, allowed actions) remain editable by admins. For additional synchronization configuration parameters, refer to [LDAP-Authentication](/install-and-configure/install-on-prem-platform/advanced-configuration#ldap-authentication). ## Troubleshooting ### LDAP Connection Closed Errors If the platform encounters `LDAP connection has been closed` errors, configure LDAP connection pooling by setting the `JAVA_OPTS` environment variable for the platform process: ``` JAVA_OPTS=-Dcom.sun.jndi.ldap.connect.pool.timeout=20000 -Dcom.sun.jndi.ldap.connect.pool.maxsize=20 ``` # Manage Teams and Users Steadybit lives transparency by allowing every user to see the performed actions of other teams and users. Even so, a Role-Based-Access-Control (RBAC) gives you the possibility to control fine-grained permissions and determine which parts users can take action. * Start by defining your [Teams](/install-and-configure/manage-teams-and-users/teams) * Invite and assign [Users](/install-and-configure/manage-teams-and-users/users) to the teams * Learn more about [Permissions and Roles](/install-and-configure/manage-teams-and-users/permissions) # Teams Teams are the grouping element for multiple users. Each team consists of multiple users and owns separate permissions for environments and attacks, allowing fine grained access control for your experiments. Creating teams is possible only with the `admin` permission. Within teams there can be owners who manage the team and its members. ## Default Team: Administrators Per default one team is created called `Administrators` (key: `ADM`). It contains only the first user of Steadybit and has access to the [Global environment](/install-and-configure/manage-environments). ## Creating Teams In the create dialog you should first enter a **name** for the Team. The name can be edited after creation by admins or team owners. You must enter a **key** for the team consisting of one to eight characters. The key cannot be changed after the creation. Each experiment belonging to this team, will be assigned an identifier with the team key as prefix. ## Permissions In the edit/create dialog admins can give permissions to * the attacks a team is allowed to execute and * which [environments](/install-and-configure/manage-environments) containing different targets the team is allowed to operate on. You can think of it as a maximum blast radius this team can attack. If permissions for teams with existing experiments are removed, the blast radius of the experiments is not widened. If you add permissions to teams with existing experiments, the new permissions will be applied to the experiments. ## Managing Team Members Admins or team owners can add or remove members to the team and assign them either the `member` or `owner` role. ## Integrations Team and team member management can be automated by the [OIDC Integration](/install-and-configure/install-on-prem-platform/oidc-integration) and [LDAP Integration](/install-and-configure/install-on-prem-platform/ldap-integration). # Users This page describes user roles, how to invite users, and how external identity systems affect user management. ## Role Types Users have two role types that determine access and permissions: * Platform role: Assigned at the platform (tenant) level. Each user has exactly one of `admin` or `user`. * Team role: Assigned within a specific team. A user can be an `owner` or a `member` in that team. See [Permissions](/install-and-configure/manage-teams-and-users/permissions) for the functional differences between these roles. {% hint style="info" %} Users who are not members of any team do not count toward license limits. {% endhint %} ## Inviting Users (SaaS only) New users can be invited by email and directly assigned to teams and roles. When an invitation is accepted, the corresponding user is created and gains access to your Steadybit tenant. ## Integrations User management can be automated by the [OIDC Integration](/install-and-configure/install-on-prem-platform/oidc-integration) and [LDAP Integration](/install-and-configure/install-on-prem-platform/ldap-integration). # Permissions The roles in Steadybit are associated with permissions, which cannot be altered. See the table for the permissions: ## Install and Configure | Permission/Role | Admin | Team Owner | Team Member | Any Authenticated User | | ------------------------------------------------------------------------- | ----- | ---------- | ----------- | ---------------------- | | [Manage teams](/install-and-configure/manage-teams-and-users/teams) | ✅[^1] | ☑️[^2] | ❌ | ❌ | | [Manage users](/install-and-configure/manage-teams-and-users/users) | ✅[^1] | ❌ | ❌ | ❌ | | [Add agents](/install-and-configure/install-agent) | ✅ | ❌ | ❌ | ❌ | | [Manage environments](/install-and-configure/manage-environments) | ✅ | ❌ | ❌ | ❌ | | [View audit log](/integrate-with-steadybit/audit-log) | ✅ | ❌ | ❌ | ❌ | | [Database export](/troubleshooting/troubleshooting#database-export) | ✅[^3] | ❌ | ❌ | ❌ | | [Manage access tokens](/integrate-with-steadybit/api#access-tokens) | ✅ | ✅ | ❌ | ❌ | | [Manage integrations](/integrate-with-steadybit/integrate-with-steadybit) | ✅ | ✅ | ❌ | ❌ | ## Prepare Rollout of Chaos Engineering | Permission/Role | Admin | Team Owner | Team Member | Any Authenticated User | | --------------------------------------------------------------------------------- | ----- | ---------- | ----------- | ---------------------- | | [Manage services](/use-steadybit/services) | ✅ | ✅ | ❌ | ❌ | | [Manage service profiles](/install-and-configure/manage-service-profiles) | ✅ | ❌ | ❌ | ❌ | | [Manage experiment templates](/install-and-configure/manage-experiment-templates) | ✅ | ❌ | ❌ | ❌ | | [Manage properties](/install-and-configure/manage-properties) | ✅ | ❌ | ❌ | ❌ | | [Manage hubs](/integrate-with-steadybit/hubs) | ✅ | ❌ | ❌ | ❌ | ## Doing Chaos Engineering | Permission/Role | Admin | Team Owner | Team Member | Shared-with Team Member[^4] | Any Authenticated User | | --------------------------------------------------------------------------------------------------------------------------- | ----- | ---------- | ----------- | --------------------------- | ---------------------- | | [Work within services](/use-steadybit/services) | ✅ | ✅ | ❌ | ❌ | ❌ | | [Design experiments](/use-steadybit/experiments/design) | ✅ | ✅ | ✅ | ❌ | ❌ | | [Share Experiment](https://github.com/steadybit/docs-public/blob/main/use-steadybit/experiments/share-experiment/README.md) | ✅ | ✅ | ❌ | ❌ | ❌ | | [Run](/use-steadybit/experiments/run), and [schedule experiments](/use-steadybit/experiments/schedule) | ✅ | ✅ | ✅ | ✅ | ❌ | | [Capture properties](/use-steadybit/experiments/properties) | ✅ | ✅ | ✅ | ☑️[^5] | ❌ | | [View experiments](/use-steadybit/experiments) | ✅ | ✅ | ✅ | ✅ | ✅ | | [Stop experiments](/use-steadybit/experiments) | ✅ | ✅ | ✅ | ✅ | ✅ | | Stop all experiments via [emergency stop](/use-steadybit/experiments/emergencystop) | ✅ | ✅ | ✅ | ✅ | ✅ | [^1]: Unless synced via [LDAP](/install-and-configure/install-on-prem-platform/ldap-integration) [^2]: Team owners can [add/delete team members](/install-and-configure/manage-teams-and-users/teams#managing-team-members) and [delete teams](/install-and-configure/manage-teams-and-users/teams) [^3]: Unless [disabled via configuration](/install-and-configure/install-on-prem-platform/advanced-configuration) [^4]: A member of a team that an experiment has been [shared with](https://github.com/steadybit/docs-public/blob/main/use-steadybit/experiments/share-experiment/README.md) [^5]: Properties for a run can be edited by shared-with team member; Properties for the design not # Manage Environments No one knows your system better than you, so it is obvious that you should be the one who structures the discovered targets. It's up to you whether you use environments to separate parts of your system from each other (like "bounded contexts" in domain driven design), to separate your environments (like development, QA and Production) or simply both. ## Default Environment: Global Per default, your system contains one environment called `Global`. This is the place where **every** target that Steadybit has discovered will be in. So, using Global for the start is fine (when trying out Steadybit), but on the long run dangerous and you should create your own. ## Define your own Environment To create or change an environment go to `/settings/environments` and choose to either add an environment or edit an existing one. An environment consists of a * **Name**: A meaningful name helps to find the environment afterwards (e.g. "Onlineshop DEV" or "Contract Management PROD"). * **Team**: For using that environment to run an experiment you need to assign the team to it. This can be done from environment-perspective (`/settings/environments` ) as well as from Team-perspective (`/settings/teams`). It limits access to the users of a team to certain environments (so targets). * **Environment Scope**: The environment scope defines targets belonging to an environment by using our [discovery data](/concepts/discovery). This is the upper limit of targets that can be attacked when running an experiment in a certain environment. ### Define Environment Scope With the query UI, you can build regular queries which are connected with an AND expression. For most use cases this can be sufficient. But sometimes, it's necessary to have even more control over the query. For instance, if you want to check for any key-value pair existence or you want to match all, but not a specific attribute, e.g. cluster.name="prod". Queries like that can easily be written with the Query language. For detailed information, please [visit the query language documentation](/concepts/query-language). Finally, don't forget to finally assign the environment to a team and give it a meaningful name. That's it, you are ready to use that environment in an experiment! ![Create a new environment](/files/7krCQzZLCC9ubIhgJBGS) ### Use Environments Once the environments are defined and assigned to teams, you can make use of them when e.g. [designing an experiment](/use-steadybit/experiments). ## Environment Variables With an environment often comes variables that are useful in an experiment associated with the provided environment (e.g., the name of the Kubernetes cluster, the HTTP base path of a load-balanced ingress, or the AWS account). You can manage environment variables to define those and use them consistently across all your experiments referencing the environment. Each variable can hold a fixed value or a [dynamic value](/use-steadybit/experiments/variables#value-settings) that is resolved from your infrastructure when an experiment run starts. You can [learn more in the experiment's variable section](/use-steadybit/experiments/variables). # Manage Service Profiles A service profile defines the set of experiments provided to a [service](/use-steadybit/services). It specifies which reliability categories matter (e.g., Scalability, Redundancy, Dependencies) and which experiment templates are instantiated per service per category. When a service is linked to a profile, Steadybit automatically generates concrete experiments from those templates using the service's target scope and validations. ## Service Profile Structure A service profile consists of **categories** and **experiment templates** within those categories. ### Categories Categories group related experiments and are shown as filter options on the service detail page (e.g., Redundancy, Scalability, Dependencies). They help teams focus on one reliability dimension at a time. ### Experiment Templates Each [experiment template](https://github.com/steadybit/docs-public/blob/main/use-steadybit/experiments/templates/README.md) in a profile defines the structure of an experiment to be generated for each service. When a service uses a profile, Steadybit instantiates a profile's template on-the-fly by substituting the service into the template — producing ready-to-run experiments scoped to that specific service. To reference to a service, you can create a template placeholder with the placeholder key `[[SERVICE]]`. Use this placeholder in, e.g., a service validation drop down or a target query (`service.id="[[SERVICE]]"`). ![Experiment template linking service in dropdown](/files/XLDHODsTLGKDI0uROQn4) {% hint style="warning" %} Be aware, that removing a template from a service profile results in deleting provided experiments and experiment runs for all services refering to this service profile. If you delete a template from one category and assign it to another one, without saving in between, no experiments are deleted. {% endhint %} ### Linking a Service Profile to a Service A service profile is selected when creating or editing a service via the **Customize** tab in the [service settings](/use-steadybit/services#customize). Every service must be linked to exactly one service profile. {% hint style="warning" %} Be aware, that changing a service's service profile results in deleting provided experiments and experiment runs that aren't part of the newly associated profile anymore. {% endhint %} ## Service Profiles You can use the existing built-in service profiles, coming from Steadybit for easy starting and best practices in reliability. Alternatively, you can create your own service profile to match organizational standards. ### Built-in Starter Service Profile Steadybit ships with a Starter built-in service profiles. The **Steadybit Starter** profile makes it easy to begin with services. It contains a broad set of experiment templates covering impactful reliability scenarios for various technologies, so teams can get meaningful results quickly without being overwhelmed. We recommend creating your own custom service profile for rolling Steadybit out into your organization. ### Custom Service Profiles Administrators can create custom service profiles to define organization-specific reliability standards. This allows you to: * Define categories that reflect your internal reliability model * Select experiment templates that match the technology stack and failure modes relevant to your services * Standardize the reliability evaluation across all teams and services Custom profiles are managed in **Settings** → **Service Profiles**. ![Custom Service Profile](/files/8dFzd5OVySiNO4smhtpK) You can define one service profile as default, to use this for every new service. {% hint style="info" %} Only administrators can create and manage service profiles. All profiles are available tenant-wide and can be assigned to any service. {% endhint %} # Manage Experiment Templates Experiment templates allow you to easily share knowledge with the teams and make it easy for everyone to do and learn Chaos Engineering. You can create your own custom Experiment templates or import existing ones from a Reliability Hub. ## Permissions and Scope Only admin users can manage experiment templates and thus create, import, edit, or delete experiment templates. All experiment templates are scoped tenant-wide and automatically shared with all teams and users. Users can even create experiments from experiment templates where they lack permissions for actions (e.g., the template contains a `Shutdown host` attack for which the team lacks permission). However, before being able to run the experiment, they need to remove the action. ## Create Templates Create and edit a template via the `Platform's settings` -> `Templates` section. Choose the Blank approach to start from scratch. Alternatively, you can upload a template from a file or create one from an existing experiment via `Save as Template`. ![The template editor's welcome screen](/files/EfFtWE7KZ7a0dJW4vLvl) An experiment template consists of three parts: the *experiment* created when using a template, *template placeholders* to make the experiment adaptable, and *template description* to make the template discoverable for users. To save a template, you must specify all required values of all three parts. ### Experiment ![Template's experiment editor](/files/vTmRweG1ECFvysnmXP2Z) To define the experiment of the template, you benefit from a similar experiment editor as when creating an experiment. #### Environment When designing the experiment of the template, all targets are resolved to the `Global` environment. However, when using a template to create an experiment, the user can select the desired environment, and all targets and variables are evaluated against the selected environment. This way, it is still easy to define a template and safe for teams to use one. #### Templating To make the experiment adaptable to different contexts, you can reference [template placeholders (see below)](#template-placeholder) and [variables](/use-steadybit/experiments/variables) in the experiment's steps (e.g., target query, parameters, and step labels), hypothesis, and experiment name. Template placeholders use the `[[placeholder]]` syntax and are documented within the template editor. Variables use the `{{variable}}` syntax and are not restricted to a single scope: when someone uses the template, they can provide a value to create an [environment variable](/use-steadybit/experiments/variables#environment) or leave it empty to fall back to an [experiment variable](/use-steadybit/experiments/variables#experiment) (see [creating an experiment from a template](/use-steadybit/experiments/design#from-template)). #### Properties Similar to experiments, an experiment in an experiment template can have properties assigned. Properties' values can be hard-coded in the experiment or can reference [template placeholders (see below)](#template-placeholder) and [variables](/use-steadybit/experiments/variables) to capture these when [using the experiment](https://github.com/steadybit/docs-public/tree/main/use-steadybit/experiments/templates/README.md). Templates can only reference defined property definitions. Learn more on [managing properties](/install-and-configure/manage-properties) and [using properties in experiments](/use-steadybit/experiments/properties). ### Template Placeholder ![Template's placeholder](/files/4n2JJvQkwaON8Sa5z35d) A template placeholder defines occurrences where the exact value needs to be specified whenever one uses a template. A template placeholder consists of a * **Displayed Name** that is a human-readable name shown to the user of a template (e.g., `AWS Zone`, or `Kubernetes Deployment` * **Placeholder Key** that defines all occurrences in the template that should be replaced later on (e.g., `[[aws_zone]]`, or `[[k8s_deployment]]`) * **Placeholder Description** is shown when using the template and should help users understand the value. The description supports Markdown and can include, e.g., the intention of the placeholder and a question to ask for the value. ### Template Description ![Template's description](/files/kl8OY0tEZRD6Kf6pFSXd) Describe your template to make it approachable for users. The template description consists of the template's * **Title**, which is shown in search results, detail view before creating an experiment from a template * **Details** that allows you to provide noteworthy context and describe, e.g., the overall intention of the template or when it is useful. The field supports Markdown. * **Tags** to structure your template by assigning it to new or existing tags. ## Edit Templates You can edit a template and change all its content whenever needed. Please note that changes don't affect experiments created from a template. ## Delete Templates After you delete a template, users can't create an experiment from it anymore. However, experiments previously created using that template are never deleted automatically. Instead, you can also [hide a template](#hide-a-template) to temporarily prevent the use of a template. ## Hide Templates ![Template list provide capability to hide templates](/files/ns15KjuBohBgf8OqmMlJ) By default, all saved templates are discoverable when creating a new experiment from a template. However, you can hide a template to prevent users from using it. To hide a template, change the toggle in `Settings`> `Template` of a specific template. Experiments that have already been created from that template are not affected. ## Export Templates as Files Once you've saved your template, you can export it as a file from the template details modal in `Settings` -> `Template`. ![Template details including possibility to download it as a file](/files/eFuaVQo63N62WfRbuAsx) The saved file can be imported later on, either as a template or as an experiment (see [import templates](#import-templates-files)). In addition, you can version your template as a file and update it automatically using Steadybit APIs. ## Import Templates Files You can upload a template file into your Steadybit platform either as a new template so that everyone can create an experiment from it (via `Settings` -> `Template` -> `Create Template`) directly as an experiment by uploading it into the experiment creation flow (`Experiments` -> `New Experiment`) and running through the template wizard. While the first option is only available to administrators, the second option is available to everyone. {% hint style="warning" %} When you export a template, the file contains a unique UUID of the template. Whenever you import the template again, and the UUID already exists, the existing template is overwritten. {% endhint %} ## Share Templates Instead of sharing template's files directly, you can also use Steadybit's [hub connection](/integrate-with-steadybit/hubs) to share a template via connected hubs. ### Import Templates from Hub The hub connection makes it super easy to benefit from the best practices of the Chaos Engineering community (e.g., via Steadybit's [Reliability Hub](https://hub.steadybit.com/)). Furthermore, if you're using Steadybit on-premise, you can easily keep multiple platforms in sync by using the hub connections via API. To import a template from a hub, go to `Settings` -> `Templates` -> `Create Template` -> `From Hub`. You first have to select a connected hub ([learn how to connect a new hub](/integrate-with-steadybit/hubs#connect-new-hubs)). ![Import template by selecting the hub connection](/files/napGpgKRi7e6eT2Jmmq5) Next, choose the templates you want to import. If your template is already imported, you can re-import it to update it from the connected hub. ![Import template by selecting the hub connection](/files/Kab2iB3MG99P89sXg50y) In addition, we provide an API to automate template syncing with connected hubs, and you can also [host your own private hub](/integrate-with-steadybit/hubs#host-your-own-hub) to share templates within your organization. ### Export Templates to Reliability Hub We're always looking forward to your contribution to Steadybit's public [Reliability Hub](https://hub.steadybit.com/). Contributing is easy, just follow these steps: 1. Create a [fork of the repository reliability-hub-db](https://github.com/steadybit/reliability-hub-db/fork) into your GitHub account 2. Check out the forked repository to your local machine 3. If you haven't already, add yourself as a maintainer to `maintainers//description.yml` (see, e.g., [maintainer Steadybit](https://github.com/steadybit/reliability-hub-db/blob/main/maintainers/com.steadybit/description.yml)). 4. Download your created template from the platform and save it to a new folder in `templates//template.json`. Please ensure the template's description includes a short explanation, motivation, and structure overview. Also, you have to parameterize all environment-specifics via template placeholders. ![Download template from template details](/files/Df6ivPERycEDqwE8pMB7) 5. Add a `description.yml` to the folder `templates/` which references your maintainer, the repository location and license information (see, e.g., [description.yml for template EC2 restart](https://github.com/steadybit/reliability-hub-db/blob/main/templates/aws-ec2.ec2-restart/description.yml)). 6. Add your template to the [index.json's](https://github.com/steadybit/reliability-hub-db/blob/main/index.json) `templates` array and update the Unix timestamp `lastChange` so that users can import it to their Steadybit platform via hub connection. 7. Create a pull request from your forked repository to [reliability-hub-db](https://github.com/steadybit/reliability-hub-db) and wait for feedback from us or the community! We're looking forward to your valuable experiment template! 🎉 # Manage Properties Properties are key-value pairs that can be used to add additional information to your experiments and runs. They can be used to store metadata about the experiment, such as the experiment's purpose, the team responsible for it, the outcome, or just a note of a specific run or any other relevant information. Properties consist of three parts: * Property definitions: The administrator can define which properties are available in the Steadybit platform. These definitions can be assigned to experiments and runs. * Property assignments: Assignments define the relationship between a property definition and an experiment. They define where the property is visible and editable. * Administrators can assign properties to each experiment in your Steadybit tenant and even mark it as a required property. * Users can assign properties to a single experiment or a single run that they are allowed to edit. This allows users to add additional information to the experiment. More details can be found in the [experiment's section of property assignments](/use-steadybit/experiments/properties#assign-properties). * Property values: The actual values of the properties. Values saved as part of an experiment design are copied to each experiment run. These values can be updated after the experiment has been run, if the assignment configuration allows so. Learn more in the [experiment's section of properties](/use-steadybit/experiments/properties). ## Examples * Adding a note to a single experiment run * Your administrator defines a property definition called `note` with the datatype `rich text`. * You can add the property `note` to a single experiment run you are allowed to edit and fill out a Markdown value like "This experiment was *really* great!". * Adding a mandatory cost center field to each experiment * Your administrator defines a property definition called `costCenter` with the datatype `integer`. * Your administrator assigns the property definition `costCenter` to all experiment designs and marks it as required. * You need to fill out the cost center for each experiment you want to run. ## Manage Property Definitions Property definitions are the basis for properties in Steadybit. A property definition consists of the following fields: * **Name**: The name of the property definition. This is the name that will be shown in the UI. * **API name**: The key of the property that will be used in the API. This is a unique identifier for the property definition. * **Field Type**: The data type of the property definition. This can be one of the following: * `String`: A text field containing any string value. * `List of Strings`: A list of text fields containing multiple string values. * `Rich Text`: A string field containing Markdown-formatted text. * `Boolean`: A checkbox that can be either true or false. * `Date`: A date field containing a date value. * `Number`: A number field containing an integer value. * `List of Numbers`: A list of number fields containing multiple integer values. * `Enumeration`: A dropdown field containing a predefined list of values. The values can be defined in the property definition. * `List of Enumerations`: A list of dropdown fields containing multiple predefined values. The values can be defined in the property definition. * `Link`: A field containing a URL. This can be used to link to external resources. * `List of Links`: A list of URL fields containing multiple links. * **Property Description**: A description of the property definition. This is optional but recommended to provide additional context for the property. Property definitions can be created and managed in the settings of the Steadybit platform. Administrators can create new property definitions, edit existing ones, and delete them. If an administrator deletes a property definition, all assignments and values of this property definition will be removed from all experiment designs and templates. Values in existing runs will not be deleted. ![Property Definitions in the settings](/files/CpkewtkS0dZNYDR0U8Hg) ![Edit a Property Definition](/files/To7oDhlgn2tiYach6XFX) Properties can also be managed and assigned using the following API endpoint. Check out [Integrate with Steadybit / API](/integrate-with-steadybit/api) how to access the API. * [Manage property definitions](https://platform.steadybit.com/api/swagger/swagger-ui/index.html?configUrl=/api/spec/swagger-config#/Properties) ## Assign Properties An administrator can assign property definitions to each experiment design. He can also mark a property as required, which means that the property must be filled in before the experiment can be run. This allows administrators to enforce certain properties for all experiments in the tenant. The assignment also defines whether the property is editable after the run has been started or if it should be read-only. This can be done in the settings section of the Steadybit platform. ### Property Assignment Configuration A property assignment configuration consists of the following fields: #### Property Definition API name The referenced property definition assigned to the experiment design or run. For the UI, this is automatically given due to the context. For the API, you have to explicitly specify the property's API name. #### Mandatory Define a property's value to be required before being able to run the experiment. #### Allow editing the value in the experiment run Decide whether the property's value can be edited at run-level or only at design level. By default, this is design-only and thus, the property's value can only be edited at the experiment design level. If activated, the property can also be edited for each experiment run individually at the run details page. ![Property Assignments in the settings](/files/lUuXuyTpd3KinxyF54Pz) ![Edit a Property Assignment](/files/rDlXNfnxbc1Rh0u38Iu1) ### API Properties can also be managed and assigned using the following API endpoint. Check out [Integrate with Steadybit / API](/integrate-with-steadybit/api) how to access the API. * [Manage property assignments](https://platform.steadybit.com/api/swagger/swagger-ui/index.html?configUrl=/api/spec/swagger-config#/Properties) ### Assign Properties Individually Additionally, users can assign properties to a single experiment design or run, if they are a member of the team. More details can be found in the [experiment's section of property assignments](/use-steadybit/experiments/properties#assign-properties). ## Properties in Experiments Properties can be used in experiment designs and runs: * A **property definition** can be assigned to an experiment design or run (if not assigned globally) * An **assigned property** can save values per experiment design or run Learn more in the [experiment's section of properties](/use-steadybit/experiments/properties). # Services Services bring everything a team needs for reliability into one place. Instead of managing scattered experiments, validations, and reports separately, a service gives you a single view of a specific application or system component — covering which targets belong to it, how to validate its health, and experiments evaluating its reliability. ![Services overview](/files/1Z7oenv7YhrMKh6K8HmE) ## What Makes Up a Service A service is defined by three core elements: * **Target Scope** — the environment and query that resolve the targets belonging to this service (e.g., all Kubernetes resources of a Kubernetes service, a set of Virtual Machines, or cloud managed services) * **Validations** — checks or load tests (HTTP, Datadog, Dynatrace, Prometheus, k6, and more) that define what 'healthy' means for this service while an experiment runs. Refer to our [Reliability Hub](https://hub.steadybit.com/?kind=CHECK%2CLOAD_TEST) for a complete list of available actions * **Properties** — optional custom metadata to categorize and describe a service (e.g. business criticality, service owner) A service always fulfills a **service profile**, which defines the set of experiment templates that the service is expected to fulfill for reliability. [Learn more how to manage and set up a custom service profile](/install-and-configure/manage-service-profiles). Based on how well and up-to-date those have been validated, Steadybit derives a service's reliability [risk](#risk) — a single indicator that summarizes the service's current reliability posture and makes it easy to compare services with each other. ## Service Detail Once a service is set up, its detail view gives you a tab per profile's category (e.g., Scalability, Redundancy, Dependencies). Followed by distinguishing between *Provided Experiments*, *Custom Experiments*, and *Advice* (see below). The header bar always shows you a quick summary of the service — its used environment, the number of resolved targets, active validations, any service properties, and the associated [risk](#risk). ### Provided Experiments Provided experiments are automatically generated from the linked service profile. Each experiment template in the profile is instantiated with your service's targets and validations, giving you ready-to-run experiments without manual setup. ![Service's provided experiments](/files/vQFEgLVijL5HnMjHfX6Y) Experiments are grouped by the categories defined in the service profile (e.g., Scalability, Redundancy, Dependencies). Each card shows the experiment name, its last run status, and a quick-run button. You can filter the list by category using the filter pills at the top. The generated experiments use your service's targets and validations directly — for example, the **Pod Redundancy** experiment is created as `SHOP-3073 Pod Redundancy` and scoped to the exact pods in your service's target scope. ![Service's provided experiment for pod redundancy](/files/m7kzanGh3QsqWetj94pH) ### Custom Experiments Beyond provided experiments, you can link any existing experiment to the service to keep all relevant reliability work in one place. ![Service's custom experiments](/files/KPJJ4MVTGO890k3UhFbp) Click **Link Experiments** to associate one or more existing experiments with the service. Linked experiments appear alongside the provided experiments, giving your team a complete picture of all reliability validation for this service. You can use the `Service Validation` step inside a custom experiment, to reuse service's validations or use other actions (e.g. checks, or load tests) to validate your infrastructure's behavior. If you are unsure what else is worth testing for this service, use **Get suggestions** in the SteadyBuddy banner to let [SteadyBuddy propose experiments](/use-steadybit/steadybuddy#get-experiment-suggestions) based on the service's targets. ### Advice The **advice** tab surfaces reliability recommendations for the targets within your service's scope. Each advice item is categorized (e.g., Scalability, Redundancy) and indicates its current state — making it easy to identify gaps and prioritize improvements. ![Service's advice](/files/K3uI6Etvdp6WEwn8rvnm) Advice is generated based on the discovered targets and [installed advice-supported extensions](https://hub.steadybit.com/extensions?tags=Advice) in your service's scope. [Learn more about advice](/use-steadybit/explorer/advice). ## Risk Every service has a **risk** associated to give you a quick indicator of its reliability posture. The risk is calculated per category defined in the service profile (e.g., Scalability, Redundancy, Dependencies), and the overall service risk is a rollup across those categories. You can see both the overall risk and the per-category breakdown in the header of the service detail page, and open **How is the risk calculated?** for a detailed explanation in the product. The higher the risk, the higher the risk value: 100 indicates the highest risk, where as 10 is the lowest achievable risk. The risk never reaches zero, because reliability is a continuous effort: the service always needs ongoing validation to stay trustworthy. ### How to Reduce the Risk The following factors reduce the risk of a service: * **Successful experiment runs** — each linked experiment (both [provided](#provided-experiments) and [custom](#custom-experiments)) that finishes with state `COMPLETED` lowers the risk. Failed, errored, or never-executed experiments keep the risk high. * **Recent runs** — the risk reflects how recently experiments were executed. Re-run your experiments at least every 30 days; the older the last run, the more the risk drifts back up. * **Strong validation coverage** — service validations, and additional checks and load tests inside your experiments increase confidence. * **Resolved advice** — working through items on the [Advice](#advice) tab reduces the risk of the affected category. An advice requiring action is associated with a high risk (`100`), whereas a required validation is a medium risk (`50`), and an implement advice results in low risk (`10`). The product surfaces inline guidance next to each experiment and category, showing you the most impactful next steps to lower the risk. ![Suggestions how to lower the risk](/files/pdQ2KHIrcnzUhPKAub5V) ### Comparing Services The risk is also surfaced outside the service detail page so you can spot where to invest next: * In the **Services** overview, each service shows its current risk — making it easy to compare the reliability posture of different services at a glance. * On each team's **Dashboard**, the top-risk services are highlighted so owners can prioritize their reliability work. ![Dashboard featuring the service with highest risk](/files/bDam6oXaMikvYH9urdby) ## Exploring Services You can explore the targets of a service directly from two entry points: * **Services overview** — click **Explore Services** to open the Explorer with all targets across all services. * **Service detail** — click **Explore Targets** in the top-right corner to open the Explorer scoped to the targets of that specific service. In the Explorer, you can group and filter targets by any attribute — for example, by Kubernetes labels, AWS zone, or deployment — to understand how your service's infrastructure is distributed and identify potential reliability gaps. Every target that belongs to a service is enriched with two additional attributes: * `service.id` — the unique identifier of the service * `service.name` — the name of the service These attributes are available throughout Steadybit: use them in the Explorer to filter or group by service, and in experiment design to target specific services or reference them in queries. [Learn more about Explorer's capabilities](/use-steadybit/explorer). ![Exploring services](/files/xUeLouYbzP6p65pI7571) ## Managing Services Create a new service via **Services** → **New Service** and configure it through four tabs: *Target Scope*, *Validations*, *Properties*, and *Customize*. A service can be created and edited by administrator or team owners. A team member can instantiate and run experiments, link custom experiment or validate advice. ### Target Scope The target scope determines which infrastructure targets belong to this service. You select an [environment](/install-and-configure/manage-environments) to restrict available infrastructure components, and then use a **query** (either Query UI or [Query Language](/concepts/query-language)) to filter targets precisely. ![Editing service's target scope](/files/q4DQFVB9CP8QOcW4Jt4F) The right panel shows all included targets matching your query, grouped by target type (e.g., Containers, Hosts, or Kubernetes resources). This gives you immediate feedback on which infrastructure components are in scope before saving. Target scope will be used when configuring validations (see next chapter), running experiments and can be explored via [explorer](/use-steadybit/explorer). ### Validations Validations define whether your service is healthy during an experiment. They are used as the steady-state checks in provided experiments. ![Editing service's validations](/files/sS4UNnoNvI4dyQuhJJd2) Click **Add Validation** to pick from available actions of type `check` or `load test`. Common validations include: * HTTP checks * Observability monitors (e.g. Datadog, Dynatrace, Prometheus metrics) * Load tests (e.g. k6, JMeter, Gatling) * Any other validation tools (e.g. Cypress UI Tests) via Jenkins jobs Refer to our [Reliability Hub](https://hub.steadybit.com/?kind=CHECK%2CLOAD_TEST) for a complete list of available actions. {% hint style="info" %} Validations that rely on targets are evaluated against the service's [target scope](#target-scope). Make sure the targets your validations need are included in the target query. For example, add `OR datadog.monitor.tags="env:stage"` when using Datadog Monitors, or `OR (target.type="com.steadybit.extension_http.client-location" AND k8s.cluster-name="stage")` when using [HTTP client locations](https://github.com/steadybit/extension-http#location-selection). {% endhint %} ### Variables Service variables can be used to abstract and align common properties throughout all experiments of the service. In this functionality they extend or override [variables of the associated environment](/install-and-configure/manage-environments#environment-variables), and can themselves be overridden by experiment variables or per-run overrides. They apply to both the experiments provided by the service and custom experiments linked to it, and can also be referenced inside this service's [validations](#validations). Like other variables, a service variable can have [different value configurations](/use-steadybit/experiments/variables#value-settings). For a dynamic value, you also choose its [evaluation scope](/use-steadybit/experiments/variables#evaluation-scope) — whether it is sampled from the service's own [target scope](#target-scope) (the default) or from the whole environment the service lives in. You can [learn more in the variable section](/use-steadybit/experiments/variables#service). ![Editing service's variables](/files/ehgUdUqrDkSo8kBbhLrI) ### Properties Custom properties let you attach metadata to a service — for example, service ownership, business criticality, or links to runbooks. Properties are defined globally and can be managed in [Manage Properties](/install-and-configure/manage-properties#manage-property-definitions). You can associate properties [to all services via settings](/install-and-configure/manage-properties#assign-properties) or to [individual services](/install-and-configure/manage-properties#assign-properties-individually). ![Editing service's properties](/files/T1Z8o9yE8joYeR5MOJYc) ### Customize Additionally, you can change the look and feel of your service by customizing the used icon and icon color. More importantly, you can change the service profile of a service - defining a different set of experiment templates used to provide experiments. Thus, adhering to different reliability expectations. {% hint style="warning" %} Be aware, that changing a service's service profile results in deleting provided experiments and experiment runs that aren't part of the newly associated profile anymore. {% endhint %} ![Editing service's profile and look\&feel](/files/JVtrOIoWxno5ZYeeNb7N) ## Service Profiles Every service is linked to a service profile that defines which experiment categories and templates are provided. Steadybit ships with a default Starter profiles — a focused set of experiments to get started with reliability testing quickly. Administrators can create custom service profiles tailored to organizational standards, which is highly recommended when rolling out Steadybit. See [Manage Service Profiles](/install-and-configure/manage-service-profiles) for details. # Explorer Steadybit's Explorer shows you an overview of all targets discovered by extensions. It is perfect to check whether Steadybit has discovered all infrastructure components and where to continue your Chaos Engineering journey. You can choose between viewing the targets on a table or exploring extensively using the Landscape. * [Landscape](/use-steadybit/explorer/landscape) * [Targets Table](/use-steadybit/explorer/targets) * [Advice](/use-steadybit/explorer/advice) # Landscape The Landscape informs your Chaos Engineering journey by being able to analyze the discovered infrastructure component in more detail. You can easily navigate in potential targets and, e.g., search for new experiments that can be used to approach a team or verify whether other components may be affected by a revealed reliability issue. ## Drill-Down Targets When using the Landscape, you have the following capabilities at your hand to better understand discovered targets * **Environment** to define the upper amount of targets that you would like to explore * **Filter targets via Query** to include targets you are interested in and exclude all targets that do not match * **Group by** to group targets by an attribute value and bring them next to each other. You can even do endless subgroupings. * **Size by** to analyze the target's attributes that have multiple different values assigned * **Color by** to highlight and differentiate attribute's values ![Explorer Landscape to analyze your system](/files/Sf5co2KaWvDGsNbUoeSS) For every target or group of targets, you can view the discovered target types and attributes. ![Landscape to show targets grouped by type and target's attributes](/files/UQACIixRIZSZiyo2hyyR) ### Attribute Configuration For 'grouping by' an attribute, you can additionally configure: ![Explorer Landscape Attribute Configuration](/files/V2QUvQ41UHbjOUfC0uLm) * **Show "unknown" group** whether a target that doesn't have a value for the chosen attribute (i.e., `k8s.deployment`) should be added to the `unknown` group. If deactivated, the target will be hidden. Otherwise, you see an additional `unknown` group. * **Assign unmapped values to "unknown"** whether a target's attribute value that isn't assigned to a bucket (see below) is added to the above-explained "unknown" group as well, or handled as separate groups. * **Buckets** allow you to group multiple target attribute values into one. For instance, grouping deployments named `hot-deals`, `fashion-bestseller`, and `toys-bestseller` into the bucket `products` will be shown in the grouping as if all three deployments are named `products` Buckets are also available in the attribute configuration of 'color by'. ## Advice Once you activate the 'Show Advice' in the sidebar, the targets are colored depending on the worst advice state (see [Advice Lifecycle](/use-steadybit/explorer/advice#advice-lifecycle)). You can see more details for each target by opening up the target sidebar or target details. ![Explorer Landscape showing you targets and advice](/files/Aro7MoOHZ8OS3r2X1SoY) Learn more about [advice](/use-steadybit/explorer/advice) in the corresponding section. ## Create Experiments Once you have identified a relevant group of targets for an experiment, you can click on that group and choose to use the target selection for an experiment. The Landscape provides the exact Query so that you can copy it into an experiment design. ![Creating an experiment from the Landscape](/files/hUrGle998ComSdBdaDXW) ## Saved Views You can save all views you have created with the Landscape to share them automatically with your team members. In addition, you can always share a view with someone else using a deep link to that particular view. Whenever you have made changes to a saved view, the Landscape will ask you whether you want to keep them or you can save them as a new view. In addition, Steadybit shares some predefined views that will help you to get started with the Landscape. They are dependent on the actual discovered targets, showing e.g. a Kubernetes map only when at least one Kubernetes cluster is discovered. ![Landscape views](/files/aC7G0GvvZlqx7SkxgnF9) # Targets ## Table View The explorer target table gives you an overview of all targets to check, e.g., for completeness of targets or whether a newly installed extension discovers the appropriate target type. You can always see the full details of a target by clicking on it. Depending on your team's permission, you can see the environment Global to see all discovered targets or choose a specific environment to list targets according to your permissions. ![Explorer targets listing all target types and discovered targets](/files/nq6QczKhBSOnxDr6E65M) ![Target details showing all discovered attributes](/files/mZH58OZjFJLn0Sm8OnDD) ## Advice Once you activate the 'Show Advice' in the sidebar, the general advice icon indicates target types having advice. You can see the advice per target in the table and details about each advice in the target details. ![Explorer Targets Table showing you targets and advice](/files/vIXqCkLSBxnSKnmJvWE9) Learn more about [advice](/use-steadybit/explorer/advice) in the corresponding section. # Advice Steadybit helps you on your Chaos Engineering journey to a reliable system by providing best practices applied to your system as advice. Advice checks whether your infrastructure configuration is appropriate and, if so, suggests, e.g., Chaos Engineering experiments to validate that your reliability mechanisms actually work. In addition, advice is fully extensible to allow you to add more best practices or customize existing ones. ## Advice Lifecycle Advice apply to a single target of a particular target type and is in one of the following states: * ![Advice icon for state 'Action Needed'](/files/jHguChyGMqeunc39Ku0d) Action Needed * ![Advice icon for state 'Validation Needed'](/files/AtyHgNhljwVZPtUMbeo1) Validation Needed * ![Advice icon for state 'Implemented'](/files/dLaOxRQlzaa2nzfsdO19) Implemented Advice may not support and skip a particular state depending on its definition. ### 1. Action Needed The advice has automatically discovered that your target's configuration requires changes to follow best practices. For instance, advising you to address redundancy configuration for Kubernetes deployments or deploying your application in multiple cloud vendor's availability zones. The advice summarizes the matter, gives additional motivation to explain why you should address it, and gives specific instructions to improve it. ![Advice for target 'fashion-bestseller' in state 'Action Needed'](/files/4kabVwQxLgwGEmUBdPon) As soon as you implement the suggested instruction, Steadybit automatically discovers the change and - depending on the advice definition - asks you to validate your changes (lifecycle 'Validation Needed') or marks the advice as 'Implemented'. ### 2. Validation Needed The advice has discovered that your target's configuration follows best practices but recommends to validate that they are actually working. Validation can't be performed automatically by Steadybit and always requires human interaction. Supported validations are: * Experiments: The advice suggests an experiment you should run to validate whether the target's configuration is working. * Text: The advice gives additional context for manually checking whether the target's configuration works. ![Advice for target 'toys-bestseller' in state 'Validation Needed'](/files/TrKFzKiCYWzUcdlFi1Up) To perform the validation, we suggest you do the corresponding action (run an experiment or follow the text-based instruction) and mark the validation as done. When you've performed all validations, the advice's states go into the end state ('Implemented'). ### 3. Implemented Great job! You followed the advice and have fulfilled all required actions and validations. ![Advice for target 'gateway' in state 'Implemented'](/files/ju1yN43CxSRh9GblEkJJ) In case you invalidate one of the validations or the discovery detects a required configuration change, the advice will return to the 'Validation Needed' or 'Action Needed' state. ## Following Advice You can see given pieces of advice in the Steadybit Explorer. The Explorer Landscape and Explorer Targets show you advice in a target-first manner, showing all advice for a given target. The Explorer Advice lists all available advice and its applicable targets. ### Landscape Once you activate the 'Show Advice' in the sidebar, the targets are colored depending on the worst advice state (see [Advice Lifecycle](#advice-lifecycle)). You can see more details for each target by opening up the target sidebar or target details. ![Explorer Landscape showing you targets and advice](/files/Aro7MoOHZ8OS3r2X1SoY) ### Targets Table Once you activate the 'Show Advice' in the sidebar, the general advice icon indicates target types having advice. You can see the advice per target in the table and details about each advice in the target details. ![Explorer Targets Table showing you targets and advice](/files/vIXqCkLSBxnSKnmJvWE9) ### Advice In the Explorer Advice, you can see all supported advice, followed by the applicable targets and the respective status. It's the perfect place if you want to resolve the same advice for multiple targets. You can also filter based on advice's status to concentrate, e.g., first on targets requiring action for a specific advice. ![Explorer Advice showing you advice and targets](/files/iP1fKk31tuDzi89qhibo) ## Query Language Steadybit's [Query Language](/concepts/query-language) fully supports advice to guide you on the Chaos Engineering journey. You can, e.g., filter targets in the Landscape and Experiments, or size targets in the Landscape by using the following attributes: * `advice.applicable` provides a set of advice definition ids that apply to the target, no matter in which lifecycle state. * `advice.status.action-needed` contains for a given target all advice in the lifecycle state 'Action Needed' * `advice.status.validation-needed` contains for a given target all advice in the lifecycle state 'Validation Needed' * `advice.status.implemented` contains for a given target all advice in the lifecycle state 'Implemented' Based on these attributes you can build queries like: #### Example 1: Show all targets that need action for a specific advice ``` advice.status.action-needed ="com.steadybit.extension_kubernetes.advice.single-aws-zone" ``` #### Example 2: Show all targets that require at least one validation for any advice ``` count(advice.status.validation-needed) > 0 ``` #### Example 3: Show all targets where advice is applicable ``` advice.applicable="com.steadybit.extension_kubernetes.advice.k8s-host-podantiaffinity" ``` ## Supported Advice We support various advice for, e.g., Kubernetes or public clouds and are constantly adding more. Check out the [Reliability Hub / Advice](https://hub.steadybit.com/advice) for our current support. ## Install Advice Advice is discovered and provided by extensions. Therefore, to use Advice inside Steadybit, you have to install an extension that supports Advice. You can find a list of extensions supporting Advice [in the Reliability Hub](https://hub.steadybit.com/extensions?tags=Advice). ## Extending Advice Advice capability is developed using the open-source [AdviceKit](https://github.com/steadybit/advice-kit), allowing you to contribute your advice or fork existing ones. You can also always check out the code of existing advice in the [Reliability Hub](https://hub.steadybit.com/advice) and its corresponding GitHub repositories. # Experiments An experiment describes a chaos engineering scenario to validate your application's reliability. You can add attacks, like network latency and pod crashes, to inject turbulent conditions and checks or load tests to validate system's state. [Experiments are designed](/use-steadybit/experiments/design) using the drag'n drop experiment editor or easing creation via [experiment templates](/use-steadybit/experiments/design#from-template) or benefitting from [provided experiments in services](/use-steadybit/services#provided-experiments). Once designed, you can [run the experiment](/use-steadybit/experiments/run) directly and see everything in action or use the [scheduler](/use-steadybit/experiments/schedule), [API](/integrate-with-steadybit/api) or [CLI](/integrate-with-steadybit/cli) to run the experiment later on. * [Design](/use-steadybit/experiments/design) * [Run](/use-steadybit/experiments/run) * [Run History](/use-steadybit/experiments/run-history) * [Schedule](/use-steadybit/experiments/schedule) * [Variables](/use-steadybit/experiments/variables) * [Properties](/use-steadybit/experiments/properties) * [Emergency Stop](/use-steadybit/experiments/emergencystop) * [Share](https://github.com/steadybit/docs-public/blob/main/use-steadybit/experiments/share.md) * [OpenTelemetry Integration](/use-steadybit/experiments/opentelemetry-integration) # Design To start and eventually run a Chaos Engineering experiment, you first need to design it. That includes defining, e.g., where to run it (environment), what actions to perform (attacks, checks, load tests), and what should be attacked (targets). ## Basic Elements An experiment generally consists of the following elements: * **Name**: Giving your experiment a meaningful name makes it easier to find it again. You can change it at any time. * **Team**: Each experiment is associated with a team. If you are a member of multiple teams, you can see and change your current team in the left-hand navigation sidebar. Once created, you can't change the experiment's team. * **Environment**: An experiment always runs in one specific environment of your system landscape. This environment spans a set of targets you want to address in an experiment, such as containers and JVM applications in a Development stage. How the [environments are configured and assigned to your team](/install-and-configure/manage-environments) is up to your admin. * **Variables**: A variable allows you to reference a value in your experiment's configuration to have a single source of truth. Steadybit supports different scopes (e.g., experiment variables or environment variables). Learn more in the [variables section](/use-steadybit/experiments/variables). * **Hypothesis**: The hypothesis should answer the question of the expected outcome. In addition, you can describe the steady state, the turbulent condition, and the expected behavior. See this example of a hypothesis: 'When requests to the recommendation service exceed 1000ms, the catalog responds using an empty recommendation list.' * **Tags**: Besides the hypothesis, tags can be applied to an experiment as useful metadata. Tags are not only listed in the experiments overview, but also you can use them to search for specific experiments. In the experiment list, the top 10 used tags are listed right below the searchbar (where you find the full list of used tags), so you have a quick access. * **Actions**: Performed when running the experiment, sequentially or in parallel. An action can be either * an **attack** to inject a fault into your system * a **check** to validate your expectation in your system or observability tooling * a **load test** to inject load into your system and validate functionality. The team influences the experiment in the following aspects: 1. **Short Handle:** Each experiment gets a unique key with a team prefix (e.g., `SRE-23`), which is used, e.g., to trigger an experiment via the [API](/integrate-with-steadybit/api). 2. **Run/Edit Permissions:** Only team members are allowed to edit or run an experiment. 3. **Environment and Action Permissions:** A team's permission determines which environments (a set of targets or infrastructure components you can attack) and actions are eligible for an experiment. ## Create New Experiment In general, you have three possibilities to get started: ![Approaches to create an experiment](/files/1vMyZN5ujvGc6W0ZpfMf) * **From scratch**: in case you have already an experiment in mind * **From template**: benefit from existing templates and apply them easily in a wizard-style to your context * **Upload** a YAML- or JSON-based file containing an exported experiment ### From Scratch When creating an experiment from scratch, you land directly into our timeline-based experiment editor. If your team can access only one environment, it is selected automatically. Otherwise, you have to define which environment you want to experiment in. In addition, you can specify the experiment's name and hypothesis at any point you like. After that, you can add all actions (attacks, checks, and load tests) to your experiment by dragging and dropping. When you place actions in the same horizontal lane, they will be performed sequentially, putting them below each other results in a parallel run. Choose actions with the desired effect to perform your overall experiment. ![Create Experiment - Drag'n Drop Editor](/files/v73T0sM8TOdVCQlJSAHz) Once you've dropped the action or selected it later, you can define its configuration on the right-hand side. #### Revealing Step's Intention When designing and running experiments as a team, it is a best practice to reveal every step's intention by renaming it. Click on a step's label in the sidebar to rename and help others understand it easily. ![Create Experiment - Rename Steps in the Editor](/files/sc1D8HiIDQP5FUgxayaW) #### Target Selection ![Create Experiment - Target Selection in the Editor](/files/90ga4GamgDynazxK3M3V) You can now select your desired targets of your environment via a target query referencing [discovered attributes](/concepts/discovery). Since these attribute values are always discovered live, some can change from one moment to the next. So, it is wise to choose stable attributes. Good examples are labels, namespaces, or symbolic names - whereas a unique identifier of targets (like the container id) is usually a bad idea. When you run the experiment, these attributes are resolved into a concrete set of targets under attack. You can preview matching targets using the 'show targets' button next to the query. #### Limiting Targets via Blast Radius Furthermore, you can limit your targets randomly to only attack a subset of your target selection and avoid attacking all at once. You can choose between the basic blast radius, a simple randomizer across all target selections, and the advanced blast radius, allowing you to group your targets on a given attribute hierarchically. **Basic Blast Radius** The basic blast radius simply randomizes all selected targets and limits the subset by a percentage or a fixed number. The total number of attacked targets is shown at the top of the sidebar. ![Create Experiment - Limiting Targets in the Editor via Basic Blast Radius](/files/D5dfVLoDLED7ScfmvFrv) **Advanced Blast Radius** The advanced blast radius allows you to hierarchically group your targets on a given attribute and attack only a subset of the groups. This is perfect whenever you need to add context to the randomization to express a more realistic blast radius. Some common examples for this are * attacking a random Kubernetes deployment in a cluster * attacking a random Kubernetes pod of a specific deployment * attacking one random service's components, but all in a random availability zone The screenshot below shows an example where we group containers first on the label `k8s.label.domain`, and afterwards - within each group - on the available zone. We will select 50% of the service-groups and one availability zone. ![Create Experiment - Limiting Targets in the Editor via Advanced Blast Radius](/files/c6KkhoRVfhieiQ5G4WoP) So, in this example, we may attack all containers in, e.g., 1. `k8s.label.domain` = `shop-inventory`, `shop-gateway`, `shop-products` and `aws.zone` = `eu-central-1a` *or* 2. `k8s.label.domain` = `shop-inventory`, `shop-gateway`, `shop-products` and `aws.zone` = `us-east-1a` *or* 3. `k8s.label.domain` = `shop-checkout`, `shop-orders`, `shop-products` and `aws.zone` = `eu-central-1b` *or* 4. ... This allows to simulate an unavailability of a random zone across random services (i.e. case 1, `eu-central-1a`), but results in some container's domain unaffected (i.e. case 1, `k8s.label.domain` = `shop-inventory`, as no containers are running in `eu-central-1a`). The actual picked values are shown in the [run modal](/use-steadybit/experiments/run#advanced-blast-radius). {% hint style="info" %} A percentage limit can easily result in no targets while running the experiment. This can happen when specifying a low percentage or having a low number of targets in a group. Steadybit rounds mathematically, so, for example, `20% of 2 targets` results in `0 targets`, and the experiment run to stop. This is not validated when designing the experiment in case of the advanced blast radius. {% endhint %} In case you use a variable, template placeholder, or the API, you need to define the unit of the radius limit, like `50%` for percentage and `1#` for an absolute number. #### Action and Attack Settings Depending on the action chosen, you can configure different settings for the step. For instance, an attack to 'fill memory' allows to configure the amount of memory to be filled, an attack to 'stop container' allows to configure whether it is a graceful termination, and an 'HTTP check' allows to configure the expected success rate. ![Action Settings Example of 'Stress CPU'](/files/a42GiHM7tDcegVBsYZuY) **Continue on Any Failures / Errors** By default, a failed/errored step causes the entire experiment run to fail/error immediately. All steps have a configuration to continue an experiment run even when a failure or error occurred in the step's execution. Once activated, the step is still marked as a failure/error, but the experiment continues running and may still eventually end in `COMPLETED`. This is further described in the [experiment run state documentation](/use-steadybit/experiments/run#state-propagation). Continue with these steps until you've designed your experiment. A reasonable experiment could easily look like the one below. Once you have saved it, you are ready to [run it](/use-steadybit/experiments#run) to learn how your system behaves. ![Create Experiment - Example in the Editor](/files/fr9pndb1wsgn4ofENDJj) ### From Template A second option for creating a new experiment is using an existing template. This approach is best when you want to learn from others' experiences and apply them to your context via a step-by-step wizard. You can simply browse all available templates or use our search feature to find one using the template's **tags**, **targets** or **actions** used in the experiment, or free text search. ![Create Experiment - Browse and Search Templates](/files/klmo5ROXjfFa96oNb2JG) Once you've decided to use an experiment template, you are guided step-by-step through applying it to your context. The first step asks you to select the environment where you eventually want to run your experiment. If the experiment template references any [variables](/use-steadybit/experiments/variables) (via the `{{variable}}` syntax), they are also listed so you can decide how each one should be resolved: * **Provide a value** — the variable is created as an [environment variable](/use-steadybit/experiments/variables#environment) on the selected environment and shared with all experiments using that environment. You only need to provide values for variables that don't exist on the environment yet; existing ones are reused automatically. * **Leave the value empty** — the variable falls back to an [experiment variable](/use-steadybit/experiments/variables#experiment). It then shows up in the created experiment, where you provide its value in the experiment variable's dropdown in the editor's subheader. If you create the experiment within a [service](/use-steadybit/services), any variable already defined as a [service variable](/use-steadybit/experiments/variables#service) is resolved from the service, so you don't need to specify it as an environment or experiment variable. ![Create Experiment - Use Template: Environment](/files/JZuO4QU6FWknU7qGnHK5) The subsequent steps depend on the selected experiment template and will guide you step-by-step to a ready-to-be-executed experiment. Each step consists of a specific question for a value (i.e., a Kubernetes deployment) to adapt the template's experiment to your context. ![Create Experiment - Use Template: Further Steps](/files/JYwK91ZvJ5PSKpjdVJy7) Eventually, you end up in the experiment editor, where you can adjust the experiment or [run it](/use-steadybit/experiments#run) to learn how your system behaves. ![Create Experiment - Use Template: Further Steps](/files/cWxhfTG0rweMWmPjhwsF) ### Via File Upload The third option for creating a new experiment is to upload a YAML- or JSON-based experiment file. Once you've uploaded the file, the experiment is created in the defined team. Please note that when the experiment file references an explicit team and environment (e.g., `team: "ADM"` or `environment: "Online shop"`), they have to exist. If you want to be flexible, you can use the variable `{{teamKey}}` to apply it to the current team and `{{environmentName}}` to apply it to the team's first environment. ## Action and Template Documentation Within the experiment editor you can read up the full documentation of an action or a template by clicking on the individual step. This helps to understand the exact details, read up use cases and a full documentation of the parameters. ![Experiment Editor - Action Documentation](/files/2R5dKD3dsJmudQWFeMYK) The documentation of a template is coming from the [experiment template's description](/install-and-configure/manage-experiment-templates#template-description). The documentation of an action is provided via the [Hub Connection](/integrate-with-steadybit/hubs), and can also be achieved for your own custom actions. ## Supported Actions Check out our [Reliability Hub](https://hub.steadybit.com/actions) to learn about actions you can leverage with Steadybit or learn more about the [concept of an action](/concepts/actions). # Run After having your [experiment fully designed](/use-steadybit/experiments) you can simply use the `Run`-button to execute it. This action can be performed if all the following conditions are met: 1. No validation errors 2. Every attack resolves at that moment to at least one target. 3. You are member of the same team as the experiment 4. [Emergency stop](/use-steadybit/experiments#Emergency-Stop) has not been triggered. Otherwise, you'll get an error message and the experiment is not started. ## Run View As soon as the experiment starts, the platform automatically switches over to the run view. The first step of the platform is to establish the connection to the matching agents. In addition, the running experiment is indicated at the top right run icon. ![Experiment Run View](/files/ELHJsDJdmvWQyYEnjFEK) Every experiment run has a unique identifier (e.g. **#33131**), which you can use to identify older experiment runs (visible on the left side). The run view itself consists of the following elements. * **Run Timeline**: At the top you see the sequence defined previously in the [design](/use-steadybit/experiments#design). While the experiment is running a special marker indicates the current point of time. Some attacks need a little bit of extra time before being started which is indicated by a light green colouring in the front. The extra time is added to the timing of the attack and is currently caused by technical reasons .You can click on each step to get more details in the [run modal](#run-modal). * **Run Status**: The run log lists the currently active or already performed experiment's steps. You can click on each step to get more details in the [run modal](#run-modal). * **Deployment Replica Count**: When using an experiment in a Kubernetes context we will automatically monitor how many PODs are ready in your cluster and indicate whenever there is a discrepancy. * **Kubernetes Event Log**: When using an experiment in a Kubernetes context we provide you access to the Kubernetes Events so that you can identify what exactly happens in the Kubernetes cluster. * **HTTP Call**: If your experiment contains a `HTTP Call`-action you can see the response time as well as HTTP response status as a separate widget in the run window. * **Monitoring Events**: In case your admin has installed a monitoring extension to Steadybit ( see [monitoring extensions in Reliability Hub](https://hub.steadybit.com/extensions?tags=Monitoring)) you can see occurring events and alerts of your setup directly in the run view. ### Run Modal You can click on each experiment step in the 'run timeline'- or 'run status'-widgets to get more details. The modal allow you to see the actions' configuration, the exact timing and affected targets (e.g., containers attacked by the blackhole-attack). ![Experiment Run Modal - Attacked Targets](/files/6HraDbvpwCfRcFy02wvG) The run modal also details more information whenever a step - and thus the experiment - has errored or failed. ![Experiment Run Modal - Failed Step](/files/o0AYGoJtjs6L8mOPQTMl) #### Advanced Blast Radius Whenever you have used the [advanced blast radius](/use-steadybit/experiments/design#limiting-targets-via-blast-radius), you can see the picked values at the top of the run modal. ![Experiment Run Modal - Advanced Blast Radius](/files/KTviqUMhnsbQZ56GfR8t) ## States An experiment run supports different level of states: * [**Experiment run**](#experiment-run) indicating an overall state of the run *(highest level)* * [**Step**](#step) each individual step performed in an experiment run * [**Target execution**](#target-execution) each individual target affected in a step *(lowest level)* The state from a lower level is propagated to the upper level, as described [below](#state-propagation). ### Experiment Run Experiment runs is in one of the following states: | State | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | REQUESTED | The experiment was requested by a user, api call or a schedule. | | CREATED | The experiment was created and all targets were resolved. | | PREPARED | The experiment is prepared, all preflight checks were successful, and agents are ready to execute the needed actions. | | RUNNING | The experiment is currently running and performing actions (e.g. attacks). | | COMPLETED | Entire experiment including all actions were successfully executed - no failure or error reported by any step. | | CANCELED | The experiment was canceled by user interaction or system (in case of a failed validation in `REQUESTED` or `CREATED`) and all attacks were rolled back. | | FAILED | The run failed due to some failing checks, for example a *HTTP Check* not reaching the required success rate. | | ERRORED | The run errored due to some technical reasons like `I/O error on POST request: Connection refused` or `Agent disconnected unexpectedly`. This shouldn't happen frequently, in case it does, let us know. We are constantly improving the platform to reduce error states. | In case an agent looses the connection to the platform during an experiment, it will immediately stop and rollback running attacks. There are some attacks (like `Stop Container`) which can't be rolled back due to it's nature. ### Step Every step that is executed as part of an experiment run is in one of the following states: | State | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | CREATED | All targets of the step have been resolved. | | PREPARED | Agents for all resolved targets are connected and ready to execute the step's action as soon as it's the step's turn in the designed experiment timeline. | | RUNNING | The step's action is currently executed on at least one target. Targets may start at different times, so a step becomes `RUNNING` as soon as its first target execution starts. | | COMPLETED | The step's action was successfully executed on all targets - no failure or error. | | CANCELED | The step was running before - i.e., at least one target execution had started - and has now been canceled, either by a user canceling the entire experiment run or by the system (e.g. when another step running in parallel caused the experiment to stop early). | | SKIPPED | The step's action was never executed, because the experiment was stopped before. | | FAILED | The step failed because of a failed check, such as an *HTTP Check* that did not reach the required success rate, or because it had no matching targets anymore. | | ERRORED | The step's action errored due to some technical reasons, such as `I/O error on POST request: Connection refused` or `Agent disconnected unexpectedly`. | ### Target Execution For every step, there are single or multiple target executions. The actual number depends on the number of targets selected in the experiment design and the nature of the step. The target executions are listed as 'selected targets' in the [run modal](#run-modal). Each target execution is in one of the following states: | State | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CREATED | The target was selected to be attacked in an experiment run's step. | | PREPARED | The agent, that discovered the selected target, is connected. | | RUNNING | The action is currently executed on the selected target. | | COMPLETED | The action was successfully executed on the selected target - no failure or error. | | CANCELED | The action was executed before and has now been canceled on the selected target, either by a user canceling the entire experiment run or by the system (e.g. when another target execution of the same step or another step running in parallel caused the experiment to stop early). | | SKIPPED | The action was never executed on the selected target, because the experiment's step was stopped before. | | FAILED | The action performed on the selected target noticed a deviation in the check, such as an *HTTP Check* that did not reach the required success rate. | | ERRORED | The action performed on the selected target errored due to some technical reasons, such as `I/O error on POST request: Connection refused` or `Agent disconnected unexpectedly`. | ### State Propagation The different levels of states (**experiment run**, **step**, **target execution**) can be propagated to higher levels (e.g., state `FAILED` at target execution level is propagated to step being `FAILED` and experiment run being `FAILED`). But also change the states of adjacent instances of the same level (i.e., a `FAILED` step causes subsequent steps to be `SKIPPED`) and level below (i.e., for a `SKIPPED` step all target executions are in the `SKIPPED` state as well). A step and its target executions therefore always agree on whether anything was actually executed: a step is only `CANCELED` if at least one of its target executions was `RUNNING`. If none of them ever started, the step and all of its target executions are `SKIPPED` - even if the action had already been sent to the agents. #### Example: State Propagation The following example shows the state propagation in action: ![Experiment run showing state propagation](/files/9zuLh49jLqh2jLD6ueAn) 1. Within step **(1)**, one target execution fails because the HTTP success rate wasn't achieved (target execution's state `FAILED`). 2. Other target executions of the step **(1)** are canceled by the system (target execution's state `CANCELED`). 3. Due to the failed target execution, step **(1)** is also ending as `FAILED`. 4. Steps running in parallel **(2)** are canceled (step's state `CANCELED`, thus all its target execution states are `CANCELED`). 5. Subsequent steps **(3)** are not performed anymore and are indicated as `SKIPPED`. 6. Steps already performed **(4)** beforehand don't change their state (i.e., `COMPLETED`). 7. The experiment will stop immediately in the state `FAILED` **(5)**. #### Example: Continue on Any Failures / Errors In some cases, state propagation is not desirable. Especially when running an experiment that attacks a large number of targets, single targets becoming unavailable may be expected and shouldn't cause the experiment to stop immediately. For these cases, you can configure a step to [continue on any failures or errors](/use-steadybit/experiments/design#continue-on-any-failures--errors) in the experiment design. Once activated, the above example looks different: ![Experiment run showing state propagation with 'continue on any failures/errors'](/files/M864mUqxLZt22vsJE7jB) 1. Within step **(1)**, one target execution fails because the HTTP success rate wasn't achieved (target execution's state `FAILED`). 2. Other target executions of the step **(1)** continue to run (target execution's state `COMPLETED`). 3. Due to the failed target execution, step **(1)** is still indicated as `FAILED`. 4. Steps running in parallel **(2)** continue to run (step's and target execution's state `COMPLETED`). 5. Subsequent steps **(3)** are performed as usual; if no additional failure or error occurs, the steps' and target executions' states are `COMPLETED`. 6. Steps already performed **(4)** beforehand don't change their state (i.e., `COMPLETED`). 7. The experiment run ended in the state `COMPLETED` **(5)** because the step **(1)** failing was ignored, and all other steps `COMPLETED`. # Run History After you [run an experiment](/use-steadybit/experiments/run), you can see the history of all past runs for it in the left sidebar of the "Runs" pane and navigate to that run by clicking it. ![Experiment Run View with History](/files/Nbbpmq0IAJZa40q9nRK1) You can visit the "Experiments > Experiment Runs" page to get the whole history across multiple teams.\ The filters allow you to select the experiment runs by team, environment, outcome, or time. You can also download a CSV report for that particular selection and use it for further processing. ![Experiment Run History](/files/2BXwmYhFKJpWms6DCjes) # Schedule Experiments can be run on the fly using the UI, API, or CLI or scheduled to run automatically in the future. Scheduling an experiment can be done via [UI](#scheduling-via-ui) or [API](#scheduling-via-api). {% hint style="info" %} To schedule an experiment, one must have permission to edit it. This means one has to be a member of the team. Please note that an experiment schedule always starts the experiment. In case of validation errors, these are documented as canceled experiment runs to check them out asynchronously. {% endhint %} ## Scheduling via UI You can open the scheduling configuration dialog in the platform either in the experiment experiment editor or the experiment list's context menu. ![Schedule experiment within the experiment editor](/files/nzhX9SQl556LjnkAtRul) When the configuration dialog opens, you can manage multiple experiment schedules. Each schedule can run the experiment just once or repeatedly in the future. ### Once Schedule The configuration is straightforward if you decide to run it once. Just select a date and time in the future and save the schedule. Please keep the "Activate Schedule" toggle enabled. Otherwise, the schedule will be ignored and not run. ![Scheduling Experiments just once](/files/RabgkTXHdlQ8RLcaAhGf) {% hint style="info" %} The date and time selections are done in the user’s configured timezone. {% endhint %} ### Recurrent Schedule Experiments can be scheduled to run repeatedly (like every hour or every Friday morning). To do so, you have to configure a cron-like expression. ![Scheduling Experiments repeatedly](/files/p6mMpISfzeHlkRB1Bk1i) {% hint style="info" %} **Just so you know**, we are using the [Quartz cron trigger syntax](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html). The Quartz cron expression is evaluated in the user’s configured timezone. The currently configured timezone will also be shown for the next experiment run. {% endhint %} ### Schedules Overview In the experiment list section on the left-hand side, you can find a quick overview of all configured schedules. Here, you can easily see when the schedules will be triggered next, enable/disable or even delete the schedule. ![Scheduled Experiment overview](/files/kLrXBOEngt6i5EB9Avwi) ### Experiment variable overrides If the experiment is making use of an [environment, service, or experiment variable](/use-steadybit/experiments/variables), you can override them is the schedule. As in the experiment editor, the exeriment is validated when you enter non supported values (e.g. entering "name" into a duration variable). Schedule variables will override environment, service, or experiment variable values for each run, triggered by this schedule. ![Experiment Schedule override](/files/VNESmO4dMWfPgtvp2ZaH) ## Scheduling via API Schedules can also be configured using the following API endpoints. Check out [Integrate with Steadybit / API](/integrate-with-steadybit/api) how to access the API. * [Create or update an experiment schedule](https://platform.steadybit.com/api/swagger/swagger-ui/index.html#/Experiment%20Schedule/upsertSchedule) * [Get an experiment schedule](https://platform.steadybit.com/api/swagger/swagger-ui/index.html#/Experiment%20Schedule/getSchedules) * [Remove an experiment schedule](https://platform.steadybit.com/api/swagger/swagger-ui/index.html#/Experiment%20Schedule/removeExperimentScheduleById) * [List all schedules currently configured](https://platform.steadybit.com/api/swagger/swagger-ui/index.html#/Experiment%20Schedule/getAllSchedulesV2) Each experiment schedule can be identified via a UUID (`id`) which can be used to update or delete a specific schedule. # Variables Steadybit supports using variables in an experiment, so you only need to define specific values once and benefit from a single source of truth. Variables are helpful if multiple steps refer to the same configuration (e.g., in a query to define the target or the HTTP endpoint URL), see the example below, or when having multiple experiments using common infrastructure names in the experiment's definition (e.g., Kubernetes cluster name). ![Variables in an experiment](/files/BXvaEHnj1ZsFAROFOBtu) ## Syntax A variable is referenced via double curly brackets, like this: `{{variable}}`. The variable is resolved within an experiment using different scopes, moving from inner to outer scope (see below). If you want to use a literal string `{{..}}` in your experiment design, you can escape it by using a single backslash like `\{{..}}`. ## Scopes A variable in an experiment design / run supports different scopes to be resolved: | Order | Scope | Defined on / managed in | Time of resolving variable | | --------- | ----------- | ---------------------------------------- | -------------------------------------------- | | outermost | Environment | An environment (Settings → Environments) | When designing and/or running the experiment | | ... | Service | A service (Service editor → Variables) | When designing and/or running the experiment | | ... | Experiment | The experiment itself | When designing and/or running the experiment | | innermost | Run | A single run or an experiment schedule | Only when running the experiment | ### Resolution Strategy When a `{{variable}}` is used, Steadybit resolves it by looking through the scopes from the innermost to the outermost and uses the first definition it finds. In other words, an inner scope shadows (overrides) the same variable key defined in an outer scope. For example, if a key `k8sClusterName` is defined both as an environment variable and as an experiment variable, the experiment uses the experiment-scoped value. Steadybit highlights such shadowing in the variables popup (see below) so you always know which value actually applies. A variable only needs to be defined in one scope to be usable, you can, for instance, reference an environment variable in an experiment without redefining it. ![Variable shadowing in an experiment an environment-scoped variable](/files/EPiwqTkKRMkRLekmE82G) ### Environment Some values are associated with an experiment's environment and change whenever you use another environment in an experiment. Also, once you change the variable's value, all experiments using the same environment will also use the new value. Environment-scoped variables are the perfect place to name, e.g., common infrastructure components. You can edit environment-scoped variables in settings -> environments. Admins can edit all environment variables, users only variables of environments their team can access. ![Variables scoped to an environment](/files/DyVT2Mv0dXuSdxbHqKCR) ### Service Service-scoped variables are defined on a [service](/use-steadybit/services#variables) and apply to every experiment that uses the service, both the experiments provided by the service and custom experiments linked to it. They sit between the environment and experiment scope: a service variable extends or overrides an [environment variable](/install-and-configure/manage-environments#environment-variables) with the same key, and can itself be overridden by an experiment variable or a run override. Use service variables to align common configuration across all of a service's experiments, for example a service-specific base URL, namespace, or duration, without repeating it in every experiment. Service variables can also be referenced inside the service's [validation](/use-steadybit/services#validations) definitions. When a service variable holds a [dynamic value](#dynamic-value), you additionally choose its **evaluation scope**: whether the value is sampled from the service's own targets (the default) or from the whole environment the service lives in. See [Evaluation scope](#evaluation-scope) for details. {% hint style="info" %} When an experiment is linked to **multiple services** and more than one of them defines the same variable key, the service that was associated later takes precedence. {% endhint %} Service variables are managed in the service editor's *Variables* tab. [Learn more in the services section](/use-steadybit/services#variables). ![Editing service's variables](/files/ehgUdUqrDkSo8kBbhLrI) ### Experiment The next scope in an experiment design is the 'experiment' scope of a variable. An experiment-scoped variable is only accessible from within the current experiment, and changing the variable's value only affects the current experiment. If you have two different experiments, they can each use the same variable name but assign different values without affecting each other, as they refer to different variable definitions. An experiment-scoped variable can shadow variables in an outer scope (i.e., a service or environment variable). If that is the case, the experiment uses the value of the experiment variable instead of the service or environment variable. Steadybit indicates shadowing as shown below, where the value for the `k8sClusterName` variable is defined as `docs-demo` for this specific experiment and `demo` for all other experiments using the environment `Global`. The variable doesn't exist for experiments that aren't using the environment `Global`. ![Variable shadowing in an experiment an environment-scoped variable](/files/EPiwqTkKRMkRLekmE82G) ### Run For single experiment runs you can override used variables. Doing so, will apply different values, which can be defined by the user. The overrides are just applied of a single run. To do so, access the modal through the `Run Experiment` button in the editor. ![How to access experiment overrides](/files/YytnO3J6Dkpw57y3KMrb) Doing so will open a modal where you can specify new values for each used variable. Experiment variable overrides will only be applied for this specific run. After the run was triggered, overrides are gone and environment/service/experiment variable values are applied again. ![Defining experiment variable overrides](/files/gwKWD5svE06kU7DBhmX9) Alternatively, you can define different overrides for different experiment schedules, see [experiment schedules](/use-steadybit/experiments/schedule) ## Value Settings Independent of its scope, every variable (environment, service, or experiment) holds one of three kinds of value: a single **fixed value**, a **list of fixed values**, or a **dynamic value**. You choose the kind in the variable's value settings dialog, reachable via the settings icon next to the value field. ![Variable Value Settings](/files/t5j62tiuVfgfPgDei25z) ### Fixed Value A fixed value is a constant string you type yourself. It resolves to exactly that value wherever the variable is used and is known already at design time. Use a fixed value for stable, well-known configuration, for example a Kubernetes cluster name, an HTTP base URL, or a fixed duration like `30s`. ![Fixed Value](/files/fzFGCTbZIm5hq0LlxRLH) ### List of Fixed Values A variable can also hold several fixed values at once. Pick *"List of fixed values"* in the value settings dialog and add each value as its own entry. A list behaves like a multi-value [dynamic value](#dynamic-value), but the values are known at design time instead of being selected from your infrastructure: * In a blast-radius query such as `... IN ({{var}})` the variable expands to one comparison value per entry, so the query matches *any* of them. For example, a variable `{{deployments}}` with the values `gateway`, `hot-deals`, and `fashion-bestseller` used as `k8s.deployment IN ({{deployments}})` matches all three deployments. * Where a single string is expected (for example a step parameter or a message), the entries are combined into a comma-separated list. Add the values as separate entries rather than typing one comma-separated string, a single fixed value is always treated as one literal (so a value may itself contain commas, spaces, or other punctuation without being split). ![List of Fixed Values](/files/fG7alwCMe0rcHVK4DmL9) ### Dynamic Value A dynamic value is not typed by hand. Instead, it is selected from your live infrastructure when the experiment run starts. You configure it as *"select one or more values of a target attribute on a target type"*. Use dynamic values when the specific value isn't known up front or should adapt to the current state of your infrastructure. For example, you might attack a pod that actually exists at run time rather than hard-coding a name, or vary the affected target between runs. The configuration consists of: * Target type — the kind of target to read the value from (e.g., a Kubernetes pod, a host, an AWS EC2 instance). * Attribute — the target attribute whose value should be used (e.g., the pod name, host name, or an AWS instance id). * Evaluation scope *(only for service variables)* — whether to sample from the service's own targets or the whole environment (see [Evaluation scope](#evaluation-scope)). * Filter *(optional)* — a query that restricts the candidate targets before a value is picked (e.g., only pods in a certain namespace). You also control the **cardinality** of the selection, how many of the matching values are picked: a single value or several. When more than one value is selected, the variable carries the whole set. In a blast-radius query such as `... IN ({{var}})` it expands to one comparison value per selected value; where a single string is expected, the selected values are combined into a comma-separated list. #### Evaluation scope For a dynamic value on a [service variable](#service), you additionally choose the **evaluation scope**, i.e. which targets the value is sampled from: * **Only targets of this service** *(default)* — only the service's own targets are considered: the service's environment narrowed by the service's [target query](/use-steadybit/services#target-scope). * **All targets of this service's environment** — all targets in the environment the service lives in are considered, ignoring the service's query. The evaluation scope only applies to service variables. For environment, experiment, and run variables, a dynamic value is always sampled from the environment. #### Run-time resolution Because a dynamic value depends on the live target index, it is resolved at the start of each run (not at experiment design time). The selected value or values then stay stable for the entire experiment run, every step that references the variable uses the same selection, and the resolution happens again only on the next run. The concrete value that was selected for a given run can be inspected afterward in the [resolved variables](#resolved-variables-in-experiment-run) view. ![Dynamic Value](/files/Llqx8LZDhmwgdsH1oQfF) {% hint style="info" %} A dynamic value resolves to a value from the target index. If, at run time, no target matches the configured type and filter, the variable cannot be resolved and the experiment run will report the unresolved variable. {% endhint %} #### Sample Values in Experiment Editor Because a dynamic value is only resolved for real when a run starts, the concrete values it will take are not known while you design the experiment. To give you a meaningful preview, Steadybit draws **one sample** of the matching values, when you open the experiment, when you change the environment, and whenever you create or edit a dynamic value, and uses that same sample everywhere in the editor. As a result the target counts per step, the *show targets* list, and the sampled values shown beneath each dynamic variable all agree with one another, instead of changing every time a preview is recalculated. The sampled values are shown directly under the variable's definition in the variables popup: ![Variables popup showing the sampled values beneath a dynamic variable](/files/B5dlYoN7gxV7o8V3snBB) This sample is only a design-time preview, it is never saved with the experiment. To draw a fresh sample at any time, use **Resample dynamic variable values** in the variables popup. Editing a dynamic value re-samples only that variable, so the sampled values for the other variables stay stable. Each run still selects its values independently when it starts (see above), so the preview shows a representative selection, not necessarily the values a future run will use, unless you deliberately run with the previewed sample (see [Run with sample values](#run-with-sample-values)). #### Run with Sample Values The primary *Run Experiment* button lets each run pick its own dynamic values at start. If you instead want to run the experiment with exactly the sample values in the editor, open the split button next to *Run Experiment* and choose **Run with sample values**. ![Run Experiment split button with the 'Run with sample values' option](/files/YytnO3J6Dkpw57y3KMrb) A dialog lists the dynamic variables in use together with their current sample values and starts a single run pinned to exactly those values. You can draw a fresh sample from within the dialog before running. !['Run with sample values' dialog listing variables in use and their sample values](/files/xyNggGCm5HrgKMjUNK2V) ## Referencing Other Variables Variables can be built from other variables using the same `{{...}}` syntax, so you can compose values from reusable building blocks instead of repeating them: * A **fixed value** can embed other variables. For example, a `baseUrl` variable defined as `https://{{host}}/api` reuses a `host` variable. * The **filter query of a dynamic value** can reference other variables. For example, a filter `k8s.namespace="{{namespace}}"` scopes the candidate targets to whatever the `namespace` variable resolves to. Referenced variables are resolved with the same [resolution strategy](#resolution-strategy) and scope rules as everywhere else, a reference can point to a variable from any scope (environment, service, experiment, or a run override). {% hint style="warning" %} References must ultimately resolve to a concrete value. Avoid circular references (a variable that refers back to itself, directly or through another variable),they cannot be resolved and the experiment run will fail. {% endhint %} ## Resolved Variables in Experiment Run In case you want to check which variables have been used for a single experiment run, you can check this in the experiment run view by clicking `See used variables` on the top section. The icon on the variable key tells you about the scope of the used variable (environment, service, experiment, or run override). For dynamic values, this view shows the concrete value that was selected for that specific run. ![Defining experiment variable overrides](/files/E1v1CdOBSWhbF0ZkghOs) # Properties Properties are key-value pairs that can be used to add additional information to your services, experiments and runs. They can be used to store metadata about the service or experiment, such as the service owner, experiment's purpose, the team responsible for it, the outcome, or just a note of a specific run or any other relevant information. In general, a user can only assign properties and capture values that have been [defined by an administrator](/install-and-configure/manage-properties#manage-property-definitions). Once you - or your administrator - has defined the property, you can assign and capture values per experiment design or experiment run. ## Examples * Defining a service's business criticality * Your administrator has defined a property definition called `serviceCritcality` with the datatype `enumeration (select one)`. * Your administrator has assigned the property definition `serviceCritcality` to all services and marks it as required. * Whenever an administrator or team owner adds or edits a service, he can capture the value of the property (e.g. `Tier 0 - Mission Critical`) * Adding a mandatory cost center field to each experiment * Your administrator has defined a property definition called `costCenter` with the datatype `integer`. * Your administrator has assigned the property definition `costCenter` to all experiment designs and marks it as required. * You need to fill out the cost center for each experiment you want to run. * Adding a note to a single experiment run * Your administrator has defined a property definition called `note` with the datatype `rich text`. * You can add the property `note` to a single experiment run, you are allowed to edit, and fill out a Markdown value like "This experiment was *really* great!". ## Assign Properties Properties that are defined by your administrator, can be assigned to a single service by administrator or team owners and to experiment design or run by any team member. This way, everyone can add additional context to your experiment designs or for a specific run. Properties assigned to an experiment design or an experiment run can be removed later on. Properties copied from the experiment design to a run cannot be removed later. ![Property Assignments in Experiment's 'Manage Properties'](/files/siwQDw8sACSdCD1MJzVs) A property assignment configuration consists of the following fields: * **Property Definition Key**: The referenced property definition assigned to the experiment design or run. * **Mandatory**: A checkbox indicating whether a value is required or not. If checked, the property must be filled in before the experiment can be run. * **Allow editing the value in the experiment run**: A checkbox indicating whether the property can be edited at run-level. If checked, the property can be edited in the run details page. ![Edit a Property Assignment](/files/rDlXNfnxbc1Rh0u38Iu1) ### API Properties can also be assigned using the following API endpoints. Check out [Integrate with Steadybit / API](/integrate-with-steadybit/api) how to access the API. * [Assign properties to a service](https://platform.steadybit.com/api/swagger/swagger-ui/index.html?configUrl=/api/spec/swagger-config#/Services/upsertService) * [Assign properties to an experiment design](https://platform.steadybit.com/api/swagger/swagger-ui/index.html?configUrl=/api/spec/swagger-config#/Experiment%20Designs/createOrUpdateExperiment) * [Assign properties to an experiment run](https://platform.steadybit.com/api/swagger/swagger-ui/index.html?configUrl=/api/spec/swagger-config#/Experiment%20Executions/updateExecutionProperties) ## Property Values Property values are the actual values of the properties that are saved as part of an experiment design and copied to each experiment run. [Variables](/use-steadybit/experiments/variables) are fully supported for the input fields of all properties (including overrides for single runs or via schedules). ![Properties in the experiment editor](/files/Obs2bVpONeirpHL4IY94) ![Properties in the run view](/files/ehpiXq1iQkVTJKmCJWEM) ### API Properties' values can also be read and edited using the following API endpoints. Check out [Integrate with Steadybit / API](/integrate-with-steadybit/api) how to access the API. * [Update the properties of a service](https://platform.steadybit.com/api/swagger/swagger-ui/index.html?configUrl=/api/spec/swagger-config#/Services/upsertService) * [Update the properties of an experiment design](https://platform.steadybit.com/api/swagger/swagger-ui/index.html?configUrl=/api/spec/swagger-config#/Experiment%20Designs/createOrUpdateExperiment) * [Get the properties of an experiment run](https://platform.steadybit.com/api/swagger/swagger-ui/index.html?configUrl=/api/spec/swagger-config#/Experiment%20Executions/getExperimentExecution) * [Update properties of an experiment run after it has completed](https://platform.steadybit.com/api/swagger/swagger-ui/index.html?configUrl=/api/spec/swagger-config#/Experiment%20Executions/updateExecutionProperties) # Emergency Stop In case a running experiment causes unforeseen severe failures every authenticated user can stop any running experiment by clicking the emergency stop button. You can find it in the 'Current Activities' at the left. ![Emergency Stop](/files/CRVCqGJ1OFTEAmR1dfpN) After that, all running experiments will be stopped and in addition starting new experiments will be prevented (via UI as well as API). # Share Steadybit offers several ways to share an experiment to make it easy to share knowledge and enable central organized rollouts. Pick the approach that matches how much central control you need, how the design should evolve and how tightly the shared experiments should stay in sync: * Deep link — Just sharing a useful experiment's scenario or run for shared analysis * [Duplicate Experiment](/use-steadybit/experiments/share/duplicate-experiment) — One-off, fully editable copy in another team * [File Import/Export](/use-steadybit/experiments/share/file-import-export) — Move an experiment across Steadybit platforms, into version control, or CI/CD * [Share With Teams](/use-steadybit/experiments/share/share-experiment) — Another team should run a specific experiment as-is, with one shared run history * [Experiment Templates](/use-steadybit/experiments/share/templates) — Other teams should start from your design and adapt it to their own needs * [Service Provided Experiment](/use-steadybit/experiments/share/service-provided) — The same design needs to be tested for different environments/targets, with central control over the design # Duplicate Duplicating an experiment creates a fully editable copy — even across teams and environments within the same Steadybit tenant. Use this when another team wants to use your experiment as a **starting point** and adapt it freely, with no expectation that future changes to the original will reach the copy. Once you expect increased reusing across teams, consider using the [experiment template](/use-steadybit/experiments/share/templates) approach. ## How Sharing Works Two artifacts are involved: * **Original Experiment** — the source of the duplicate. After duplication, it continues to live and evolve independently. * **Duplicated Experiment** — a new, independent experiment placed into a team and environment of your choice. Once created, it has no link back to the original. Because the duplicate is a detached copy, design changes do not propagate in either direction. ## Single Source of Truth | Aspect | Source of truth | | ------------------- | -------------------------------------------------- | | Experiment instance | Per duplicate — each duplicate is a new experiment | | Experiment design | Detached copy at the time of duplication | | Experiment runs | Per duplicate | For sharing where design changes should propagate, use [Service Provided Experiments](/use-steadybit/experiments/share/service-provided) or [Share Experiment](/use-steadybit/experiments/share/share-experiment) instead. ## Duplicate via Experiment List On the experiments overview page, every experiment row has a 3-dot menu. Choose **Duplicate Experiment**. ![Duplicate via Experiment List](/files/xkYwhRelLnH68ZmoKPBH) ## Duplicate via Experiment Designer In the experiment designer, the ![Button](/files/N6HkeAJbyHJb4xrEP3kN) duplicate button creates a copy of the currently opened experiment. ![Duplicate via Designer](/files/hqOMvsIyXEg4aH760uoA) ## Choose Name, Team and Environment After duplicating, you are prompted for the new experiment's name, description, team and environment. ![Duplicate Popup](/files/jgqgqf2WNkVsmIC4N2LQ) Confirm the dialog to save the new experiment and open it in the experiment designer. ![Duplicated Experiment](/files/8srUcabEzd0oYVgmufDR) ## When to Use This Approach Duplicating an experiment is the right choice when: * Another team should start from a copy of your experiment and freely adapt it * The original and the copy should evolve independently going forward * You want to keep the duplicate in a different team or environment within the same tenant For other sharing needs, see the [overview of sharing options](/use-steadybit/experiments/share). # File Import / Export Exporting an experiment as a JSON file lets you share its design across Steadybit platforms — for example to move an experiment between on-prem instances, archive it in version control, or hand it off out-of-band. Use this when sharing must cross a Steadybit platform boundary, or when the experiment design should live alongside other artifacts in your repository. ## How Sharing Works Two artifacts are involved: * **Exported JSON file** — a snapshot of the experiment's design at the moment of export, including team and environment references. * **Imported Experiment** — a new experiment created from the file in the importing platform. Once imported, it is a regular, fully editable experiment with no link back to the original. Because the file is a snapshot, file import/export creates a **detached copy** rather than a live link. ## Single Source of Truth | Aspect | Source of truth | | ------------------- | --------------------------------------------------- | | Experiment instance | Per import — each import creates its own experiment | | Experiment design | Detached copy at the time of export | | Experiment runs | Per imported experiment | There is no propagation of design changes after import. To apply updates, re-export and re-import — or use [Service Provided Experiments](/use-steadybit/experiments/share/service-provided) for a single source of truth that propagates. ## Export an Experiment Open the experiment in the designer and click ![Export Button](/files/pBCHJcTZ1FrsfSCpxoxK) **View Experiment as JSON**. ![Experiment Editor Export](/files/YVL21TeNskK9z1shJtvY) A popup appears with a download button and the option to copy the JSON to your clipboard. From there, you can edit it in the JSON editor of your choice. ![Download JSON](/files/GZKHhEvS7yKRiM2LQ48e) ## Import an Experiment Click **New Experiment** in the experiment list and drop a JSON file onto the upload area (or click to pick one). The file is parsed, imported, and the resulting experiment is opened in the designer. ![New Experiment](/files/I2kmBG31fVHyTJdfME5I) ## Flexible Team and Environment Assignment The exporting team and environment are written into the file. On import, the same team and environment must exist and be accessible to you. Otherwise, you have to change the team and environment in the import-flow. To make exported experiments portable from beginning on, replace the concrete values with variables before export: * `{{teamKey}}` — applies the experiment to the importing user's current team * `{{environmentName}}` — applies the experiment to that team's first environment ## When to Use This Approach File import/export is the right choice when: * You need to share a design across Steadybit platform instances (e.g. between on-prem tenants) * You want to keep experiment definitions in version control alongside other code * You plan to use the API to automate experiment creation e.g. in CI/CD * Each receiving environment should own a fully editable, independent copy For other sharing needs, see the [overview of sharing options](/use-steadybit/experiments/share). # Share with Teams Sharing an experiment lets another team run an experiment owned by your team in your environment without creating a copy. The experiment design stays in the owning team and remains the single source of truth, while the team it is shared with can run it, schedule it and adjust [per-run custom properties](/use-steadybit/experiments/properties). Use this when another team should use a specific experiment as-is and the owning team must keep full control over the design. ## How Sharing Works Two teams are involved: * **Owning Team** — owns the experiment, controls the design, and decides which other teams it is shared with. * **Receiving Team** — picked when sharing; gets read-only access to the design and full control over the runs. A shared experiment is not duplicated. The receiving team sees the same instance as the owning team, just with restricted edit rights. ## Single Source of Truth | Aspect | Source of truth | | ------------------- | -------------------------------------------------------------------------------------- | | Experiment instance | The owning team's experiment (one experiment, accessible to multiple teams) | | Experiment design | The owning team — only the owning team can edit the design | | Experiment runs | All runs land on the owning team's experiment, visible to every team it is shared with | ## Permissions Only **Administrators** and **Team Owners** can share an experiment. See [Permissions](/install-and-configure/manage-teams-and-users/permissions) for details. A team that an experiment has been shared with can: * Run the experiment * Schedule the experiment * [Edit custom properties of a run](/use-steadybit/experiments/properties) The receiving team **cannot** edit the experiment design itself. Any change to the design has to be done by the owning team. ## Share an Experiment Open the experiment you want to share and select **Share with teams** in the experiment designer's context menu. Choose the team you want to share it with and confirm. ![Experiment designer's context menu allowing to share the experiment with other teams](/files/lZHVF9Vmu18xL0XqwbiS) ![Experiment sharing](/files/23Yleao4ZvRyifLqcdp8) ## Find Shared Experiments Experiments shared with your team appear in **Experiments** → **Shared Designs** and can be filtered based on the owning team. From there you can run, schedule, or override run properties just like for your own experiments. ![List of shared experiment designs](/files/J0CPsbhoPgMjSc0xNU7R) ## When to Use This Approach Sharing an experiment design is the right choice when: * Another team should run a specific experiment owned by your team without duplicating it * The experiment design must stay in one place and be controlled by the owning team For other sharing needs, see the [overview of sharing options](/use-steadybit/experiments/share). # Templates Saving an experiment as a template makes its design available to every team in your Steadybit tenant — and optionally to the wider Chaos Engineering community via the Reliability Hub. Each team can then instantiate the template into their own experiment, with team-specific values filled in via placeholders or environment variables. Use this when you want a proven scenario to be a **starting point** that other teams adapt to their context, rather than a frozen experiment they run as-is. ## How Sharing Works Two artifacts are involved: * **Experiment Template** — the shared, reusable design. It can declare placeholders and environment variables so that team-specific values are filled in at instantiation time. * **Instantiated Experiment** — a concrete experiment created from the template. Once instantiated, it is a regular experiment owned by the instantiating team. Templates are tenant-wide: every team can see the catalog and instantiate any template into their own team and environment. ## Single Source of Truth | Aspect | Source of truth | | ------------------- | ----------------------------------------------------------------- | | Experiment instance | Per instantiation — one new experiment per team | | Experiment design | Initially the template; once instantiated, the design is detached | | Experiment runs | Per instantiated experiment | {% hint style="warning" %} Template changes are **not** propagated to experiments that were already instantiated from it. For a single source of truth that propagates updates, use [Service Provided Experiments](/use-steadybit/experiments/share/service-provided) instead. {% endhint %} ## Permissions Saving an experiment as a template requires **administrator** permission. See [Permissions](/install-and-configure/manage-teams-and-users/permissions) for details. ## Save Experiment as Template Open the experiment you want to share and select **Save as Template** in the save-split button. ![Experiment Editor - Save as Template](/files/hdEErNH76ETPrVPceAgx) The experiment is opened in the experiment template's editor. Here, you must at minimum fill out the template description (title and details). It is also advisable to replace concrete parameters with [template placeholders](/install-and-configure/manage-experiment-templates) or environment variables, so other teams can adapt the experiment to their own context. ![Experiment Template Editor](/files/rKWpK9ntowKcpAzzvRIs) Once you're done, save the experiment template. ## Find and Use Templates ### Within Your Steadybit Tenant All experiment templates are automatically available to every team in your Steadybit tenant. Each team can instantiate a template into a concrete experiment owned by their team and bound to their environment. You can also share deep links to a template's details page. ![Template details](/files/Fyo4zuE3L0kV5CCWIqTY) ### Sharing With the Community You can publish valuable templates to the Chaos Engineering community via Steadybit's [Reliability Hub](https://hub.steadybit.com/). Read more in the [manage experiment template section](/install-and-configure/manage-experiment-templates#share-templates). ## When to Use This Approach Sharing an experiment as a template is the right choice when: * Other teams should start from your design but adapt it with team-specific targets, validations, or parameters * The instantiated experiments should evolve independently per team * You want to publish a reliability scenario to the wider community via the Reliability Hub * You want to prepare for a [Service Provided Experiments](/use-steadybit/experiments/share/service-provided) For other sharing needs, see the [overview of sharing options](/use-steadybit/experiments/share). # Service-Provided A service provided experiment is the strongest form of sharing in Steadybit: the experiment design lives in a single [experiment template](/use-steadybit/experiments/share/templates) and is automatically rolled out, read-only, to every [service](/use-steadybit/services) whose [service profile](/install-and-configure/manage-service-profiles) references it. Use this when you want a reliability scenario — for example *Pod Redundancy* or *Loss of a Zone* — to be applied identically across many services, while keeping a single place to evolve the design. ## How Sharing Works Three building blocks are involved: * **Experiment Template** — the canonical experiment design. It is authored once and contains placeholders such as `[[SERVICE]]` that are filled in per service. * **Service Profile** — groups multiple experiment templates into categories (e.g. Scalability, Redundancy, Dependencies) and defines which templates a service is expected to fulfill. * **Service** — links to exactly one service profile and contributes its target scope and validations. When a service is linked to a profile, Steadybit instantiates each template in the profile on-the-fly, substituting the service's targets and validations. The result is a **provided experiment** that appears on the service detail page under *Provided Experiments*, ready to run without any manual design work. ## Single Source of Truth | Aspect | Source of truth | | ------------------- | -------------------------------------------------------------------------------------- | | Experiment instance | Per service — each service has its own provided experiment generated from the template | | Experiment design | Controlled by the experiment template and propagates on changes | | Experiment runs | Per service — each service produces its own runs against its own targets | Because the design lives only in the template, every service that uses the same profile sees the same scenario, with the only difference being the service-specific targets and validations. ## Permissions Provided experiments cannot be edited in the experiment designer. The design is owned by the template; on a service, you can only: * Run the experiment against the service's targets * Schedule the experiment To change *what* the experiment does, edit the underlying experiment template. ![Read-only Service Provided Experiment](/files/qP9pruYxcpGspJTNL1z5) Edits to a template are propagated automatically to every provided experiment derived from it, across all services and teams that use a profile referencing the template. There is no per-service copy to keep in sync — fix a step or tighten a check once, and every service immediately runs the updated scenario on its next execution. This is what sets service provided experiments apart from [duplicating an experiment](/use-steadybit/experiments/share/duplicate-experiment) or [instantiating a template](/use-steadybit/experiments/share/templates) into an individual experiment, where the resulting design is detached and changes have to be re-applied manually. ## Lifecycle Considerations Because provided experiments are derived rather than copied, removing the link between template, profile and service also removes the derived experiments: {% hint style="warning" %} Removing a template from a service profile deletes the provided experiments and their runs for every service that uses the profile. Changing a service's profile deletes the provided experiments and runs that are no longer part of the new profile. {% endhint %} Plan template and profile changes accordingly, and prefer evolving an existing template over removing and re-adding it when you want to preserve run history. ## When to Use This Approach Service provided experiments are the right choice when: * The same reliability scenario should apply to many services in a standardized way * You want a single place to evolve the experiment design and have changes apply everywhere * Each service should keep its own run history against its own targets and validations For other sharing needs, see the [overview of sharing options](/use-steadybit/experiments/share). # OpenTelemetry Integration For every experiment run, Steadybit collects distributed tracing spans using [OpenTelemetry](https://opentelemetry.io/) across the Steadybit platform and agents. Access to this data benefits users, extension authors and Steadybit maintainers alike. Here are some scenarios as part of which you might access this data: * Steadybit interests your organization, and you are in the process of building trust in the solution. As part of this, you want to understand what is happening as part of experiments – including the nitty-gritty details. * You are developing an extension, and something went wrong. You want to know precisely how your extension was called, the parameters, and how it responded. * To correlate experiment runs with other monitoring and observability data, e.g., in your Jaeger or Zipkin installations. * Something went wrong, and you need help from Steadybit's support staff to resolve the situation. Attach the distributed tracing data to give them context. As the following sections show, Steadybit enables the collection of this data automatically for simple use cases. However, you can instruct the Steadybit agents to report this data to your observability pipeline. This document explains both approaches. ![Trace encompassing the Steadybit platform and three Steadybit agents in Jaeger](/files/BoWRCE3FdSAWAQfFC1ss) ## Download through the Experiment Run View Steadybit collects and persists distributed tracing data across its platform and agents without further configuration for every experiment run. This is the simplest way to get started – and the option relevant to most customers. You can download the distributed tracing data as multiple [OTLP JSON files](https://opentelemetry.io/docs/reference/specification/protocol/). The UI explains importing and inspecting this data within the open-source tool [Jaeger](https://www.jaegertracing.io/). Distributed tracing data for experiments is retained for 28 days within the Steadybit platform. ![Downloading a zip file containing distributed tracing data through the experiment run view](/files/QPkOgb54dMCjbrr6g9fU) ## Exporting OpenTelemetry Data ***Note: OpenTelemetry data export is currently an experimental capability.*** It can be helpful to have Steadybit observability data within your systems. Steadybit agents can be instructed to export distributed tracing data to OpenTelemetry-compatible systems. This section explains how to configure the Steadybit agents to achieve this. To validate the configuration, the section contains optional guidance on how to set up a local Jaeger instance, Zipkin instance and an OpenTelemetry collector. ### Agent Configuration The Steadybit agent internally leverages the OpenTelemetry SDK auto-configuration module. Consequently, all of the [module's configuration parameters](https://github.com/open-telemetry/opentelemetry-java/blob/v1.24.0/sdk-extensions/autoconfigure/README.md#sampler) are supported! This section only shows the most basic configuration to achieve data export. The configuration parameters are set through environment variables, as the following `shell` snippet shows. You may also pass these environment variables when deploying the agent through any other mechanism, e.g., Helm charts. ```bash # enable the auto-configuration mechanism export JAVA_OPTS="-Dotel.java.global-autoconfigure.enabled=true" # Name the service. You most likely wanna keep it as 'steadybit-agent' export OTEL_SERVICE_NAME="steadybit-agent" # Define where to export the data to. export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" # The Steadybit agent does not currently expose any metrics through OpenTelemetry. export OTEL_METRICS_EXPORTER="none" ``` ### Sample Jaeger and OpenTelemetry Collector Setup The following sections explain how to spin up a local Jaeger instance, a Zipkin instance and an OpenTelemetry collector. These steps are optional for a successful configuration of the export mechanism. We list these here for your convenience if you want to check the setup locally. #### OpenTelemetry Collector Configuration We start with a configuration for an OpenTelemetry collector. The collector will accept the telemetry data from Steadybit agents, batch it and then forward it to both Jaeger and Zipkin. Store this in a file called `otel-config.yml` within your current working directory. ```yaml receivers: otlp: protocols: grpc: http: processors: batch: exporters: otlp: endpoint: jaeger-service:4317 tls: insecure: true zipkin: endpoint: http://zipkin-service:9411/api/v2/spans tls: insecure: true service: pipelines: traces: receivers: [ otlp ] processors: [ batch ] exporters: [ otlp, zipkin ] ``` #### Docker Compose Configuration Next, we start all the systems locally using [Docker compose](https://docs.docker.com/compose/). Note the comments about UI endpoints within the snippet. Store this in a file called `docker-compose.yml` within your current working directory. Then run `docker-compose up` to start everything. Once the startup completes, you can use the following URLs to interact with the systems: * Jaeger UI: * Zipkin UI: * OpenTelemetry collector OTLP gRPC endpoint: ```yaml version: "4.0" services: zipkin-service: image: openzipkin/zipkin:latest ports: # UI and ingestion endpoint - "9411:9411" jaeger-service: image: jaegertracing/all-in-one:latest ports: # UI endpoint - "16686:16686" # OTLP gRPC endpoint - "4317" # OTLP HTTP endpoint - "4318:4318" environment: - COLLECTOR_OTLP_ENABLED=true collector: image: otel/opentelemetry-collector:latest command: [ "--config=/etc/otel-collector-config.yml" ] volumes: - ./otel-config.yml:/etc/otel-collector-config.yml ports: # OTLP gRPC endpoint the agent will be interacting with - "4317:4317" depends_on: - jaeger-service - zipkin-service ``` # Reporting Steadybit's integrated reporting feature gives you a comprehensive overview of your service reliability risk, your experiment activity, and your adoption of Steadybit across users, teams and environments. Use it to track progress over time, share evidence with stakeholders, and spot regressions across your infrastructure. ![Example of a report showing average service risk](/files/seVzu0KJPbDlSEEyOHfB) You can filter the reports based on different dimensions (e.g. service, team, environment, service) or use the legend's labels below any report to filter the data shown in the graphs. All charts can be downloaded as CSV, PNG or PDF. ## Service Risk The **Service Risk** reports answer questions like how risky are services on average, how are they distributed across risk levels, and which reliability categories are driving the risk. [Learn more about Steadybit's service risk](/use-steadybit/services#risk). Each report writes a fresh data point per service whenever the service's risk changes, or — at the latest — once per day. This means your historical timeline always reflects the property values that were in effect at the time. You can filter Service Risk reports by * Timeframe * Teams * Environments * Services * Service Properties (enum-typed [custom properties](/use-steadybit/experiments/properties)). This allows to focus on services being tagged, e.g., with `Tier 0 - Mission Critical` and drill down on [reliability categories](/install-and-configure/manage-service-profiles#categories) like 'Scalability'. ### Average Risk Over Time Track the rolling average risk across your services. Useful as a single-pane health number to share with stakeholders or to spot regressions when new services are onboarded or a service profile changes. ![Average service risk over time](/files/seVzu0KJPbDlSEEyOHfB) ### Risk Distribution See how many of your services fall into the low, medium and high risk levels. Easy to share with stakeholders the value of your reliability work by showing trending of services into low risk levels. ![Example distribution of services by risk level](/files/a50GEcDU3J6Cdwsr2OmI) ### Risk by Category Break down the average risk per reliability category (e.g. Redundancy, Scalability, Dependencies) merged globally across all service profiles. Use this to identify which dimensions of reliability need the most investment across your services — independent of which service profile a service uses. ![Example risk by category](/files/XEbNExtNx5sBBHG2GKVn) ## Experiment Runs The Experiment Runs report gives you an overview of all experiment runs that have been executed — including their outcomes, what triggers them, and how they move between completed and failed over time. Use it to make experiment activity visible across teams and to spot when chaos coverage starts to drift. You can filter the reports by the following criteria: * Timeframe * Teams * Environments * Services ### Number of Runs Find out how many experiments your teams have run in total. ![Example number of experiment runs](/files/uY6081vCO4nZbxycROfL) ### Attack Types Identify which attacks your teams have used most frequently. ![Example experiment run attack types](/files/GyoxJPa8RLqMk8H864Lk) ### Trigger Check out what typically triggers an experiment run, e.g., API, CLI, UI, or schedule. ![Example experiment run trigger](/files/3MlQpaVYfeeswt4Nufs4) ### Result Drill down into the experiment runs by the result and compare the numbers of completed, canceled, failed, and errored experiment runs. ![Example experiment run result](/files/gAFjoRGBwXVDiiNwjiUo) ### Result (Completed vs. Failed) Compare the portion of completed experiment runs to failed experiment runs to identify the frequency of identifying issues. ![Example experiment run completed vs. failed](/files/xy0KWXeNO4xwQzkgA8iv) ### Issues Discovered Identify how many experiment runs turned from completed to failed. We count experiment failures that were immediately preceded by a completed experiment run. ![Example experiment run discovering issues](/files/a9cHSpolDfP7cvg2cwKg) ### Issues Fixed Identify how many experiment runs turned from failed to completed. We count experiment runs completed that were immediately preceded by a failed experiment run. ![Example experiment run showing fixed issues](/files/pKU6sF0N6iBACultPtbI) ## Experiments The Experiments report gives you an overview of experiments that have been designed in your environment — how many designs exist, what channels teams use to create them, and which methods (from scratch, template, or advice) they prefer. Useful for tracking the spread of experiment authoring across the organization. You can filter the report by the following criteria: * Timeframe * Teams * Environments ### Number of Experiments Find out how many experiments your teams have designed in total. ![Example number of experiments](/files/7KTVSE8GQUMdGvrR0TmR) ### Creation Channel Identify which channel is used the most across your teams to create an experiment: UI, API, or CLI ![Example of experiment creation channels](/files/wMAE5uvN2AtjGROjYpT0) ### Creation Method Identify which method is used the most across your teams to create an experiment: From scratch, template, or advice ![Example of experiment creation method](/files/QYlrteTPgrFVuNCiSeBr) ## Others These reports give you an overview of the adoption of Steadybit across your organization. You can filter the reports by timeframe. ### Users Identify the progress you have made to roll out Steadybit in your organization by seeing the number of invited users. ![Example of users](/files/568mLXg89otREoxh7KfT) ### Teams Easily report on the numbers of teams having access to a safe Chaos Engineering in your organization. ![Example of teams](/files/BbELm5OAD9JYYm0s9zlq) ### Environments Find out how many environments you have created to roll out a safe Chaos Engineering across your organization. ![Example of environments](/files/mT5O46fgdGltUOd0fnUI) # SteadyBuddy {% hint style="info" %} **SteadyBuddy is part of Steadybit Labs** and is currently only available upon request. [Learn more](https://steadybit.com/blog/introducing-steadybit-labs-help-shape-the-future-of-reliability-testing/) {% endhint %} SteadyBuddy is Steadybit's AI-powered assistant built into the platform that helps you design, run, and understand chaos experiments using natural language. Instead of clicking through the experiment editor, you can describe what you want to test, ask why an experiment failed, or let SteadyBuddy propose ready-to-run experiments tailored to your environment. ![SteadyBuddy chat overview with experiment suggestions](/files/KMl80KF1S9S8aTo46xUj) ## How SteadyBuddy Works SteadyBuddy acts **on your behalf** and only sees what you are allowed to see. Every question, suggestion, and experiment draft is scoped to: * **Your team** — only the actions, environments, and targets available to your currently selected team are used as context. * **Your permissions** — SteadyBuddy reads the same data you can read in the UI/API. It cannot surface targets, environments, or runs you have no access to. {% hint style="warning" %} **SteadyBuddy never performs destructive operations on its own.** Today, it performs **read-only** operations: it reads your actions, environments, targets, advice, and experiment runs to answer questions and to draft experiments. It does not create, modify, or delete anything without an explicit, human-initiated action — for example, you clicking a button in the UI. As the assistant gains the ability to take more actions on your behalf, we will add additional safeguards around potentially destructive operations. {% endhint %} ## Availability and Setup How SteadyBuddy is enabled depends on whether you use the Steadybit SaaS platform or run Steadybit on-prem. ### SaaS — Opt In For SaaS customers, SteadyBuddy is **off by default** and must be explicitly enabled by a tenant administrator. This is an opt-in step because chat and suggestions are processed by an external model provider (see [Data Processing and Privacy](#data-processing-and-privacy)). To enable SteadyBuddy, access it from the navigation or go to **Settings → Data & Access**. Once enabled, every user within your tenant can start using SteadyBuddy. If you are not an administrator, ask one of your tenant admins to enable it for you. ![SaaS opt-in to enable SteadyBuddy](/files/XLk6LXcb33WkNGlM4gfP) ### On-Prem — Configure a Provider For self-hosted (on-prem) installations, there is no separate opt-in: SteadyBuddy is available once you configure a model provider in the platform's environment variables and the AI capability is part of your license. On-prem deployments run against your own AI provider. See [configuration options / AI](/install-and-configure/install-on-prem-platform/advanced-configuration#steadybuddy) for the provider, model, and retention settings. ## Where to Use SteadyBuddy There are two ways to talk to SteadyBuddy: * **The SteadyBuddy sidebar** — available on every page via **Ask SteadyBuddy** at the bottom of the screen. It stays open while you navigate, so you can keep a conversation running next to whatever you are looking at. * **The SteadyBuddy page** — open it from the navigation for a full-width chat. Use this when the assistant is your starting point rather than your companion. Some pages give the conversation a context to work with, shown as a pill above the message box: | Page | What SteadyBuddy knows | | -------------------- | ----------------------------- | | An experiment run | The run and its experiment | | An experiment design | The experiment being designed | | A service's detail | The service | With a context in place you can just ask *"Why did this fail?"* without naming the run. Remove the pill if you want to ask something unrelated. **Ask SteadyBuddy** always starts a new conversation. To continue an earlier one, use the chat history in the sidebar's bottom bar, or open the SteadyBuddy page. ![SteadyBuddy sidebar generating experiment suggestions for a service](/files/0o0oj59fiBN3HtR97nIl) ## Data Processing and Privacy SteadyBuddy only sends the context needed to answer your request. Depending on what you do, this can include: your typed messages, the names and metadata of actions and environments available to your team, target type and attribute summaries for the selected environment, reliability advice, and experiment run results. It never sends credentials or secrets. {% hint style="info" %} **SaaS:** SteadyBuddy uses **Anthropic** models. Your data is **not used to train any models** — this is covered by our [data privacy terms](https://steadybit.com/imprint/). {% endhint %} **On-prem:** processing happens through your own chosen AI provider. No data ever leaves your infrastructure — neither to a Steadybit-hosted service nor to any other AI provider. Chat history is retained for a limited time (30 days by default), and you can [delete any individual chat](#work-with-your-chat-history) at any time. On-prem, retention periods are [configurable](/install-and-configure/install-on-prem-platform/advanced-configuration#steadybuddy). ## What You Can Do ### Get Experiment Suggestions Ask SteadyBuddy what to test and it analyzes the targets it finds to propose ready-to-run experiment ideas. Each suggestion comes with a short description and a **More details** view; from there you can **Build experiment** to open it in the editor. Use this when you are getting started in a new environment or with a new service, or need guidance on what to test next. You can scope suggestions in two ways: **For an environment** — on the SteadyBuddy page, pick an environment and SteadyBuddy proposes experiments for the targets it contains. ![SteadyBuddy showing experiment suggestions for the selected environment](/files/DAK3YC1N3xfk0Itd9H3a) **For a service** — on a [service's detail page](/use-steadybit/services#service-detail), use **Get suggestions** in the SteadyBuddy banner (or **Suggest experiments for this service** in the sidebar). The suggestions are scoped to that service's targets and its environment, and appear in the sidebar next to the service, so you can compare them against the service's existing [provided](/use-steadybit/services#provided-experiments) and [custom experiments](/use-steadybit/services#custom-experiments) without leaving the page. Experiment suggestions use the same flow described in [the following section](#create-experiments-from-scratch-via-chat). ### Create Experiments from Scratch via Chat Describe the experiment you want in your own words — for example, *"Test how my checkout service behaves when one of its pods is killed"*. SteadyBuddy asks clarifying questions when needed, then produces an experiment draft. You can: * **Open Experiment Draft** to review and edit it in the experiment editor, or * **Run Experiment** to execute it immediately. The draft is scoped to your selected environment and uses only actions and targets your team can access. ![SteadyBuddy chat conversation showing an inline experiment draft with Open/Run options](/files/eHntKnzPfKg5l2WKgIa4) The experiment draft must be saved manually before it appears in the Experiments section or can be run later. ### Analyze an Experiment Run Ask questions like *"Why did my last experiment fail?"*. SteadyBuddy reads the relevant run data and explains what happened, grounding its answer in the actual execution results rather than guessing. ![SteadyBuddy chat conversation showing an analysis of a failed experiment run](/files/41yvbGyWAcALQa8Ktx7H) ### Work With Your Chat History SteadyBuddy keeps a history of your past conversations. On the SteadyBuddy page it lives in a collapsible panel on the left — use the panel toggle to show or hide it, and **New chat** to start a fresh conversation at any time. In the [sidebar](#where-to-use-steadybuddy), the same history is reachable from the bottom bar. Your previous conversations appear under **Recent chats**. From there you can: * **Open** any past chat to continue where you left off. * **Rename** a chat to give it a more descriptive title — hover over an entry and open its **⋯** menu, then choose **Rename**. * **Delete chat** from the same **⋯** menu to remove a conversation you no longer need. * **Search** across your chats to quickly find a specific conversation by its title. ![SteadyBuddy search across the chat history](/files/i2GPNWuBBlH8akuZrc24) Your chat history is scoped to you and your currently selected team, and follows the same [retention rules](#data-processing-and-privacy) as the rest of SteadyBuddy. ### Usage Limits AI usage is subject to a budget tied to your plan. When the budget for the current period is reached, the chat and suggestions are not available anymore. Please [contact us](https://steadybit.com/contact-us/) to enable AI features again. # Overview Steadybit is highly flexible, and you can seamlessly integrate it into your use cases and processes. For this, we offer various integration points that you can build upon. ## Integration Points ### Command-Line Interface (CLI) The CLI is ideal for integrating experiment runs or checking advice into your Continuous Integration/Continuous Delivery (CI/CD) pipeline. Under the hood, it uses the platform's [API](#api). Because the CLI is an npm package, it is agnostic to your actual CI/CD tool in action. [Learn more about the CLI](/integrate-with-steadybit/cli). ### API The platform's API allows you to perform every action that can be done via the user interface also programmatically. This is ideal for setting up Steadybit in an automated manner, such as creating teams and assigning environments, as well as for advanced use cases involving experiment creation and execution, e.g., utilizing an experiment template to create and run an experiment. [Learn more about the API](/integrate-with-steadybit/api). ### Webhooks The platform offers integrations via webhooks to get notified of events. We differentiate between a custom webhook, which can listen to events to integrate with external systems, and a preflight webhook, which can allow/disallow the starting of experiment runs. Learn more about [custom webhooks](/integrate-with-steadybit/webhooks/custom-webhooks) and [preflight webhooks](/integrate-with-steadybit/webhooks/preflight-webhooks). ### Extension Kits Extension Kits allow you to extend the Chaos Engineering capabilities by adding support for additional technologies or proprietary applications. For instance, to support a custom attack, check, load test, or observability integration, you need to implement [DiscoveryKit](/integrate-with-steadybit/extensions/extension-kits#discoverykit) and [ActionKit](/integrate-with-steadybit/extensions/extension-kits#actionkit). [EventKit](/integrate-with-steadybit/extensions/extension-kits#eventkit) is perfect when you want to react to experiment events, and [PreflightKit](/integrate-with-steadybit/extensions/extension-kits#preflightkit) is perfect whenever you want to have control over starting an experiment run. Last but not least, [AdviceKit](/integrate-with-steadybit/extensions/extension-kits#advicekit) allows you to ease your rollout by checking organization-specific best practices. [Learn more about Extension Kits](/integrate-with-steadybit/extensions/extension-kits). ## IP Ranges If your environment restricts outbound or inbound traffic to/from Steadybit (for example, via a firewall or security group), you can allowlist the following fixed CIDR range for our SaaS platform: | Scope | CIDR | | ----------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | All inbound connections to and outbound connections from the Steadybit SaaS platform (`platform.steadybit.com` / `platform.steadybit.io`) | `5.60.96.56/29` | This single range covers: * **Inbound** — agents, users, and API clients connecting to the Steadybit platform. * **Outbound** — webhooks, preflight checks, and any other call originated by the Steadybit platform back to your systems. ## When to use which Integration Point? This section highlights some key differentiators. Don't hesitate to reach out to us if you need support by discussing your integration use case. ### Webhooks vs. Extension Kits | | Sender | Interaction Type | Back-channel | Preventing experiment runs | Stop running Experiments | Listening to experiment lifecycle | Changing Properties | | -------------------------------------------------------------------------------- | -------- | ---------------- | ------------ | :------------------------: | :----------------------: | :-------------------------------: | :-----------------: | | [Preflight Webhook](/integrate-with-steadybit/webhooks/preflight-webhooks) | Platform | Synchronous | ✅ | ✅ | ❌ | ❌ | ✅ | | [Custom Webhook](/integrate-with-steadybit/webhooks/custom-webhooks) | Platform | Asynchronous | ❌ | ❌ | ❌ | ✅ | ❌ | | [PreflightKit](/integrate-with-steadybit/extensions/extension-kits#preflightkit) | Agent | Asynchronous | ✅ | ✅ | ✅ | ❌ | ✅ | | [EventKit](/integrate-with-steadybit/extensions/extension-kits#eventkit) | Agent | Asynchronous | ❌ | ❌ | ❌ | ✅ | ❌ | | [ActionKit](/integrate-with-steadybit/extensions/extension-kits#actionkit) | Agent | Asynchronous | ✅ | ❌ | ️✅/❌ | ❌ | ✅ | The remaining Extension Kits ([AdviceKit](/integrate-with-steadybit/extensions/extension-kits#advicekit) and [DiscoveryKit](/integrate-with-steadybit/extensions/extension-kits#discoverykit)) serve different purposes and are therefore not included in this comparison. ### CLI vs. API | | [Validate advice status](/use-steadybit/explorer/advice) | [CRUD of experiments](/use-steadybit/experiments/design) | [Run experiments](/use-steadybit/experiments/run) | [Use experiment templates](/use-steadybit/experiments/design#from-template) | [Schedule Experiments](/use-steadybit/experiments/schedule) | [Emergency stop](/use-steadybit/experiments/emergencystop) | [Read targets](/use-steadybit/explorer/targets) | [Configure Steadybit](#user-content-fn-1)[^1] | | --- | -------------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------- | --------------------------------------------- | | CLI | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | API | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | [^1]: Administrative functionality like managing [teams and users](/install-and-configure/manage-teams-and-users), [environments](/install-and-configure/manage-environments), [experiment templates](/install-and-configure/manage-experiment-templates), [hub connections](/integrate-with-steadybit/hubs), and [properties](/install-and-configure/manage-properties). # Audit Log The audit log retains historical information about actions triggered within the Steadybit platform. For example, the audit logs help to answer questions such as: * When was the experiment executed, by whom, and which hosts did it affect? * When did we engage the experiment kill switch? At the moment, only a handful of activities are logged for auditing purposes. In the future, this will be extended. You can access the audit log through the `/api/audit-log` endpoint. For more information, please see our [API documentation](https://platform.steadybit.com/api/swagger/swagger-ui/index.html?configUrl=/api/spec/swagger-config#/Audit%20Log). ### Retention Audit logs are retained for up to *90 days* for our SaaS platform. Customers leveraging an on-premise Steadybit platform installation can extend the retention through the `steadybit.auditlog.retention.period` configuration option. The configuration option accepts shorthands for time units, e.g., `90d` for 90 days. # API The Steadybit API gives you programmatic access to the platform. Every feature available in the UI is also available via the API. All API requests require an access token for authentication. ## Access Tokens To authenticate API requests, create an access token and pass it via the `Authorization` header in your request. Access tokens can be managed in the UI under Settings → API Access Tokens. ![Management of API Access Token](/files/3FVRO7FJqIsEBechSLtV) ### Token Types Steadybit supports three token types: **Team Tokens** are associated with one or more teams and can be used to access experiments and team-related operations within those teams. Admins and team owners can create team tokens. **Wildcard Tokens** grant access to all teams the token creator is an owner of. Team ownerships are resolved dynamically at authentication time, so the token automatically reflects any future team changes. Admins and team owners can create wildcard tokens. **Wildcard Tokens created by admin users allow access to all teams.** **Admin Tokens** grant access to platform management APIs (e.g., teams, environments, users). They are not associated with any team and can only be created by administrators. Tokens are automatically invalidated if their initial conditions no longer hold, e.g., if an administrator is changed to a normal user, admin tokens are invalidated. ### Token Expiration Access tokens can have an optional expiration date. Once expired, a token can no longer be used for authentication. Tokens without an expiration date never expire. Expired tokens can be **recreated** with a new expiration date. Recreating a token generates a new secret while preserving the original name, type, and team associations. The old token is invalidated. ### Token Permissions Who can manage tokens depends on the token type: | Action | Admin Token | Team Token and Wildcard Token | | -------- | ----------- | --------------------------------------------- | | Create | Admin only | Admin or team owner (of all associated teams) | | Delete | Admin only | Admin or token creator | | Recreate | Admin only | Admin or token creator | ### Creating a Token via the UI You can create a new access token in the UI under Settings → API Access Tokens. ![Add a new API access token](/files/fFcKLQ08mbLBE6mCr0AG) {% hint style="info" %} The token value is shown only once at creation time. Make sure to save it in a safe place. {% endhint %} ### Creating a Token via the API Access tokens can also be managed programmatically via the `/api/access-tokens/v2` endpoints. ```bash curl -X 'POST' \ 'https://platform.dev.steadybit.com/api/access-tokens/v2' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -H 'Authorization: accessToken ' \ -d '{ "name": "CI/CD access token", "type": "TEAM", "teams": [ "ADM", "DEV" ], "expiresAt": "2027-01-01T00:00:00Z" }' ``` Details on the Access Token API endpoint can be found in specification linked in the [OpenApi Specification](#openapi-specification) section. ### Creating an Admin Token via Internal API (On-Prem) On-premises customers can create admin tokens via an internal API. This is useful for automated provisioning of environments, teams, and templates. {% hint style="warning" %} Tokens created via the internal API are associated with an implicit "machine" user that cannot be removed or disabled. Experiments scheduled with such a token will continue to execute even after the token is deleted. This differs from tokens associated with a regular user, if that user is removed or loses permissions, their scheduled experiments will fail. {% endhint %} **Via CLI** SSH into the platform server and run: ```bash /scripts/createAdminToken.sh -t -n ``` Example: ```bash /scripts/createAdminToken.sh -t onprem -n AdminToken Z8pChlF2************* ``` **Via HTTP API** SSH into the platform server and call: ```bash curl \ -H 'Content-Type: application/json' \ -X POST \ -d '{"name":"'$NAME'","tenantKey":"'$TENANTKEY'"}' \ http://localhost:9090/actuator/adminaccesstoken ``` {% hint style="info" %} This endpoint is only accessible from localhost and is not reachable from outside the server. {% endhint %} ## OpenApi Specification We provide a [OpenApi 3.0 Specification for the API](https://platform.steadybit.com/api/spec) as well as an [interactive documentation](https://platform.steadybit.com/api/swagger). In case you are using our on-prem variant you can access it at `http:///api/spec`. ### Requests and Responses All API requests require a specified access token via the `Authorization` header in the format `Authorization: accessToken `. If applicable, request and response bodies are expressed using `json` or `yml`, depending on the used `Content-Type` and `Accept` headers. Success or failure of an API call is expressed via HTTP status. #### Too Many Requests API endpoints are rate limited and may return the HTTP status code `429 - Too Many Requests`. In this case the `Retry-After` response header contains the number of seconds to wait before executing further requests, see [RFC 7231](https://www.rfc-editor.org/rfc/rfc7231.html#section-7.1.3). Furthermore, the response headers `RateLimit-Limit`, `RateLimit-Remaining` and `RateLimit-Reset`, as defined in the IETF draft [RateLimit Header Fields for HTTP](https://www.ietf.org/archive/id/draft-polli-ratelimit-headers-02.html), are returned containing more details. ```bash curl \ -v \ -H "Authorization: accessToken "\ -H "Accept: application/json"\ https://platform.steadybit.com/api/ [...] < HTTP/1.1 429 Too Many Requests < ratelimit-limit: 100;w=60 < ratelimit-remaining: 0 < ratelimit-reset: 46 < retry-after: 46 [...] ``` ### Example: Create Experiment This is how you can create an experiment (`json` is supported as well): ```bash curl \ -i \ -H 'Content-Type: application/x-yaml' \ -H 'Authorization: accessToken ' \ https://platform.steadybit.com/api/experiments \ --data ' --- name: Experiment API Test team: ADM environment: Global lanes: - steps: - ! actionType: check:http parameters: method: "GET" url: "https://example.com" headers: [] successRate: 100 maxConcurrent: 5 requestsPerSecond: 1 duration: "10s" followRedirects: false readTimeout: "5s" connectTimeout: "5s" statusCode: "200-299" ' ``` The `Location` header of the response indicates the url of the newly created experiment: ``` location: https://platform.steadybit.com/api/experiments/ADM- ``` ### Example: Run Experiment You can then run the experiment: ```bash curl \ -i \ -X POST \ -H 'Authorization: accessToken ' \ https://platform.steadybit.com/api/experiments/ADM-1/execute ``` ### Create a golang client with oapi-codegen In case you want to [generate the structs](https://github.com/oapi-codegen/oapi-codegen), you should add this parameter to your configuration file: ```yaml compatibility: circular-reference-limit: 11 ``` Here is an example configuration to generate a Go client with net/http: ```yaml package: api generate: std-http-server: true models: true output: gen.go compatibility: circular-reference-limit: 11 ``` And then in your golang file: ```go //go:generate go run github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen --config=config.yaml https://platform.steadybit.com/api/spec ``` # CLI The [steadybit CLI ](https://github.com/steadybit/cli)allows you to connect your repository to Steadybit and having a single source of truth for all your experiments. By enabling you to create / update and retrieve your experiment definitions you have the truth versionized at one place. Of course, you can also run your experiments from the command line interface. This allows you to implement GitOps practices easily. ## Installation Nothing more than a simple ```bash npm install -g steadybit ``` Afterwards you have to create a profile and authenticate via an [access token](/integrate-with-steadybit/api#access-tokens). Check out our public GitHub repository [steadybit CLI](https://github.com/steadybit/cli#authorization) to learn more. ## Usage The CLI is mostly self explaining. For a detailed documentation have a look at our public GitHub repository [steadybit CLI](https://github.com/steadybit/cli) # Remote MCP Server {% hint style="info" %} **The Remote MCP Server is part of Steadybit Labs** and is currently only available upon request. [Learn more](https://steadybit.com/blog/introducing-steadybit-labs-help-shape-the-future-of-reliability-testing/) {% endhint %} Steadybit ships a hosted [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server, so you can connect AI agents such as Claude Code, Claude Desktop, or GitHub Copilot in VS Code directly to your Steadybit tenant. Once connected, the assistant can explore your environments, targets, actions, services, experiments, and execution results — and, with the right permissions, help you design, save, and run experiments — so you can understand your system, reason about resilience, and act on it. Every tool runs with the permissions of the credential you connect with — the same team permissions, environment allowances, and license that apply in the UI and public API — so the assistant can only do what that credential is allowed to do. With OAuth the credential is your own user account; an access token acts with the token's own scope (a single team, all teams, or an admin token's tenant-wide access). {% hint style="info" %} The MCP server is a licensed feature. If the endpoints below reject your requests, ask your Steadybit administrator to confirm that MCP is enabled for your tenant. The AI-assisted tools (experiment design, suggestions, and run analysis) additionally require the Steadybit AI feature. {% endhint %} ## Capabilities The server groups its tools into a few areas: **Explore & analyze** * Environments, teams, targets (with counts and statistics), and actions * Services and their Service Risk breakdown * Experiments and experiment templates * Executions (runs): per-step results, notably failed targets, logs, and the metrics collected during a run * Reliability advice for your targets * Reporting trends — experiment executions and Service Risk over time **Design & author** * Turn a natural-language description into an experiment design, or get experiment suggestions for an environment or service * Create a new experiment, update an existing one, or create one from a template **Run** * Start an experiment execution, or cancel a running one {% hint style="warning" %} Running an experiment injects **real faults** into your infrastructure. Treat the run tools as destructive — a well-behaved client will confirm with you before starting a run, and execution requires the same team and environment permissions as running from the UI or public API. {% endhint %} ## Authentication There are two ways to authenticate an MCP client. Pick whichever fits your client and workflow — both connect to the same server and expose the same capabilities. | Method | Endpoint | Best for | | ---------------- | --------------------------------------------- | ------------------------------------------- | | **Access token** | `https://platform.steadybit.com/mcp` | Any MCP client; scripted or headless setups | | **OAuth** | `https://platform.steadybit.com/mcp/` | Clients with an interactive browser login | ### Method 1 — Access token The simplest option, supported by every MCP client that can send an HTTP header. Create an [access token](/integrate-with-steadybit/api#access-tokens) in the UI under **Settings → API Access Tokens**, then pass it in the `Authorization` header when adding the server. The token already carries its tenant, so you connect to the plain `/mcp` endpoint. For [Claude Code](https://docs.claude.com/en/docs/claude-code): ```bash claude mcp add --transport http steadybit \ https://platform.steadybit.com/mcp \ --header "Authorization: Bearer " ``` The same works for any other MCP client — point it at `https://platform.steadybit.com/mcp` and configure an `Authorization: Bearer ` header. ### Method 2 — OAuth With OAuth, the client opens your browser, you log in to Steadybit through your identity provider, and you approve access on a consent screen — no token to copy or store. Point the client at the tenant-scoped endpoint `https://platform.steadybit.com/mcp/`, replacing `` with your tenant key. For [Claude Code](https://docs.claude.com/en/docs/claude-code): ```bash claude mcp add --transport http steadybit \ https://platform.steadybit.com/mcp/ ``` On first use the client opens a browser window where you sign in and confirm the connection:
OAuth consent screen granting an MCP client access to Steadybit
Access is granted based on your existing Steadybit membership: the assistant acts as you, within the same tenant and permissions you already have. #### Supported clients The OAuth path works with MCP clients that Steadybit has registered with its identity provider: * **Claude Code** * **Claude Desktop / Web** * **VS Code** — including **GitHub Copilot** chat and agent mode, which connect through VS Code's built-in MCP client Other clients that require Dynamic Client Registration (for example Cursor, Windsurf, the Codex CLI, and the GitHub Copilot CLI) are not supported on the OAuth path today. Use the [access token](#method-1-access-token) method with those clients instead. # Experiment Badges Get a badge for your experiment to integrate the latest run state, e.g., in wikis, ticketing tools, or wherever else it is valuable for you. You can easily integrate the badge using HTML, Markdown, or just a hosted image. We support two badges: * an [experiment badge](#experiment-badges) that shows the latest run state of a specific experiment * an [incident badge](#badges-linked-to-external-references) that either creates an experiment linked to an external reference or shows the latest run as soon as it is created state ## Experiment Badges You can reference an experiment on any external web page using badges, like this one: ![Example Badge](https://platform.steadybit.com/api/experiments/SHOP-61/badge.svg?tenantKey=demo) It’s an excellent way to get a link to your experiment and view the status without navigating and logging into the Steadybit platform. Go to an experiment that interests you, and click on the experiment badge icon on the top right. ![Steadybit Experiment Editor - Create Experiment Badge](/files/9pQ7dnGVi0gCqDVc3S5W) You can choose your badge's desirable format and scale and copy the resulting script. ![Steadybit Experiment Editor - Configure Experiment Badge](/files/vVMSw7ezNwQeHp9ne2LO) Alternatively, below are examples to copy and adjust to create your badge manually ### Example Markdown {% code title="example.md" %} ```markdown Template: [![{{expermiment-key}}](https://platform.steadybit.com/api/experiments/{{experiment-key}}/badge.svg?tenantKey={{tenant-key}})](https://platform.steadybit.com/experiments/{{team-key}}/edit/{{experiment-key}}/executions/?tenant={{tenant-key}}~) Example: [![SHOP-61](https://platform.steadybit.com/api/experiments/SHOP-61/badge.svg?tenantKey=demo)](https://platform.steadybit.com/experiments/SHOP/edit/SHOP-61/executions/?tenant=demo~) ``` {% endcode %} ### Example HTML {% code title="example.html" %} ```html Template: {{experiment-key}} Example SHOP-61 ``` {% endcode %} ## Tag Badges Experiment badges always link to a single existing experiment. If you want to refer to experiments that have a common purpose, you can instead use tag badges. Example use cases for this are linking to experiments reproducing a past issue from within your post-mortem documentation, indicating whether a specific architecture requirement is fulfilled, or providing visual evidence of the outcome of your latest disaster recovery test. Unlike experiment badges, tag badges can be created even if no experiment with the chosen tag exists, allowing you to use them as a call to action for creating a new experiment. This is how they look like in different states:
Tag badge when no experiment with the chosen tag exists

Tag badge when no experiment with the chosen tag exists

Tag badge when a single experiment with the chosen tag exists

Tag badge when a single experiment with the chosen tag exists

Tag badge when multiple experiments with the chosen tag exist

Tag badge when multiple experiments with the chosen tag exist

To create a tag badge, navigate to the Steadybit dashboard and click on 'Create a Badge'. ![Steadybit Dashboard - Create a Badge](/files/4zbeJLlfIG93QlIg4Ccd) You then either choose an existing tag or create a new one. That tag will be automatically added to any experiment created via the badge and used to identify experiments that should be associated with it. You can then configure the desired format and scale for your badge and provide the text that will be shown when no experiment with the chosen tag exists. ![Steadybit Dashboard - Create a Tag Badge](/files/rdntI1iLFaznR5hjofTM) Alternatively, below are some examples that you can copy and adjust to create your tag badge manually: ### Markdown {% code title="example.md" %} ```markdown Template: [![{{external-reference}}](https://platform.steadybit.com/api/badges/linked-badge.svg?tenantKey=demo&tag={{tag}})](https://platform.steadybit.com/api/badges/link?tenantKey=demo&tag={{tag}}) Example: [![INCIDENT-100](https://platform.steadybit.com/api/badges/linked-badge.svg?tenantKey=demo&tag=INCIDENT-100&createCaption=Create%20experiment%20for%20incident%20100)](https://platform.steadybit.com/api/badges/link?tenantKey=demo&tag=INCIDENT-100) ``` {% endcode %} ### HTML {% code title="example.html" %} ```html Template: {{external-reference}} Example INCIDENT-100 ``` {% endcode %} # Extensions Steadybit ships with various discoveries, attacks, checks and more to run chaos-engineering experiments and automatic weak spot identification. These capabilities ensure an excellent out-of-the-box experience. However, not every technology is supported natively by Steadybit. To this end, Steadybit exposes extension mechanisms through which everyone can add capabilities. For example, this enables * the discovery of new targets that one can attack within experiments, * the definition of new attacks for existing or new targets, * custom verification logic to run as part of experiments or * the usage of custom (shell) scripts in experiments. ## Terminology To understand the following content, we must clarify the used terminology. This clarification will help you find relevant content and skip the content you may not need. ### Extensions Extensions are what most regular users will interact with to extend Steadybit’s capabilities. They are deployable units (typically containers) exposing a remote HTTP interface with which Steadybit agents communicate. Extensions are deployed manually within customers’ networks. ### Extension APIs Extension authors, maintainers and contributors will leverage extension APIs to enhance Steadybit’s capabilities. Typically, extensions are required to expose a remote HTTP interface complying with extension APIs. ### Extension Kits Kits are the combination of extension APIs, conventions, contracts, documentation, examples and more. They are leveraged mainly by the same audience that would author extension. ## Next Steps * [What exactly is an extension?](/integrate-with-steadybit/extensions/anatomy-of-an-extension) * [What officially maintained extensions are there?](https://hub.steadybit.com/extensions) * [How do I install an extension?](/integrate-with-steadybit/extensions/extension-installation) * [What extension kits are available?](/integrate-with-steadybit/extensions/extension-kits) # Anatomy of an Extension To get the most out of Steadybit extensions, we believe that everyone can benefit from a fundamental understanding of the anatomy of extensions. This understanding will help you choose, deploy and secure extensions. This document provides an overview of a typical extension's architecture. ## Architecture From a high-level, extensions are: * Separately deployed processes, * which are typically packaged as Docker images, * running within your infrastructure that * expose an HTTP interface that complies with one or more extension APIs. That is a mouthful – let's dissect this. The following UML sequence diagram shows how the Steadybit agent would call an extension. ![UML sequence diagram showing how an extension implementing ActionKit APIs would be called by the Steadybit agent.](/files/MwAn4PPWvPDtub2pZAbG) Let us first note that the Steadybit agent and the extension are deployed within customers' infrastructure. Steadybit agents take on the role of outposts facilitating the communication between the Steadybit platform and extensions. This approach ensures that extensions do not need to be internet accessible. Instead, you can place them within networks that do not have a direct internet connection. From a security perspective, this is highly beneficial as it avoids several security challenges. Furthermore, it allows extensions to receive configuration, especially credentials, that Steadybit should not be aware of. Extensions are typically packaged as Docker images and consequently run as containers. Because extensions may require additional configuration & permissions, users carry the responsibility for the rollout of extensions. We offer ready-made Helm charts to facilitate the rollout of our official extensions. Extensions expose HTTP interfaces. These interfaces need to comply with extension APIs. For example, a custom attack implementation must comply with the API defined in ActionKit. Such extensions must define an HTTP endpoint that returns a list of supported actions. The UML sequence diagram above depicts this. The agent always initiates communication between itself and extensions. Consequently, agents need to know how to reach extensions, not vice versa. Learn how to configure extensions in agents [in our separate article](/integrate-with-steadybit/extensions/extension-installation). # Extension Installation Extensions follow common installation patterns that look something like this: 1. Deploy the extension within your infrastructure. 2. Make sure the extension is reachable and known by the Steadybit agent. Please check [Install Agent -> Extension Registration](/install-and-configure/install-agent/extension-registration) for details. Most extensions will require extension-specific configuration. For example, the Kong extension needs to know how to reach and authenticate with Kong API gateways, while the AWS extensions require the configuration of IAM roles. As a result, no single set of instructions covers all extensions. However, extensions' documentation should describe the exact configuration and installation steps. ## Manage your Extensions In the platform, you can manage your extensions via the extensions tab. You can see the extensions that are currently installed. Go to [Settings -> Extensions](https://platform.steadybit.com/settings/extensions) to see the list of installed extensions. ![Extensions Overview](/files/BxyGpCLsgEpG3YqnUJWd) # Extension Kits Steadybit's Extension Kits allow you to extend the Chaos Engineering capabilities by adding support for additional technologies or proprietary applications. So, you only need to know about these kits if you are considering authoring a custom extension. Steadybit provides development kits for the following purposes: * [ActionKit](#actionkit) to create your own custom Chaos Engineering attacks, checks for integrating, e.g., into observability or load test integration * [AdviceKit](#advicekit) to implement your own [advice](/use-steadybit/explorer/advice) and ease identifying experiments and tracking progress * [DiscoveryKit](#discoverykit) to support discovery of new infrastructure components * [EventKit](#eventkit) to send events about running experiments from Steadybit to other third-party applications (e.g., observability) * [ExtensionKit](#extensionkit) utility classes and best practices for extension authors using the Go programming language * [PreflightKit](#preflightkit) to implement preflight actions that can be used to prevent experiment executions All these development kits are agnostic to the programming language and rely on HTTP interfaces. However, if you want to develop your extension using Go, we recommend our [ExtensionKit](#extensionkit) to benefit from helpful utility classes and best practices. ## ActionKit
ActionKit logo depicting the text action kit and a crosshair icon

ActionKit logo depicting the text action kit and a crosshair icon

The Steadybit ActionKit enables the extension of Steadybit with new action capabilities that you can use within experiments. For example, ActionKit can be used to author open/closed source: * attacks to attack AWS, Azure, and Google Cloud services that Steadybit cannot natively attack, * integrate load testing tools, * health and state checks and * every other runnable action! You can learn more about ActionKit through its [GitHub repository](https://github.com/steadybit/action-kit). ## AdviceKit
AdviceKit logo depicting the text advice kit and a shield icon

AdviceKit logo depicting the text advice kit and a shield icon

The Steadybit AdviceKit enables the extension of Steadybit's advice for providing guidance on reliability configuration issues and suggesting appropriate experiments. You can learn more about AdviceKit through its [GitHub repository](https://github.com/steadybit/advice-kit). ## DiscoveryKit
DiscoveryKit logo depicting the text discovery kit and a radar dish icon

DiscoveryKit logo depicting the text discovery kit and a radar dish icon

The Steadybit DiscoveryKit enables the extension of Steadybit with new discovery capabilities. For example, DiscoveryKit can be used to author open/closed source discoveries for: * proprietary technology, * non-natively supported open-source tech, * hardware components and * every other *'thing'* you would want to see and attack with Steadybit. You can learn more about DiscoveryKit through its [GitHub repository](https://github.com/steadybit/discovery-kit). ## EventKit
EventKit logo depicting the text event kit and a bell icon

EventKit logo depicting the text event kit and a bell icon

EventKit allows extensions to consume events from the Steadybit platform to integrate with third-party systems. Extensions leveraging EventKit are similar to webhooks but do not face the typical web routing issues as Steadybit agents handle this aspect. You can use EventKit to: * Forward audit logs to an external system. * Add markers to monitoring systems' charts during experiment runs. * Capture experiment run statistics. * Report information about experiment runs to Slack, Discord etc. You can learn more about EventKit through its [GitHub repository](https://github.com/steadybit/event-kit). ## ExtensionKit
ExtensionKit logo depicting the text extension kit and a wrench icon

ExtensionKit logo depicting the text extension kit and a wrench icon

Through kits like ActionKit and DiscoveryKit, Steadybit can be extended with new capabilities. ExtensionKit on the other hand contains helpful utilities and best practices for extension authors leveraging the Go programming language. You can learn more about ExtensionKit through its [GitHub repository](https://github.com/steadybit/extension-kit). ## PreflightKit
PreflightKit logo depicting the text preflight kit and a preflight icon

PreflightKit logo depicting the text preflight kit and a preflight icon

PreflightKit enables the extension of Steadybit with new preflight capabilities that you can use to prevent experiment executions. For example, PreflightKit can be used to author open/closed source: * preflight actions to allow or disallow experiment executions based on the experiment and its targets, * preflight actions to prevent experiment executions based on the time of day / maintenance windows, * or anything else you can think of! You can learn more about PreflightKit through its [GitHub repository](https://github.com/steadybit/preflight-kit). # Hubs Steadybit's hubs are the home for the chaos engineering community! They allow everyone to browse and contribute open-source extensions and templates made for Steadybit. Steadybit hosts its own [Reliability Hub](https://hub.steadybit.com/) and always loves to see contributions via [pull requests](https://github.com/steadybit/reliability-hub-db). Steadybit's chaos engineering platform lets you connect to a hub to enable [importing experiment templates](/install-and-configure/manage-experiment-templates) or integrating [documentation of an action](/use-steadybit/experiments/design#action-and-template-documentation). ## Hub Connections Admins can manage connected hubs in the platform's settings. By default, the platform connects to our [Reliability Hub](https://hub.steadybit.com/). {% hint style="info" %} On-premise platform installations connect to a bundled local copy of our Reliability hub's content. Thus, air-gapped environments are supported even when reading data from the hub connection. {% endhint %} ### Connect New Hubs Steadybit supports connecting your own hub to the platform. An administrator can manage hub connections via `Settings`> `Hubs`. To add a hub, you need to specify the URL to the hub's index.json (see section [hub convention](#hub-connections)). ![Platform - connect new hub](/files/CZJd9sGz1bfzYgkcIwwE) Once you've added your hub, you can [import templates from the hub easily](/install-and-configure/manage-experiment-templates) and view documentation of actions integrated into the experiment editor. ### Disconnect Hub When you disconnect a connected hub, you can decide to remove imported templates. Experiments created from a template are never deleted when you disconnect a hub. ![Platform - disconnect hub](/files/bcKB4hpUAl5i00pTbdOk) ## Host Your Own Hub You can host your own hub to share content within your organization instead of with the public community. This is especially beneficial when you have developed a proprietary extension or need to share organization-specific templates. So far, we only support hosting a private hub's database and connecting it to the platform. We do not yet support hosting a white-labeled hub UI. ### Hub Convention To host your own hub, you have to serve a JSON-based endpoint via HTTP with a last modified unix timestamp (`lastChange`) and path references to templates and action documentation (see below). The platform will only update the referenced content when the unix timestamp (`lastChange`) has changed. ```json { "lastChange": 1719498293, "templates": [ "/templates/aws-zone.zone-outage/template.json", "/templates/kubernetes-deployment.time-to-readiness/template.json" //... ], "actions": [ "/actions/com.steadybit.extension_container.stress_cpu/description.yml", "/actions/com.steadybit.extension_host.shutdown/description.yml", //... ] } ``` You can check out the `index.json` and all referenced content of [Steadybit's Reliability Hub](https://hub.steadybit.com/) in our [open-source GitHub repository](https://github.com/steadybit/reliability-hub-db/blob/main/index.json). #### Experiment Templates Experiment templates, that can be [imported to your platform](/install-and-configure/manage-experiment-templates), are listed in the `templates` section and are referencing the [exported templates](/install-and-configure/manage-experiment-templates#export-templates-as-files) from your platform. #### Action Documentation The [documentation of an action](/use-steadybit/experiments/design#action-and-template-documentation) references a `description.yml` with the following format: ```yaml --- id: com.steadybit.extension_host.shutdown label: Trigger Shutdown Host description: Triggers a reboot or shutdown of the host. icon: | kind: attack category: state targetType: com.steadybit.extension_host.host extension: com.steadybit.extension_host promotedActions: - com.steadybit.extension_aws.ec2_instance.state tags: - Host - Kubernetes ``` The `id` needs to match the action-id provided by your extension. Also, in the same folder as the `description.yml`, the platform will look for a `summary.mdx` containing Markdown content, see for example: ```markdown # Introduction This action executes a shutdown on the host by issuing the `shutdown` command/syscall (depending on the operating system). You can instruct the action to issue a host restart when desired - in which case the command/syscall will be adapted as necessary. ... # Parameters | Parameter | Description | Default | |-----------|---------------------------------------------|---------| | Reboot | Should the host reboot after shutting down? | true | ``` # Preflight Actions {% hint style="info" %} Preflight actions are an enterprise feature. Please [reach out to us](https://steadybit.com/contact) if you want to get access. {% endhint %} Preflight actions are triggered by Steadybit whenever an experiment is about to start and allow you to prevent an experiment from running. To decide whether that specific experiment run is allowed to start, you get a list of all expected affected targets in the extension action call. Please note that, due to concurrency, these affected targets may change in case one of the targets is gone when the actual step starts or new ones are discovered. Preflight actions can additionally be used to perform checks during the experiment run, e.g., to check if a target is still healthy or if the environment is still in a state that allows the experiment to run. An interval can be configured as an additional trigger. ## Precondition You need to write your own extension with [preflight kit](https://github.com/steadybit/preflight-kit) added to it. Implement the preflight action at your needs. See [docs](https://github.com/steadybit/preflight-kit/blob/main/docs/preflight-api.md). ## Configure You can add preflight actions at `Settings` -> `Integrations` -> `Preflight Actions`. ![Add Preflight Action Integration](/files/lHTABxaIJ8al5fHVozgd) A preflight action integration has the following parameters to be specified: | | | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | The preflight action's name, it is shown in the experiment run. | | **Team** | If no team is specified, preflight actions will be performed for all teams. If you specify a team, preflight actions are only made for this team. | | **Preflight Action** | The Preflight Action to use from a registered extension | | **Triggers** | When the Preflight Action should trigger | | - *Before the experiment is started* | default - can't be turned off | | - *While the experiment is running* | The interval specifies the time Steadybit will wait between the end and the next execution of a preflight action. | ## Experiment Runs During the experiment run, you can see the triggered preflight actions. If an preflight action fails, the experiment run fails, and no targets are attacked. ![Preflight Action Success](/files/9qfwCLUErVbp23qPxGJV) ![Preflight Action Failure - Stopped Experiment Run](/files/Y9LhhgDi03qtdPVaevWJ) ## Developing actions A preflight action uses [preflight kit](https://github.com/steadybit/preflight-kit). See our [docs](https://github.com/steadybit/preflight-kit/blob/main/README.md#getting-started) to get started. ### Lifecycle of Preflight actions A preflight action can be in one of the following lifecycle statuses, indicated in the experiment run: | | | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **CREATED** | The preflight action was created and has sent the request to the configured action in the extension. It is still waiting for the response. | | **SUCCESSFUL** | The preflight action was resolved successfully. The experiment is allowed to continue (if all preflight actions are successful). | | **FAILED** | The preflight action resolved with a failure. The experiment will fail. Optionally, the response may contain a message as a reason for experiment failure. | | **ERRORED** | Technical error happened while requesting the preflight action, e.g., the extension URL couldn't be resolved, or the HTTP request timed out. | # Slack Notifications You want to receive a Slack message whenever an experiment has started or failed? This can be achieve via `Settings` -> `Integrations` -> `Slack`. > If you use an on-prem installation make sure to set the `STEADYBIT_WEB_PUBLIC_URL`, so we can add links to open Steadybit via links in the message. | | | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | The name for this integration will not show up in the message. | | **URL** | The Slack webhook URL, which you can create in Slack by navigating `Manage apps` -> `Custom Integrations` -> `Incoming Webhooks`. | | **Channel** |

The Slack channel which receives the message. If omitted the default from the Slack Webhook configuration will be used.
optional

| | **Icon Url** |

You may specify a different icon to be used for the message.
optional

| | **Team** | If no team is specified, you'll receieve all events. If you do specify a team you'll only receive notifications relevant for this team | | **Events** | You may select the events you want to recieve. | If you want, you can also use a [custom Webhook ](/integrate-with-steadybit/webhooks/custom-webhooks)to further customize Slack message or integrate with other chat applications. # Webhooks Steadybit can inform you about events via webhooks. These events can be, e.g., a newly created experiment, a running experiment, or an engaged killswitch. You can use a [Custom Webhook](/integrate-with-steadybit/webhooks/custom-webhooks) to integrate with other systems and, e.g., trigger an action whenever a new experiment was created. Alternatively, you can prevent an experiment from running or being scheduled by using [Preflight Webhooks](/integrate-with-steadybit/webhooks/preflight-webhooks). Find out how to make use of * [Custom Webhooks](/integrate-with-steadybit/webhooks/custom-webhooks) * [Preflight Webhooks](/integrate-with-steadybit/webhooks/preflight-webhooks) # Custom Webhooks Custom webhooks are triggered by Steadybit whenever an experiment has progressed or the killswitch's status changes. ## Configure You can configure custom webhooks at `Settings` -> `Integrations` -> `Custom webhook`. The content type is `application/json`, and the message is described in our [OpenAPI specification](https://platform.steadybit.com/api/spec) as `WebhookPayload`. | | | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | The name for this integration will not show up in the JSON body. | | **URL** | The URL, which will receive an HTTP Post request with the JSON body | | **Secret** |

You may specify a secret that will be used to sign the body. Verifying the signature.
optional

| | **Team** | If no team is specified, you'll receieve all events. If you do specify a team, you'll only receive events relevant to this team | | **Events** | Choose the events you want to receive. | ## Supported Events You can decide to react to the following events individually or get informed about all events. ### Experiment Executions The body contains the event identifier (`event`), the `time`, and the experiment `execution` (see below). | Event | Event Identifier | Description | | -------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Created** | `experiment.execution.created` | A new experiment execution was just started. As soon as all required agents have connected, the first step will be executed | | **Step Started** | `experiment.execution.step-started` | One step of a running experiment started. | | **Step Completed** | `experiment.execution.step-completed` | One step of a running experiment completed successfully (e.g. a check succeeded or the attack was performed). | | **Step Failed** | `experiment.execution.step-failed` | One step of a running experiment failed (e.g. a check didn't match the defined expectation). | | **Step Errored** | `experiment.execution.step-errored` | One step of a running experiment errored (e.g. a check or attack couldn't be executed due to a technical error). | | **Canceled** | `experiment.execution.step-canceled` | The experiment execution was canceled, e.g., by a user. | | **Completed** | `experiment.execution.completed` | The experiment completed succesfully, e.g., all steps have been completed successfully. | | **Failed** | `experiment.execution.failed` | The experiment execution failed because at least one step failed. | | **Errored** | `experiment.execution.errored` | The experiment execution errored because at least one step errored. | | **Preflight Checks** | `experiment.execution.preflight` | A [preflight webhook check](/integrate-with-steadybit/webhooks/preflight-webhooks) is performed before the experiment is allowed to start. | ### Killswitch The body contains the event identifier (`event`), the `time` and the `killswitch`'s state (see below). | Event | Event Identifier | Description | | ----------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | **Engaged Emergency Stop** | `killswitch.disengaged` | The [emergency stop](/use-steadybit/experiments/emergencystop) was triggered to stop all experiment runs and prevent future runs. | | **Disengaged Emergency Stop** | `killswitch.engaged` | The [emergency stop](/use-steadybit/experiments/emergencystop) was disengaged to allow future experiment runs. | ## Developing Webhooks A webhook uses an HTTP POST request at an endpoint reachable from the Steadybit platform. The HTTP request sends a body with the content-type `application/json`. Our [OpenAPI specification](https://platform.steadybit.com/api/spec) describes the exact body in `WebhookPayload`. ### Examples This section covers some example requests to ease developing the webhook endpoint. #### Experiment Execution Created The request body that is sent to your endpoint depends on the specific experiment that is executed. ```bash curl --request POST \ --url https:// \ --header 'accept: */*' \ --header 'accept-encoding: gzip' \ --header 'content-type: application/json' \ --data '{ "event": "experiment.execution.created", "time": "2024-01-01T00:00:00.000000000Z", "execution": { "id": 78151, "experimentKey": "ADM-1", "teamKey": "ADM", "environment": "a7efb4f1-9b8e-46a7-a3eb-ffc29ba75eae", "name": "Stress CPU", "created": "2024-01-01T00:00:00.000000Z", "createdVia": "UI", "experimentVersion": 0, "state": "CREATED", "steps": [ { "id": "0192fc6e-b1b8-7593-a2d6-ed9443172ba0", "state": "CREATED", "ignoreFailure": false, "parameters": { "cpuLoad": "1", "workers": 0, "duration": "30s" }, "actionId": "com.steadybit.extension_container.stress_cpu", "actionKind": "ATTACK", "radius": { "targetType": "com.steadybit.extension_container.container", "percentage": 100, "predicate": { "operator": "AND", "predicates": [ { "key": "k8s.deployment", "operator": "EQUALS", "values": [ "gateway" ] } ] } }, "targetExecutions": [ { "type": "com.steadybit.extension_container.container", "name": "aa815cb361e65806a6938670d71a7af94666cf3a08acce990c1d131ae4533b07", "attributes": [ { "key": "aws.account", "value": "244471902119" } ] } ], "totalTargetCount": 1 } ] } }' ``` #### Killswitch Disengaged ```bash curl --request POST \ --url https:// \ --header 'accept: */*' \ --header 'accept-encoding: gzip' \ --header 'content-type: application/json' \ --data '{ "event": "killswitch.disengaged", "time": "2024-01-01T01:00:00.000000Z", "killswitch": { "engagedBy": "99c9ed40-6b85-439c-bd14-04787f2443d2", "engaged": "2024-01-01T00:00:00.000000Z", "disengagedBy": "e7ff1799-0430-4b1f-a835-c383b2a19543", "disengaged": "2024-01-01T01:00:00.000000Z" } }' ``` ### Verify Webhook Requests If a secret is provided a signature of the body is computed using `HMAC SHA-256` and sent as `X-SB-Signature` http header. You can use this header to verify the message. Here is an example of doing this in Java: ```java private static boolean validateSignature(byte[] body, String secret, String header) throws Exception { //calculate the signature using the secret Mac mac = Mac.getInstance("HmacSHA256"); mac.init(new SecretKeySpec(secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256")); byte[] signature = mac.doFinal(body); //remove the algorithm prefix and decode the hex to bytes[] byte[] receivedSignature = Hex.decode(header.replaceFirst("^hmac-sha256 ", "")); //compare using time-constant algorithm return MessageDigest.isEqual(signature, receivedSignature); } ``` # Preflight Webhooks {% hint style="info" %} Preflight webhooks are an enterprise feature. Please [reach out to us](https://steadybit.com/contact) if you want to get access. {% endhint %} Preflight webhooks are triggered by Steadybit whenever an experiment is about to start and allow you to prevent an experiment from running. To decide whether that specific experiment run is allowed to start, you get a list of all expected affected targets in the webhook call. Please note that, due to concurrency, these affected targets may change in case one of the targets is gone when the actual step starts or new ones are discovered. ## Configure You can add preflight webhooks at `Settings` -> `Integrations` -> `Preflight webhook`. ![Add Preflight Webhook](/files/ZDA3OoVkueAq0zOkctKj) A webhook has the following parameters to be specified: | | | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | The preflight webhook's name, it is shown in the experiment run. | | **URL** | The URL, which will receive an HTTP Post request with the HTTP request body | | **Secret** | **optional**You may specify a secret which is used to sign the body to [verify the webhook request](#verify-webhook-requests). | | **Team** | If no team is specified, preflight checks will be performed for all teams. If you specify a team, preflight checks are only made for this team. | | **Events** | Right now, there is only one event here: `Execution preflight checks`, which is triggered before starting an experiment run. | ## Experiment Runs During the experiment run, you can see the triggered preflight webhooks. If a webhook fails, the experiment run fails, and no targets are attacked. ![Preflight Webhook Success](/files/9qfwCLUErVbp23qPxGJV) ![Preflight Webhook Failure - Stopped Experiment Run](/files/XXmzsUDT9SZp6Hztshui) ## Developing Webhooks A webhook uses an HTTP POST request at an endpoint reachable from the Steadybit platform. The HTTP request sends a body with the content-type `application/json`. Our [OpenAPI specification](https://platform.steadybit.com/api/spec) describes the exact body in `WebhookPayload`. The webhook must return an HTTP status 2xx to allow an experiment to run. The experiment will not run if the webhook returns an HTTP status code other than 2xx. Optionally, you can return a `message` in the HTTP response to show why the experiment isn't allowed to start. The response body can be found in the [OpenAPI specification](https://platform.steadybit.com/api/spec) as `PreflightWebhookResponseAO`. ### Lifecycle of Preflight Webhooks A preflight webhook can be in one of the following lifecycle statuses, indicated in the experiment run: | | | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **CREATED** | The preflight webhook was created and has sent the request to the configured webhook. It is still waiting for the response. | | **SUCCESSFUL** | The preflight webhook was resolved successfully with an HTTP status code 2xx. The experiment is allowed to continue (if all preflight webhooks are successful). | | **FAILED** | The preflight webhook resolved with a non-2xx HTTP status code. The experiment will fail. Optionally, the response may contain a message as a reason for experiment failure. | | **ERRORED** | Technical error happened while requesting the HTTP endpoint, e.g., the URL couldn't be resolved, or the HTTP request timed out. | {% hint style="info" %} A webhook will timeout after 55 seconds. In that case, the preflight check is marked as `ERRORED`, and the experiment will not start. If the webhook resolves later, the actual result will be submitted to the preflight check step in the experiment. {% endhint %} ### Examples This section covers some example requests to ease the development of the preflight webhook endpoint. The request body sent to your endpoint depends on the experiment that is tried to be executed. #### Request The below curl command can be used to mock a preflight request from Steadybit to your endpoint. Please note, that some of the metadata (i.e., step's `parameters`, and `targetExecutions`' `attributes`) have been omitted. ```bash curl --request POST \ --url https:// \ --header 'accept: */*' \ --header 'accept-encoding: gzip' \ --header 'content-type: application/json' \ --data '{ "event": "experiment.execution.preflight", "time": "2024-11-05T10:51:31.586222005Z", "execution": { "id": 60963, "experimentKey": "GITHUB-42", "teamKey": "GITHUB", "environment": "de75d3d2-6e23-4b9a-b5b5-0df898bd039e", "name": "Copy of AWS Zone Outage of eu-central-1a for toys-bestseller", "hypothesis": "When AWS Availability Zone eu-central-1a is down for toys-bestseller, Kubernetes manages this accordingly by routing the traffic within expected failure rates so that the offered features still work. As soon as the zone is available again, the pod is ready within 60s.", "created": "2024-11-05T10:51:31.316731Z", "createdVia": "UI", "experimentVersion": 1, "state": "CREATED", "steps": [ { "id": "a8f99390-b906-43b5-b456-becacd295304", "state": "CREATED", "ignoreFailure": false, "parameters": { "url": "https://demo.steadybit.io/products", "method": "GET", "duration": "140s", }, "customLabel": "HTTP Endpoint works all the time", "actionId": "com.steadybit.extension_http.check.periodically", "actionKind": "CHECK", "radius": {}, "targetExecutions": [ { "type": "agent", "name": "prod-demo/steadybit-agent/steadybit-agent-0", "attributes": [ { "key": "agent.hostname", "value": "prod-demo/steadybit-agent/steadybit-agent-0" } ] } ], "totalTargetCount": 1 } { "id": "0192fbf3-7c7c-701b-af5a-8bbc567b7b9f", "state": "PREPARED", "ignoreFailure": false, "parameters": { "duration": "20s" }, "customLabel": "Wait for AWS Zone outage" }, { "id": "0192fbf3-7c7c-701b-af5a-8bbc567b7ba0", "predecessorId": "0192fbf3-7c7c-701b-af5a-8bbc567b7b9f", "state": "CREATED", "ignoreFailure": false, "parameters": { "ip": [], "port": [], "duration": "60s", "hostname": [], "failOnHostNetwork": true }, "customLabel": "WHEN: Zone outage of eu-central-1a for toys-bestseller", "actionId": "com.steadybit.extension_container.network_blackhole", "actionKind": "ATTACK", "radius": { "targetType": "com.steadybit.extension_container.container", "percentage": 50, "predicate": { "operator": "AND", "predicates": [ { "operator": "AND", "predicates": [ { "key": "aws.zone", "operator": "EQUALS", "values": [ "eu-central-1a" ] } ] } ] } }, "targetExecutions": [ { "type": "com.steadybit.extension_container.container", "name": "411f01843ca1fbe53a7d30367acf25f0a79242659ffc36b56516b880d2731847", "attributes": [ { "key": "aws.account", "value": "111111111111" }, { "key": "k8s.label.service-tier", "value": "2" }, { "key": "aws.zone", "value": "eu-central-1a" } ] } ], "totalTargetCount": 1 } ] } }' ``` #### Response: Allow Run The webhook must return an HTTP status 2xx to allow an experiment to run. Optionally, you can return a `message` in the HTTP response to show details on why the experiment is allowed to start. For example: ```json {"message": "Jane Doe authorized this experiment because it runs in a permitted execution window."} ``` #### Response: Disallow Run To disallow an experiment to run, the webhook must return an HTTP status code other than 2xx. Optionally, you can return a `message` in the HTTP response to show why the experiment isn't allowed to start. ```json {"message": "Jane Doe rejected this experiment run because it needs to run in a permitted execution window."} ``` #### Response: Property Modifications Preflight Webhooks can also modify properties of the experiment execution by returning `modifications` in the response body. Examples: ```json {"modifications": [ { "type": "set_property_value", "propertyKey": "approvedBy", "value": "Daniel" }, { "type": "add_value_to_list_property", "propertyKey": "observations", "value": "This looks interesting!" } ]} ``` Currently, two modification types are supported: * `set_property_value`: Sets the value of a property identified by `propertyKey` to the provided `value`. If the property does not exist, it will be added. * `add_value_to_list_property`: Adds the provided `value` to a list property identified by `propertyKey`. If the property does not exist, it will be added. If it exists but is not a list, the execution will fail. Properties needs to be `editableInExecution` if inherited from the experiment design. You can learn more about properties [here](/install-and-configure/manage-properties). ### Verify Webhook Requests You can verify that the call to the preflight webhook is legitimate by verifying the signature. as soon as the webhook's optional secret is configured. To do that, you need to configure the optional webhook's secret. The body's signature is computed using `HMAC SHA-256` and sent as an `X-SB-Signature` HTTP header. You can use this header to verify the message. Here is an example of doing this in Java: ```java private static boolean validateSignature(byte[]body,String secret,String header)throws Exception{ //calculate the signature using the secret Mac mac=Mac.getInstance("HmacSHA256"); mac.init(new SecretKeySpec(secret.getBytes(StandardCharsets.UTF_8),"HmacSHA256")); byte[]signature=mac.doFinal(body); //remove the algorithm prefix and decode the hex to bytes[] byte[]receivedSignature=Hex.decode(header.replaceFirst("^hmac-sha256 ","")); //compare using time-constant algorithm return MessageDigest.isEqual(signature,receivedSignature); } ``` # How to troubleshoot If you need help finding and resolving problems within Steadybit, you can look into your tenant database and metrics or enable our technical support team to assist you. ## Database export By downloading a copy of your Steadybit tenant database, you'll be able to analyze it using a third-party application. 1. In the Steadybit platform, go to "Settings."
Screenshot of the Settings menu in the Steadybit platform
2. Navigate to the "Troubleshooting" menu.
Screenshot of the Troubleshooting menu in the Steadybit platform
3. On the "Database export" section, choose "Database export (.zip)."
Screenshot of the Database export section in the Steadybit platform
You'll find a ZIP file in your local downloads folder with a copy of your tenant database. ## Technical support by Steadybit Steadybit's technical support team can log in to your tenant account to investigate and fix problems you might experience within the platform. But they can only do that if you allow it. {% hint style="info" %} We can only view your data and never change it or run an experiment through your tenant. And the audit log records any login and logout activity of our team in your account. Besides, you can withdraw the authorization at any time. {% endhint %} 1. In the Steadybit platform, go to "Settings." 2. Navigate to the "Troubleshooting" menu. 3. Click the "Let Steadybit troubleshoot your tenant" button.
Screenshot of the Steadybit troubleshoot your tenant button in the Steadybit platform
## Further help If you still need help resolving a problem within Steadybit, take a look at the common fixes linked below. * [Extensions](/troubleshooting/common-fixes/extensions) * [Agents](/troubleshooting/common-fixes/agents) * [On-prem platform](/troubleshooting/common-fixes/on-prem-platform) For more help, [contact us](https://www.steadybit.com/contact). # Common fixes # Extensions #### Why is the extension-container failing due to failed volume mounting? When deploying the extension-container, the extension startup may fail with an error message as shown below. ``` MountVolume.SetUp failed for volume "..." : hostPath type check failed: /run/.../runc/k8s.io is not a directory ``` This error indicates that the extension runs on the wrong container runtime (Docker, cri-o, or containerd). To fix this issue, you must configure another one in the installation. You can change the extension's runtime easily via helm parameter. If you've used the Agent helm-chart to deploy Agent and extensions, you can configure the `extension-container.container.engine` parameter. ``` --set extension-container.container.engine=... //containerd, docker or cri-io ``` If you deployed the extension-container standalone, the parameter's name is `container.engine` (without the `extension-container` prefix). #### My nodes report that they run on containerd but the extension-container fails with the error message `SetUp failed for volume "runtime-socket" : hostPath type check failed: /run/containerd/containerd.sock is not a socket file` You might run on k3s, which uses containerd as the container runtime, but has a different path for the containerd socket. You can fix this issue by setting the correct path to the containerd socket via the `extension-container.containerEngines.containerd.socket` parameter. ``` --set extension-container.containerEngines.containerd.socket=/var/run/k3s/containerd/containerd.sock ``` #### Why can't I install the extensions container, host or jvm on my Kubernetes cluster? If you are using for example GKE Autopilot, you might not be able to install the extension container, host or jvm on your Kubernetes cluster. This is because the extension needs to run with special privileges and this is not possible on GKE Autopilot. Please use a GKE Standard cluster instead. The error could look like this: ``` autopilot-default-resources-mutator:Autopilot updated DaemonSet steadybit-agent/steadybit-agent-extension-host: adjusted resources to meet requirements for containers [extension-host] (see http://g.co/gke/autopilot-resources) Violations details: {"[denied by autogke-default-linux-capabilities]":["linux capability 'SYS_ADMIN,SYS_RESOURCE,SYS_BOOT,SYS_TIME,NET_ADMIN' on container 'extension-host' not allowed; Autopilot only allows the capabilities: 'AUDIT_WRITE,CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,MKNOD,NET_BIND_SERVICE,NET_RAW,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT,SYS_PTRACE'."],"[denied by autogke-disallow-hostnamespaces]":["enabling hostNetwork is not allowed in Autopilot.","enabling hostPID is not allowed in Autopilot."],"[denied by autogke-no-host-port]" ``` #### Why can't I install the extensions-container on my Docker-Desktop with Kubernetes enabled? Docker-Desktop is not supported by the extensions-container. Please use minikube instead. [Minikube](https://minikube.sigs.k8s.io/docs/start/) The error could look like this: ``` Warning FailedMount 11s (x6 over 26s) kubelet MountVolume.SetUp failed for volume "runtime-runc-root" : hostPath type check failed: /run/docker/runtime-runc/moby is not a directory ``` #### We see intermittent OOMs on extensions or timeouts due to high CPU usage. This makes the extensions unreliable. We aim for sane defaults in our Helm Charts regarding CPU and Memory requests/limits and design the extensions to have a low resource consumption. However, resource usage depends on the size of your environment. You need to increase the settings if you have massive Kubernetes clusters or big container hosts.\ \ Running observability tools that instrument your applications by injecting processes (e.g., Dynatrace) or manipulating bytecode (e.g., Instana) can lead to increased resource consumption. You might want to exclude the Steadybit Agent and Extensions from this, or need to adapt your resource requests. #### Extension-jvm warning about `Dynamic loading of agents` To be able to discover we need access to the java process. This is done by using the `attach` mechanism of the JVM. This is a standard mechanism and is used by many other tools. You may see a warning in the logs of the JVM that the extension is attaching to the JVM. This is normal and expected. It will look like this: ``` WARNING: A Java agent has been loaded dynamically (...javaagent-init.jar) WARNING: Dynamic loading of agents will be disallowed by default in a future release ``` To avoid this warning or be able to use this extension in future java releases you can use the `-XX:+EnableDynamicAgentLoading` flag in your JVM commandline to be able to load the javaagent dynamically. #### I'm seeing unversioned extension content in my platform. How do I resolve this? This can occur when an extension is deployed using a non-released version, such as SNAPSHOT or a main branch build. In these cases, the platform registers the extension without a fixed version identifier. Once you switch back to a released version, the previously registered unversioned entries are not automatically removed. To resolve this, you need to clean them up manually by going to `settings` → `extensions` → `actions`, check the unversioned actions and choose `delete selected actions`. After deletion, the platform will remove the unversioned action definitions. Affected agents will automatically re-submit the correct, versioned action definitions. ![extension-unversioned.jpg](/files/rhKHgbYRWF1cBlFLuB50) # Agents ### We get certificate errors when the agent connects to our on-premise Steadybit installation. What do we need to do? The most common reasons for connectivity issues are related to: * Self-signed certificates that are not trusted. * Root certificate authorities that are not trusted. * Incomplete certificate chains. At runtime, this will typically manifest in the agent through log entries like this: ``` 2022-03-14T20:31:30.562 WARN [istration-task-1] | AgentRegistrator | agent-platform-connector - 0.7.17 | Failed to register agent: I/O error on POST request for "https://steadybit.acme.example.com/api/agents": PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target ``` To analyze the situation from your end, we recommend starting by ensuring that TLS is correctly configured. You can do so via the `openssl` command-line tool. ``` openssl s_client -showcerts -connect {{YOUR DOMAIN HERE}}:443 -servername {{YOUR DOMAIN HERE}} ``` This should print a whole lot of information. Most relevant are the `Verify return code:` sections. These sections should always report `Verify return code: 0 (ok)`. For other return codes, please check out the following sub-sections: **Verify return code: 19 (self-signed certificate in certificate chain)** This happens for self-signed certificates. You can typically resolve these problems through the `STEADYBIT_AGENT_EXTRA_CERTS_PATH` environment variable. This environment variable should point to a directory containing your root certificates. The agent will load all certificates in this directory into the Java key store upon agent startup. This, in turn, makes the agent trust your custom certificates. You can also check the environment variable's effect by adding the `-CAfile root-certificate.pem` command-line argument to the `openssl` command shown in the previous section. **Verify return code: 21 (unable to verify the first certificate)** The agent requires a complete certificate chain configuration for security reasons and this return code indicates that your server responded with an incomplete certificate chain. You can fix this issue by modifying the server that terminates the TLS connection. Please refer to your server/proxy/CDN documentation to learn how to configure a complete certificate chain. ### Installation agent on AWS EKS cluster 1. First: make sure to configure the [Amazon-EBS-CSI-Driver](https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html#adding-ebs-csi-eks-add-on) 2. Afterwards add the Amazon-EBS-CSI-Driver addon on your EKS cluster, with newly created IAM role 3. Then add your first node group to the cluster. ### Occasional connection timeouts on the agent -> extension discovery calls, which cause remove targets from discovery We are using resilience4j for the retry mechanism. The default configuration is to retry 3 times with a wait duration of 30s with an exponential backoff multiplier of 2. This means that the first retry will be after 30s, the second after 60s, and the third after 120s. If all retries fail, the agent will remove the target from the discovery. You can configure the retry mechanism by setting the following environment variables: | Environment Variable | Description | | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `RESILIENCE4J.RETRY_INSTANCES_HTTPDISCOVERY_MAXATTEMPTS` | Optional - Resilience4j: The maximum number of attempts (including the initial call as the first attempt) for DiscoveryKit resources | | `RESILIENCE4J.RETRY_INSTANCES_HTTPDISCOVERY_WAITDURATION` | Optional - Resilience4j: A fixed wait duration between retry attempts for DiscoveryKit resources | | `RESILIENCE4J.RETRY_INSTANCES_HTTPDISCOVERY_ENABLEEXPONENTIALBACKOFF` | Optional - Resilience4j: Enable or disable exponential backoff for DiscoveryKit resources | | `RESILIENCE4J.RETRY_INSTANCES_HTTPDISCOVERY_EXPONENTIALBACKOFFMULTIPLIER` | Optional - Resilience4j: The multiplier for exponential backoff for DiscoveryKit resources | | `STEADYBIT_AGENT_HTTP_DISCOVERY_USE_RETRY` | Optional - Resilience4j: Enable/Disable the retry mechanism. Default is true / enabled | ### Agent takes a long time registering the extensions and to submit the first targets In a very large cluster it might take a while to read all pods in your cluster and scan them for extensions. You can limit the extension auto-registration to a single namespace using the environment variable `STEADYBIT_AGENT_EXTENSIONS_AUTOREGISTRATION_NAMESPACE` (helm-value `agent.extensions.autoregistration.namespace`). ### Install Agent and extension-kubernetes in a managed Kubernetes cluster where you are only allowed to deploy to one namespace Install the agent/extension with the following helm settings to use roles instead of clusterroles: ```shell --set rbac.roleKind="role" \ --set agent.extensions.autoregistration.namespace= \ --set extension-kubernetes.role.create=true \ --set extension-kubernetes.roleBinding.create=true \ --set extension-kubernetes.clusterRole.create=false \ --set extension-kubernetes.clusterRoleBinding.create=false \ ``` Full example: ```shell helm upgrade steadybit-agent --install --namespace \ --create-namespace \ --set agent.key="" \ --set global.clusterName="" \ --set extension-container.container.engine="" \ --set agent.registerUrl=""\ --set rbac.roleKind="role" \ --set agent.extensions.autoregistration.namespace="" \ --set extension-kubernetes.role.create=true \ --set extension-kubernetes.roleBinding.create=true \ --set extension-kubernetes.clusterRole.create=false \ --set extension-kubernetes.clusterRoleBinding.create=false \ steadybit/steadybit-agent ``` # On-prem platform {% hint style="info" %} This part of the documentation is only intended in the context of a supported PoC (Proof of Concept) together with the Steadybit team. Please, [book an appointment](https://www.steadybit.com/book-demo) to scope your PoC before continuing to evaluate the on-prem solution. If you just want to try out Steadybit, we recommend you [sign up for our SaaS platform](https://signup.steadybit.com). {% endhint %} This page describes some common issues and how to solve them. ### Pulling the platform images fails with `unauthorized: authentication required` or pods are in ImagePullBackOff The platform images on `docker.steadybit.io` require authentication. The username is `_` and the password is your agent key. * When deploying via our Helm chart, the image pull secret is created automatically from `platform.tenant.agentKey`. Verify that the agent key is set correctly: ```bash kubectl get secret -n steadybit-platform steadybit-platform-pull-secrets -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d ``` * When pulling images manually, e.g., on plain Docker hosts or to mirror them into an internal registry, log in first: ```bash docker login docker.steadybit.io --username _ --password ``` ### Platform and Postgres are in CrashLoopBackOff * Check the logs of the platform and Postgres containers ```bash kubectl logs -f -n steadybit-platform steadybit-platform-postgresql-0 --previous kubectl logs -f -n steadybit-platform steadybit-platform-0 --previous ``` * Verify that the Postgres password is correct and base64 encoded in the manifest file ### Create Heap dump Prerequisites: * If the platform is launched under docker, you need to have a dedicated volume or use an existing one for this mount path. * A network access to the host machine to retrieve the file. The platform can suffer from out of memory issues at JVM level. If that's happen, a heap dump might be needed to diagnose further, for providing it, add this environment variable : ```yaml - name: JAVA_TOOL_OPTIONS value: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/PATH_TO_BE_MOUNTED/heapdump-%p.hprof ``` Then you need to retrieve the heat dump. Usually by copying the file from the destination to your machine : ```bash scp ec2-user@1.2.3.4:/PATH_TO_BE_MOUNTED/heapdump-*.hprof /tmp/ ``` ### Kubernetes namespaces and deployments show up multiple times in the landscape table * Check the logs of the agents ```bash kubectl logs -f -n steadybit-agent steadybit-agent-0 ``` * If you see this error: `Missing permissions to create leases for the leader elections` or `Cannot perform leader election. All agents will behave as leader.` * Check if the agent has the correct permissions to create leases. ```bash NAME APIGROUP↑ GET LIST WATCH CREATE PATCH UPDATE DELETE DEL-LIST leases coordination.k8s.io ✓ ✓ ✓ ✓ × ✓ × × ``` ### Agents are not able to connect to the platform during an experiment * Check if you can reach the platform from the agent: ```bash kubectl exec -it -n steadybit-agent steadybit-agent-0 -- curl -k https://steadybit-platform.steadybit-platform.svc.cluster.local:8080 ``` * Check if the agent can reach the Websocket port of the platform. * This is usally port 7878 and can be configured in the platform manifest via environment variable `STEADYBIT_WEB_PUBLIC_EXPERIMENT_PORT` (helm chart: `platform.publicWebsocketPort`) * If setting the port is not enough, you can set the url via environment variable `STEADYBIT_WEB_PUBLIC_EXPERIMENT_URL` (helm chart: `platform.ingressOrigin`) * Please also check your ingress configuration. ```yaml spec: rules: - http: paths: - pathType: Prefix path: /ws backend: service: name: steadybit-platform port: number: 7878 - pathType: Prefix path: / backend: service: name: steadybit-platform port: number: 80 ``` * You can try to connect to the websocket port via curl: ```bash curl 'https://platform.steadybit.com:443/ws' \ -H 'Upgrade: websocket' \ -H 'Connection: Upgrade' \ -H 'Sec-WebSocket-Key: dummy' \ -H 'Sec-WebSocket-Version: 13' \ -v --http1.1 ``` ``` > GET /ws HTTP/1.1 > Host: platform.steadybit.com > User-Agent: curl/8.1.2 > Accept: */* > Upgrade: websocket > Connection: Upgrade > Sec-WebSocket-Key: dummy > Sec-WebSocket-Version: 13 > < HTTP/1.1 101 Switching Protocols < Date: Thu, 14 Dec 2023 14:37:03 GMT < Connection: upgrade < upgrade: websocket ``` ### Platform is behind Nginx and the agents are not able to connect to the platform Error message in the platform logs: ``` java.lang.IllegalArgumentException: Invalid character found in method name [0x160x030.....[us ]. HTTP method names must be tokens at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:407) ``` Solution: * set the nginx backend protocol is HTTPS instead of HTTP ### Configured the Platform with a oidc provider and the redirect to the platform is been send as http instead of https Example error message in the browser: ``` The redirect URI 'http://steadybit-platform.example.com/oauth2/login/code/default' specified in the request does not match the redirect URIs configured for the application 'xyz'. Make sure the redirect URI sent in the request matches one added to your application. ``` Solution: Set the environment variable: server.tomcat.remoteip.trusted-proxies to a regex that matches the CIDRs of the loadbalancer or reverse proxy. Add the following environment variable to the platform manifest: (Example for Google Cloud Load Balancer CIRDs regex) ```yaml env: - name: server.tomcat.remoteip.trusted-proxies value: "(35\\.191\\.(?:[0-9]|[1-9][0-9]|1(?:[0-9][0-9])|2(?:[0-4][0-9]|5[0-5]))\\.(?:[0-9]|[1-9][0-9]|1(?:[0-9][0-9])|2(?:[0-4][0-9]|5[0-5])))|(130\\.211\\.(?:[0-3])\\.(?:[0-9]|[1-9][0-9]|1(?:[0-9][0-9])|2(?:[0-4][0-9]|5[0-5])))" ```