On-prem platform

This part of the documentation is only intended in the context of a supported PoC (Proof of Concept) together with the Steadybit team. Please, book an appointment to scope your PoC before continuing to evaluate the on-prem solution.

If you just want to try out Steadybit, we recommend you sign up for our SaaS platform.

This page describes some common issues and how to solve them.

Platform and Postgres are in CrashLoopBackOff

• Check the logs of the platform and Postgres containers

    kubectl logs -f -n steadybit-platform steadybit-platform-postgresql-0 --previous
    kubectl logs -f -n steadybit-platform steadybit-platform-0 --previous

• Verify that the Postgres password is correct and base64 encoded in the manifest file

Create Heap dump

Prerequisites:

• If the platform is launched under docker, you need to have a dedicated volume or use an existing one for this mount path.

• A network access to the host machine to retrieve the file.

The platform can suffer from out of memory issues at JVM level. If that's happen, a heap dump might be needed to diagnose further, for providing it, add this environment variable :

- name: JAVA_TOOL_OPTIONS
  value: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/PATH_TO_BE_MOUNTED/heapdump-%p.hprof

Then you need to retrieve the heat dump. Usually by copying the file from the destination to your machine :

scp ec2-user@1.2.3.4:/PATH_TO_BE_MOUNTED/heapdump-*.hprof /tmp/

Kubernetes namespaces and deployments show up multiple times in the landscape table

• Check the logs of the agents

    kubectl logs -f -n steadybit-agent steadybit-agent-0

• If you see this error: Missing permissions to create leases for the leader elections or Cannot perform leader election. All agents will behave as leader.

  • Check if the agent has the correct permissions to create leases.

  NAME            APIGROUP↑                    GET            LIST           WATCH          CREATE         PATCH          UPDATE          DELETE          DEL-LIST
  leases          coordination.k8s.io           ✓              ✓              ✓              ✓              ×              ✓               ×               ×

Agents are not able to connect to the platform during an experiment

• Check if you can reach the platform from the agent:

    kubectl exec -it -n steadybit-agent steadybit-agent-0 -- curl -k https://steadybit-platform.steadybit-platform.svc.cluster.local:8080

• Check if the agent can reach the Websocket port of the platform.

  • This is usally port 7878 and can be configured in the platform manifest via environment variable STEADYBIT_WEB_PUBLIC_EXPERIMENT_PORT (helm chart: platform.publicWebsocketPort)

  • If setting the port is not enough, you can set the url via environment variable STEADYBIT_WEB_PUBLIC_EXPERIMENT_URL (helm chart: platform.ingressOrigin)

  • Please also check your ingress configuration.

  spec:
  rules:
    - http:
        paths:
          - pathType: Prefix
            path: /ws
            backend:
              service:
                name: steadybit-platform
                port:
                  number: 7878
          - pathType: Prefix
            path: /
            backend:
              service:
                name: steadybit-platform
                port:
                  number: 80

  • You can try to connect to the websocket port via curl:

  curl 'https://platform.steadybit.com:443/ws' \
    -H 'Upgrade: websocket' \
    -H 'Connection: Upgrade' \
    -H 'Sec-WebSocket-Key: dummy' \
    -H 'Sec-WebSocket-Version: 13' \
    -v --http1.1

  > GET /ws HTTP/1.1
  > Host: platform.steadybit.com
  > User-Agent: curl/8.1.2
  > Accept: */*
  > Upgrade: websocket
  > Connection: Upgrade
  > Sec-WebSocket-Key: dummy
  > Sec-WebSocket-Version: 13
  >
  < HTTP/1.1 101 Switching Protocols
  < Date: Thu, 14 Dec 2023 14:37:03 GMT
  < Connection: upgrade
  < upgrade: websocket

Platform is behind Nginx and the agents are not able to connect to the platform

Error message in the platform logs:

java.lang.IllegalArgumentException: Invalid character found in method name [0x160x030.....[us ]. HTTP method names must be tokens
	at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:407)

Solution:

• set the nginx backend protocol is HTTPS instead of HTTP

Configured the Platform with a oidc provider and the redirect to the platform is been send as http instead of https

Example error message in the browser:

The redirect URI 'http://steadybit-platform.example.com/oauth2/login/code/default' specified in the request does not match the redirect URIs configured for the application 'xyz'. Make sure the redirect URI sent in the request matches one added to your application.

Solution:

Set the environment variable: server.tomcat.remoteip.trusted-proxies to a regex that matches the CIDRs of the loadbalancer or reverse proxy. Add the following environment variable to the platform manifest: (Example for Google Cloud Load Balancer CIRDs regex)

env:
  - name: server.tomcat.remoteip.trusted-proxies
    value: "(35\\.191\\.(?:[0-9]|[1-9][0-9]|1(?:[0-9][0-9])|2(?:[0-4][0-9]|5[0-5]))\\.(?:[0-9]|[1-9][0-9]|1(?:[0-9][0-9])|2(?:[0-4][0-9]|5[0-5])))|(130\\.211\\.(?:[0-3])\\.(?:[0-9]|[1-9][0-9]|1(?:[0-9][0-9])|2(?:[0-4][0-9]|5[0-5])))"