# On-prem platform {% hint style="info" %} This part of the documentation is only intended in the context of a supported PoC (Proof of Concept) together with the Steadybit team. Please, [book an appointment](https://www.steadybit.com/book-demo) to scope your PoC before continuing to evaluate the on-prem solution. If you just want to try out Steadybit, we recommend you [sign up for our SaaS platform](https://signup.steadybit.com). {% endhint %} This page describes some common issues and how to solve them. ### Platform and Postgres are in CrashLoopBackOff * Check the logs of the platform and Postgres containers ```bash kubectl logs -f -n steadybit-platform steadybit-platform-postgresql-0 --previous kubectl logs -f -n steadybit-platform steadybit-platform-0 --previous ``` * Verify that the Postgres password is correct and base64 encoded in the manifest file ### Create Heap dump Prerequisites: * If the platform is launched under docker, you need to have a dedicated volume or use an existing one for this mount path. * A network access to the host machine to retrieve the file. The platform can suffer from out of memory issues at JVM level. If that's happen, a heap dump might be needed to diagnose further, for providing it, add this environment variable : ```yaml - name: JAVA_TOOL_OPTIONS value: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/PATH_TO_BE_MOUNTED/heapdump-%p.hprof ``` Then you need to retrieve the heat dump. Usually by copying the file from the destination to your machine : ```bash scp ec2-user@1.2.3.4:/PATH_TO_BE_MOUNTED/heapdump-*.hprof /tmp/ ``` ### Kubernetes namespaces and deployments show up multiple times in the landscape table * Check the logs of the agents ```bash kubectl logs -f -n steadybit-agent steadybit-agent-0 ``` * If you see this error: `Missing permissions to create leases for the leader elections` or `Cannot perform leader election. All agents will behave as leader.` * Check if the agent has the correct permissions to create leases. ```bash NAME APIGROUP↑ GET LIST WATCH CREATE PATCH UPDATE DELETE DEL-LIST leases coordination.k8s.io ✓ ✓ ✓ ✓ × ✓ × × ``` ### Agents are not able to connect to the platform during an experiment * Check if you can reach the platform from the agent: ```bash kubectl exec -it -n steadybit-agent steadybit-agent-0 -- curl -k https://steadybit-platform.steadybit-platform.svc.cluster.local:8080 ``` * Check if the agent can reach the Websocket port of the platform. * This is usally port 7878 and can be configured in the platform manifest via environment variable `STEADYBIT_WEB_PUBLIC_EXPERIMENT_PORT` (helm chart: `platform.publicWebsocketPort`) * If setting the port is not enough, you can set the url via environment variable `STEADYBIT_WEB_PUBLIC_EXPERIMENT_URL` (helm chart: `platform.ingressOrigin`) * Please also check your ingress configuration. ```yaml spec: rules: - http: paths: - pathType: Prefix path: /ws backend: service: name: steadybit-platform port: number: 7878 - pathType: Prefix path: / backend: service: name: steadybit-platform port: number: 80 ``` * You can try to connect to the websocket port via curl: ```bash curl 'https://platform.steadybit.com:443/ws' \ -H 'Upgrade: websocket' \ -H 'Connection: Upgrade' \ -H 'Sec-WebSocket-Key: dummy' \ -H 'Sec-WebSocket-Version: 13' \ -v --http1.1 ``` ``` > GET /ws HTTP/1.1 > Host: platform.steadybit.com > User-Agent: curl/8.1.2 > Accept: */* > Upgrade: websocket > Connection: Upgrade > Sec-WebSocket-Key: dummy > Sec-WebSocket-Version: 13 > < HTTP/1.1 101 Switching Protocols < Date: Thu, 14 Dec 2023 14:37:03 GMT < Connection: upgrade < upgrade: websocket ``` ### Platform is behind Nginx and the agents are not able to connect to the platform Error message in the platform logs: ``` java.lang.IllegalArgumentException: Invalid character found in method name [0x160x030.....[us ]. HTTP method names must be tokens at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:407) ``` Solution: * set the nginx backend protocol is HTTPS instead of HTTP ### Configured the Platform with a oidc provider and the redirect to the platform is been send as http instead of https Example error message in the browser: ``` The redirect URI 'http://steadybit-platform.example.com/oauth2/login/code/default' specified in the request does not match the redirect URIs configured for the application 'xyz'. Make sure the redirect URI sent in the request matches one added to your application. ``` Solution: Set the environment variable: server.tomcat.remoteip.trusted-proxies to a regex that matches the CIDRs of the loadbalancer or reverse proxy. Add the following environment variable to the platform manifest: (Example for Google Cloud Load Balancer CIRDs regex) ```yaml env: - name: server.tomcat.remoteip.trusted-proxies value: "(35\\.191\\.(?:[0-9]|[1-9][0-9]|1(?:[0-9][0-9])|2(?:[0-4][0-9]|5[0-5]))\\.(?:[0-9]|[1-9][0-9]|1(?:[0-9][0-9])|2(?:[0-4][0-9]|5[0-5])))|(130\\.211\\.(?:[0-3])\\.(?:[0-9]|[1-9][0-9]|1(?:[0-9][0-9])|2(?:[0-4][0-9]|5[0-5])))" ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.steadybit.com/troubleshooting/common-fixes/on-prem-platform.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.