--- title: "Sentry" url: https://develop.sentry.dev/self-hosted/troubleshooting/sentry/ --- # Troubleshooting Sentry ## [CSRF verification failed](https://develop.sentry.dev/self-hosted/troubleshooting/sentry.md#csrf-verification-failed) Since version 24.1.0, Sentry migrated to Django 4 which contains stricter CSRF protection. By default, the trusted CSRF origins is set to your `system.url-prefix`, but in some cases where your Sentry deployment can be accessed from multiple domains, you will need to configure `CSRF_TRUSTED_ORIGINS` on your `sentry.conf.py` file. **python** ```python # Assuming your Sentry instance can be accessed from sentry.example.com, 10.100.10.10 and 127.0.0.1. CSRF_TRUSTED_ORIGINS = ["https://sentry.example.com", "http://10.100.10.10", "http://127.0.0.1:9000"] ``` See [Django's documentation on CSRF](https://docs.djangoproject.com/en/4.2/ref/settings/#std-setting-CSRF_TRUSTED_ORIGINS) for further detail. ## [Containers taking too much CPU/RAM usage](https://develop.sentry.dev/self-hosted/troubleshooting/sentry.md#containers-taking-too-much-cpuram-usage) If you're seeing a higher incoming traffic load, then it's expected. If this is the case, you might want to increase your machine resources. However, if you have very low incoming traffic and the CPU/RAM constantly spikes, you might want to check on your `top` (or `htop`, or similar) command to see which process are taking the most resources. You can then track down the corresponding Docker container and see if there are any logs that might help you identify the issue. Usually, most containers that are not a dependency (like Postgres or Kafka) will consume some good amount of CPU & RAM during startup, specifically for catching up with backlogs. If they are experiencing a [bootloop](https://en.wikipedia.org/wiki/Booting#Bootloop), it usually comes down to invalid configuration or a broken dependency. ## [`sentry-data` volume not being cleaned up](https://develop.sentry.dev/self-hosted/troubleshooting/sentry.md#sentry-data-volume-not-being-cleaned-up) You may see the `sentry-data` taking too much disk space. You can clean it manually (or putting the cleanup cronjob in place). Find the Docker mountpoint for the volume by executing: **bash** ```bash docker volume inspect sentry-data # Or if you prefer to do it directly (assuming you have `jq` on your system): docker volume inspect sentry-data | jq -r .[0].Mountpoint ``` Then run the following command to remove the contents of the volume for the last 30 days (change the `30` to whatever you want, it's in days): **bash** ```bash # `/var/lib/docker/volumes/sentry-data/_data` refers to the mountpoint of the volume # from the output of the previous command. Change it if it's different. find /var/lib/docker/volumes/sentry-data/_data -type f -mtime +30 -delete ``` ## [Background workers haven't checked in recently](https://develop.sentry.dev/self-hosted/troubleshooting/sentry.md#background-workers-havent-checked-in-recently) If you are seeing an error such as **bash** ```bash Background workers haven’t checked in recently. It seems that you have a backlog of 200 tasks. Either your workers aren’t running or you need more capacity. ``` you may benefit from using additional, dedicated workers. This is achieved by creating new `worker` services in `docker-compose.override.yml` and tying them to specific queues using the `-Q queue_name` argument. An example would be: **yaml** ```yaml worker1: << : *sentry_defaults command: run worker -Q events.process_event ``` To see a more complete example, please see [a sample solution on our community forum](https://forum.sentry.io/t/how-to-clear-backlog-and-monitor-it/10715/14?u=byk). ## [Monitor check-ins stop ingesting](https://develop.sentry.dev/self-hosted/troubleshooting/sentry.md#monitor-check-ins-stop-ingesting) If `ingest-monitors` or `ingest-occurrences` stops making progress after an upgrade, the consumer may be wedged (meaning the process is still alive, but it is stuck on one operation and no longer makes forward progress) in the memcached client path. A low-effort check is to add a socket timeout to the default cache config on `sentry/sentry.conf.py` and see whether ingestion recovers immediately after restarting the affected container using `docker compose restart `. **python** ```python CACHES["default"]["OPTIONS"].update({ "timeout": 5, "connect_timeout": 3, }) ``` If this fixes the wedge, you most likely hit the `pymemcache` hang described in [issue #4301](https://github.com/getsentry/self-hosted/issues/4301). In that case, also check that your Sentry stack is using `django.core.cache.backends.memcached.PyMemcacheCache`, and watch for worker logs that stop partway through monitor check-in processing. If the timeout change does not help, collect a process dump or `strace` from the wedged consumer and look for blocked calls in `pymemcache/client/base.py::_recv()` or similar memcached client frames. ## [Cannot Load JavaScript or CSS Files From Web Interface](https://develop.sentry.dev/self-hosted/troubleshooting/sentry.md#cannot-load-javascript-or-css-files-from-web-interface) If you are running your Sentry instance behind a CDN like Cloudflare, Fastify, or the like, you may see some errors of invalid JavaScript or CSS files being loaded from the web interface. This is caused by some static asset files that are already optimized by the bundlers, but aren't being served with minified extensions (for example, `.min.js`). Therefore, the CDN that you are using will try to optimize the files a second time, which will result in corrupted files. Some known paths where you may see this error: * \_static/dist/sentry/entrypoints/sentry.css * \_static/dist/sentry/entrypoints/app.js To fix this, you can disable the auto optimization performed by your CDN. ## [Relay rejects envelope events](https://develop.sentry.dev/self-hosted/troubleshooting/sentry.md#relay-rejects-envelope-events) If you're seeing the following error in your Relay logs: **bash** ```bash ERROR relay_server::services::projects::cache::service: failed to fetch project from source: Fetch { project_key: ProjectKey("00000000000000000000000000000000"), previous_fetch: None, initiated: Instant { tv_sec: 16838910, tv_nsec: 923025932 }, when: Some(Instant { tv_sec: 16839069, tv_nsec: 382950946 }), revision: Revision(None) } tags.project_key="00000000000000000000000000000000" tags.has_revision=false error=upstream error failed to send message to service error.sources=[failed to send message to service] ``` This means Relay is unable to fetch the project configuration from Redis. This can happen if the worker (in the `taskworker` container) is lagging behind, making it unable to write new project configurations to Redis, which Relay then cannot fetch. To fix this, you can trigger a manual project configuration invalidation by running the following command. Warning: This will temporarily halt event ingestion until the project configurations are written to Redis again. **bash** ```bash docker compose exec web ./bin/invalidate-project-configs ``` ## [Disabling SSO providers](https://develop.sentry.dev/self-hosted/troubleshooting/sentry.md#disabling-sso-providers) In case of downtime or misconfiguration of your SSO provider (SAML, OIDC, etc.), you can disable SSO authentication by manually modifying the entry in PostgreSQL. This will allow you to log in with the default email/password method. First, you need to access the PostgreSQL container, and run `psql` to access the database: **bash** ```bash docker compose exec postgres psql -U postgres ``` Then, view the `sentry_authprovider` table to find the row ID of the SSO provider you want to disable: **sql** ```sql -- Execute `\x on` to make the output more readable \x on SELECT * FROM sentry_authprovider; ``` If you're using SAML, the result will look something like this (notice the `id` for the provider you want to disable; in this case, it's `1`): **bash** ```bash id | 1 provider | saml2 config | {"idp": {"slo_url": "[REDACTED]", "sso_url": "[REDACTED]", "x509cert": "[REDACTED]", "entity_id": "[REDACTED]"}, "auth_attributes": {"Email": ["[REDACTED]"], "UserID": ["[REDACTED]"], "SureName": ["S"], "FullName": ["[REDACTED]"], "UserName": ["[REDACTED]"], "FirstName": ["[REDACTED]"]}, "attribute_mapping": {"last_name": "SureName", "first_name": "FirstName", "identifier": "UserID", "user_email": "Email"}} date_added | 2025-04-21 22:51:30.750328+00 sync_time | last_sync | default_role | 50 default_global_access | t flags | 0 organization_id | 1 ``` You can delete the row and set up the SSO provider again later: **sql** ```sql DELETE FROM sentry_authidentity WHERE auth_provider_id = 1; DELETE FROM sentry_authprovider WHERE id = 1; ``` No restart of the Sentry services is needed, and you should be able to log in with email/password immediately after running these commands.