Troubleshooting Sentry
Since version 24.1.0, Sentry migrated to Django 4 which contains stricter CSRF protection. By default, the trusted CSRF origins is set to your system.url-prefix, but in some cases where your Sentry deployment can be accessed from multiple domains, you will need to configure CSRF_TRUSTED_ORIGINS on your sentry.conf.py file.
# Assuming your Sentry instance can be accessed from sentry.example.com, 10.100.10.10 and 127.0.0.1.
CSRF_TRUSTED_ORIGINS = ["https://sentry.example.com", "http://10.100.10.10", "http://127.0.0.1:9000"]
# Assuming your Sentry instance can be accessed from sentry.example.com, 10.100.10.10 and 127.0.0.1.
CSRF_TRUSTED_ORIGINS = ["https://sentry.example.com", "http://10.100.10.10", "http://127.0.0.1:9000"]
See Django's documentation on CSRF for further detail.
If you're seeing a higher incoming traffic load, then it's expected. If this is the case, you might want to increase your machine resources.
However, if you have very low incoming traffic and the CPU/RAM constantly spikes, you might want to check on your top (or htop, or similar) command to see which process are taking the most resources. You can then track down the corresponding Docker container and see if there are any logs that might help you identify the issue.
Usually, most containers that are not a dependency (like Postgres or Kafka) will consume some good amount of CPU & RAM during startup, specifically for catching up with backlogs. If they are experiencing a bootloop, it usually comes down to invalid configuration or a broken dependency.
You may see the sentry-data taking too much disk space. You can clean it manually (or putting the cleanup cronjob in place).
Find the Docker mountpoint for the volume by executing:
docker volume inspect sentry-data
# Or if you prefer to do it directly (assuming you have `jq` on your system):
docker volume inspect sentry-data | jq -r .[0].Mountpoint
docker volume inspect sentry-data
# Or if you prefer to do it directly (assuming you have `jq` on your system):
docker volume inspect sentry-data | jq -r .[0].Mountpoint
Then run the following command to remove the contents of the volume for the last 30 days (change the 30 to whatever you want, it's in days):
# `/var/lib/docker/volumes/sentry-data/_data` refers to the mountpoint of the volume
# from the output of the previous command. Change it if it's different.
find /var/lib/docker/volumes/sentry-data/_data -type f -mtime +30 -delete
# `/var/lib/docker/volumes/sentry-data/_data` refers to the mountpoint of the volume
# from the output of the previous command. Change it if it's different.
find /var/lib/docker/volumes/sentry-data/_data -type f -mtime +30 -delete
If you are seeing an error such as
Background workers haven’t checked in recently. It seems that you have a backlog of 200 tasks. Either your workers aren’t running or you need more capacity.
Background workers haven’t checked in recently. It seems that you have a backlog of 200 tasks. Either your workers aren’t running or you need more capacity.
you may benefit from using additional, dedicated workers. This is achieved by creating new worker services in docker-compose.override.yml and tying them to specific queues using the -Q queue_name argument. An example would be:
worker1:
<< : *sentry_defaults
command: run worker -Q events.process_event
worker1:
<< : *sentry_defaults
command: run worker -Q events.process_event
To see a more complete example, please see a sample solution on our community forum.
If ingest-monitors or ingest-occurrences stops making progress after an upgrade, the consumer may be wedged (meaning the process is still alive, but it is stuck on one operation and no longer makes forward progress) in the memcached client path. A low-effort check is to add a socket timeout to the default cache config on sentry/sentry.conf.py and see whether ingestion recovers immediately after restarting the affected container using docker compose restart <container_name>.
CACHES["default"]["OPTIONS"].update({
"timeout": 5,
"connect_timeout": 3,
})
CACHES["default"]["OPTIONS"].update({
"timeout": 5,
"connect_timeout": 3,
})
If this fixes the wedge, you most likely hit the pymemcache hang described in issue #4301. In that case, also check that your Sentry stack is using django.core.cache.backends.memcached.PyMemcacheCache, and watch for worker logs that stop partway through monitor check-in processing.
If the timeout change does not help, collect a process dump or strace from the wedged consumer and look for blocked calls in pymemcache/client/base.py::_recv() or similar memcached client frames.
If you are running your Sentry instance behind a CDN like Cloudflare, Fastify, or the like, you may see some errors of invalid JavaScript or CSS files being loaded from the web interface. This is caused by some static asset files that are already optimized by the bundlers, but aren't being served with minified extensions (for example, .min.js). Therefore, the CDN that you are using will try to optimize the files a second time, which will result in corrupted files.
Some known paths where you may see this error:
- _static/dist/sentry/entrypoints/sentry.css
- _static/dist/sentry/entrypoints/app.js
To fix this, you can disable the auto optimization performed by your CDN.
If you're seeing the following error in your Relay logs:
ERROR relay_server::services::projects::cache::service: failed to fetch project from source: Fetch { project_key: ProjectKey("00000000000000000000000000000000"), previous_fetch: None, initiated: Instant { tv_sec: 16838910, tv_nsec: 923025932 }, when: Some(Instant { tv_sec: 16839069, tv_nsec: 382950946 }), revision: Revision(None) } tags.project_key="00000000000000000000000000000000" tags.has_revision=false error=upstream error failed to send message to service error.sources=[failed to send message to service]
ERROR relay_server::services::projects::cache::service: failed to fetch project from source: Fetch { project_key: ProjectKey("00000000000000000000000000000000"), previous_fetch: None, initiated: Instant { tv_sec: 16838910, tv_nsec: 923025932 }, when: Some(Instant { tv_sec: 16839069, tv_nsec: 382950946 }), revision: Revision(None) } tags.project_key="00000000000000000000000000000000" tags.has_revision=false error=upstream error failed to send message to service error.sources=[failed to send message to service]
This means Relay is unable to fetch the project configuration from Redis. This can happen if the worker (in the taskworker container) is lagging behind, making it unable to write new project configurations to Redis, which Relay then cannot fetch.
To fix this, you can trigger a manual project configuration invalidation by running the following command. Warning: This will temporarily halt event ingestion until the project configurations are written to Redis again.
docker compose exec web ./bin/invalidate-project-configs
docker compose exec web ./bin/invalidate-project-configs
In case of downtime or misconfiguration of your SSO provider (SAML, OIDC, etc.), you can disable SSO authentication by manually modifying the entry in PostgreSQL. This will allow you to log in with the default email/password method.
First, you need to access the PostgreSQL container, and run psql to access the database:
docker compose exec postgres psql -U postgres
docker compose exec postgres psql -U postgres
Then, view the sentry_authprovider table to find the row ID of the SSO provider you want to disable:
-- Execute `\x on` to make the output more readable
\x on
SELECT * FROM sentry_authprovider;
-- Execute `\x on` to make the output more readable
\x on
SELECT * FROM sentry_authprovider;
If you're using SAML, the result will look something like this (notice the id for the provider you want to disable; in this case, it's 1):
id | 1
provider | saml2
config | {"idp": {"slo_url": "[REDACTED]", "sso_url": "[REDACTED]", "x509cert": "[REDACTED]", "entity_id": "[REDACTED]"}, "auth_attributes": {"Email": ["[REDACTED]"], "UserID": ["[REDACTED]"], "SureName": ["S"], "FullName": ["[REDACTED]"], "UserName": ["[REDACTED]"], "FirstName": ["[REDACTED]"]}, "attribute_mapping": {"last_name": "SureName", "first_name": "FirstName", "identifier": "UserID", "user_email": "Email"}}
date_added | 2025-04-21 22:51:30.750328+00
sync_time |
last_sync |
default_role | 50
default_global_access | t
flags | 0
organization_id | 1
id | 1
provider | saml2
config | {"idp": {"slo_url": "[REDACTED]", "sso_url": "[REDACTED]", "x509cert": "[REDACTED]", "entity_id": "[REDACTED]"}, "auth_attributes": {"Email": ["[REDACTED]"], "UserID": ["[REDACTED]"], "SureName": ["S"], "FullName": ["[REDACTED]"], "UserName": ["[REDACTED]"], "FirstName": ["[REDACTED]"]}, "attribute_mapping": {"last_name": "SureName", "first_name": "FirstName", "identifier": "UserID", "user_email": "Email"}}
date_added | 2025-04-21 22:51:30.750328+00
sync_time |
last_sync |
default_role | 50
default_global_access | t
flags | 0
organization_id | 1
You can delete the row and set up the SSO provider again later:
DELETE FROM sentry_authidentity WHERE auth_provider_id = 1;
DELETE FROM sentry_authprovider WHERE id = 1;
DELETE FROM sentry_authidentity WHERE auth_provider_id = 1;
DELETE FROM sentry_authprovider WHERE id = 1;
No restart of the Sentry services is needed, and you should be able to log in with email/password immediately after running these commands.
During the upgrade to 26.5.0, you may encounter an error similar to this message:
django.db.utils.IntegrityError: check constraint "notifmsg_metric_or_workflow_exclusive" of relation "sentry_notificationmessage" is violated by some row
django.db.utils.IntegrityError: check constraint "notifmsg_metric_or_workflow_exclusive" of relation "sentry_notificationmessage" is violated by some row
You can resolve this by running the following SQL command in your PostgreSQL database (using docker compose exec postgres psql -U postgres -d postgres). This will delete all rows that violate the check constraint, which should allow the migration to complete successfully. Note that this will lead to data loss in notification messages.
DELETE FROM sentry_notificationmessage WHERE NOT ((action_id IS NULL AND group_id IS NULL AND incident_id IS NOT NULL AND trigger_action_id IS NOT NULL) OR (action_id IS NOT NULL AND group_id IS NOT NULL AND incident_id IS NULL AND trigger_action_id IS NULL));
DELETE FROM sentry_notificationmessage WHERE NOT ((action_id IS NULL AND group_id IS NULL AND incident_id IS NOT NULL AND trigger_action_id IS NOT NULL) OR (action_id IS NOT NULL AND group_id IS NOT NULL AND incident_id IS NULL AND trigger_action_id IS NULL));
If that fails, you can try to drop the check constraint:
ALTER TABLE "sentry_notificationmessage" DROP CONSTRAINT "notifmsg_metric_or_workflow_exclusive";
ALTER TABLE "sentry_notificationmessage" DROP CONSTRAINT "notifmsg_metric_or_workflow_exclusive";
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").