Self-hosted Data Flow
Learn about the data flow of self-hosted Sentry
This diagram shows the data flow of self-hosted Sentry. It is similar with Application Architecture but we are focusing more on the self-hosted components.
Copied
graph LR
kafka@{ shape: cyl, label: "Kafka\n(eventstream)" }
redis@{ shape: cyl, label: "Redis" }
postgres@{ shape: cyl, label: "Postgres" }
memcached@{ shape: cyl, label: "Memcached" }
clickhouse@{ shape: cyl, label: "Clickhouse" }
smtp@{ shape: win-pane, label: "SMTP Server" }
symbol-server@{ shape: win-pane, label: "Public/Private Symbol Servers" }
internet@{ shape: trap-t, label: "Internet" }
internet --> nginx
nginx -- Event submitted by SDKs --> relay
nginx -- Web UI & API --> web
subgraph querier [Event Querier]
snuba-api --> clickhouse
end
subgraph processing [Event Processing]
kafka --> snuba-consumer --> clickhouse
snuba-consumer --> kafka
kafka --> snuba-replacer --> clickhouse
kafka --> snuba-subscription-scheduler --> clickhouse
kafka --> snuba-subscription-executor --> clickhouse
redis -- As a celery queue --> sentry-consumer
kafka --> sentry-consumer --> kafka
kafka --> sentry-post-process-forwarder --> kafka
sentry-post-process-forwarder -- Preventing concurrent processing of the same event --> redis
vroom-blob-storage@{ shape: cyl, label: "Blob Storage\n(default is filesystem)" }
kafka -- Profiling event processing --> vroom -- Republish to Kafka to be consumed by Snuba --> kafka
vroom --> snuba-api
vroom -- Store profiles data --> vroom-blob-storage
outgoing-monitors@{ shape: win-pane, label: "Outgoing HTTP Monitors" }
redis -- Fetching uptime configs --> uptime-checker -- Publishing uptime monitoring results --> kafka
uptime-checker --> outgoing-monitors
end
subgraph ui [Web User Interface]
sentry-blob-storage@{ shape: cyl, label: "Blob Storage\n(default is filesystem)" }
web --> worker
web --> postgres
web -- Caching layer --> memcached
web -- Queries on event (errors, spans, etc) data (to snuba-api) --> snuba-api
web -- Avatars, attachments, etc --> sentry-blob-storage
worker -- As a celery queue --> redis
worker --> postgres
worker -- Alert & digest emails --> smtp
web -- Sending test emails --> smtp
end
subgraph ingestion [Event Ingestion]
relay@{ shape: rect, label: 'Relay' }
sentry_ingest_consumer[sentry-ingest-consumers]
relay -- Process envelope into specific types --> kafka --> sentry_ingest_consumer -- Caching event data (to redis) --> redis
relay -- Register relay instance --> web
relay -- Fetching project configs (to redis) --> redis
sentry_ingest_consumer -- Symbolicate stack traces --> symbolicator --> symbol-server
sentry_ingest_consumer -- Save event payload to Nodestore --> postgres
sentry_ingest_consumer -- Republish to events topic --> kafka
end
- Events from the SDK is sent to the
relay
service. - Relay parses the incoming envelope, validates whether the DSN and Project ID are valid. It reads project config data from
redis
. - Relay builds a new payload to be consumed by Sentry ingest consumers, and sends it to
kafka
. - Sentry
ingest-*
consumers ( with*
[wildcard] being the event type [errors, transaction, profiles, etc]) consumes the event, caches it inredis
and starts thepreprocess_event
task. - The
preprocess_event
task symbolicates stack traces withsymbolicator
service, and processes the event according to its event type. - The
preprocess_event
task saves the event payload to nodestore (default nodestore backend ispostgres
). - The
preprocess_event
task publishes the event tokafka
under theevents
topic.
- The
snuba-consumer
service consumes events fromevents
topic and processes them. After the events are written to clickhouse, snuba publishes error & transaction events topost-process-forwarder
. - The Sentry
post-process-forwarder
consumer consumes messages and spawns apost_process_group
task for each processed error & issue occurance.
- The
web
service is what you see, it's the Django web UI and API that serves the Sentry's frontend. - The
worker
service mainly consumes tasks fromredis
that acts as a celery queue. One notable task is to send emails through the SMTP server.
Was this helpful?
Help improve this content
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").