---
title: "Kafka"
url: https://develop.sentry.dev/self-hosted/troubleshooting/kafka/
---

# Troubleshooting Kafka

## [How Kafka Works](https://develop.sentry.dev/self-hosted/troubleshooting/kafka.md#how-kafka-works)

This section is aimed for those who have Kafka problems, but are not yet familiar with Kafka. At a high level, Kafka is a message broker which stores messages in a log, very similar to an array, format. It receives messages from producers that write to a specific topic, and then sends them to consumers that are subscribed to that topic. The consumers can then process the messages.

On the inside, when a message enters a topic, it will be written to a certain partition. You can think of a partition as physical a box that stores messages for a specific topic. In a distributed Kafka setup, each partition might be stored on a different machine/node, but if you only have a single Kafka instance, then all the partitions are stored on the same machine.

When a producer sends a message to a topic, it will either stick to a certain partition number based on the partition key (example: partition 1, partition 2, etc.) or it will choose a partition in a round-robin manner. A consumer will then subscribe to a topic and will automatically be assigned to one or more partitions by Kafka. The consumer will then start receiving messages from the assigned partitions. **Important to note: the number of consumers cannot exceed the number of partitions**. If you have more consumers than partitions, the extra consumers will receive no messages.

Each message in a topic will have an "offset" (number). You can think of this like an "index" in an array. The offset will be used by the consumer to track where it is in the log, and what's the last message it has consumed. Offsets are scoped to a partition, therefore a partition in a topic can have the same offset numbers. If the consumer is not able to keep up with the producer, it will start to lag behind. Most of the time, we want "lag" to be as low as possible. The easiest solution to lagging is adding more partitions and increasing the number of consumers.

The differences with other types of queues or brokers like RabbitMQ or Redis is that Kafka has a concept called "retention time". Messages that are stored on Kafka and consumed by consumers won't be deleted immediately. Instead, they will be stored for a certain period of time. By default, self-hosted Sentry uses Kafka with a retention time of 24 hours. This means that messages that are older than 24 hours will be deleted. If you want to change the retention time, you can do so by modifying the `KAFKA_LOG_RETENTION_HOURS` environment variable in the `kafka` service.

### [Visualize Kafka](https://develop.sentry.dev/self-hosted/troubleshooting/kafka.md#visualize-kafka)

You can visualize the Kafka consumers and their offsets by bringing an additional container, such as [Kafka UI](https://github.com/provectus/kafka-ui) or [Redpanda Console](https://github.com/redpanda-data/console) into your Docker Compose.

Kafka UI:

```yaml
kafka-ui:
  image: provectuslabs/kafka-ui:latest
  restart: on-failure
  environment:
    KAFKA_CLUSTERS_0_NAME: "local"
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:9092"
    DYNAMIC_CONFIG_ENABLED: "true"
  ports:
    - "8080:8080"
  depends_on:
    - kafka
```

Or, you can use Redpanda Console:

```yaml
redpanda-console:
  image: docker.redpanda.com/redpandadata/console:latest
  restart: on-failure
  entrypoint: /bin/sh
  command: -c "echo \"$$CONSOLE_CONFIG_FILE\" > /tmp/config.yml; /app/console"
  environment:
    CONFIG_FILEPATH: "/tmp/config.yml"
    CONSOLE_CONFIG_FILE: |
      kafka:
        brokers: ["kafka:9092"]
        sasl:
          enabled: false
      schemaRegistry:
        enabled: false
      kafkaConnect:
        enabled: false
  ports:
    - "8080:8080"
  depends_on:
    - kafka
```

It's recommended to put this on `docker-compose.override.yml` rather than modifying your `docker-compose.yml` directly. The UI will then can be accessed via `http://localhost:8080/` (or `http://<your-ip>:8080/` if you're using a reverse proxy).

## [Offset Out Of Range Error](https://develop.sentry.dev/self-hosted/troubleshooting/kafka.md#offset-out-of-range-error)

```log
Exception: KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}
```

This happens where Kafka and the consumers get out of sync. Possible reasons are:

1. Running out of disk space or memory
2. Having a sustained event spike that causes very long processing times, causing Kafka to drop messages as they go past the retention time
3. Date/time out of sync issues due to a restart or suspend/resume cycle

Ideally, you want to have zero lag for all consumer groups. If a consumer group has a lot of lag, you need to investigate whether it's caused by a disconnected consumer (e.g., a Sentry/Snuba container that's disconnected from Kafka) or a consumer that's stuck processing a certain message. If it's a disconnected consumer, you can either restart the container or reset the Kafka offset to 'earliest.' Otherwise, you can reset the Kafka offset to 'latest.'

##### Tip

Choose "earliest" if you want to start re-processing events from the beginning. Choose "latest" if you are okay with losing old events and want to start processing from the newest events.

### [Recovery](https://develop.sentry.dev/self-hosted/troubleshooting/kafka.md#recovery)

##### Warning

These solutions may result in data loss for the duration of your Kafka event retention (defaults to 24 hours) when resetting the offset of the consumers.

#### [Proper solution](https://develop.sentry.dev/self-hosted/troubleshooting/kafka.md#proper-solution)

The *proper* solution is as follows ([reported](https://github.com/getsentry/self-hosted/issues/478#issuecomment-666254392) by [@rmisyurev](https://github.com/rmisyurev)).

This example assumes you found the error from the `snuba-errors-consumer` container. Your consumer group name and topic name may be different.

1. Shutdown the corresponding Sentry/Snuba container that's using the consumer group (You can see the corresponding containers by inspecting the `docker-compose.yml` file):

   ```yaml
   snuba-errors-consumer:
     <<: *snuba_defaults
     command: rust-consumer --storage errors --consumer-group snuba-consumers ...
   ```

   According to the snippet above from `docker-compose.yml`, the consumer group is `snuba-consumers`. You need to find whether other containers are also using the same consumer group. In this case, `snuba-errors-consumer`, `snuba-outcomes-consumer`, `snuba-outcomes-billing-consumer`, `snuba-replays-consumer`, `snuba-profiling-profiles-consumer`, `snuba-profiling-functions-consumer` and `snuba-profiling-profile-chunks-consumer` are using the same consumer group, so you need to stop all of them:

   ```shell
   docker compose stop snuba-errors-consumer snuba-outcomes-consumer snuba-outcomes-billing-consumer snuba-replays-consumer snuba-profiling-profiles-consumer snuba-profiling-functions-consumer snuba-profiling-profile-chunks-consumer
   ```

2. Receive consumers list to make sure `snuba-consumers` is there:

   ```shell
   docker compose exec kafka kafka-consumer-groups --bootstrap-server kafka:9092 --list
   ```

3. Get group info for `snuba-consumers`. Here you will see the topics that the consumer group is subscribed to along with their partitions and current offset:

4. Watch what is going to happen with the offset by using dry-run (optional). This example uses the `events` topic found from the previous step:

   ```shell
   docker compose exec kafka kafka-consumer-groups --bootstrap-server kafka:9092 --group snuba-consumers --topic events --reset-offsets --to-latest --dry-run
   ```

5. Set offset to latest and execute. Make sure to replace `events` with the topic you found from step 3:

   ```shell
   docker compose exec kafka kafka-consumer-groups --bootstrap-server kafka:9092 --group snuba-consumers --topic events --reset-offsets --to-latest --execute
   ```

6. Start the previously stopped Sentry/Snuba containers:

   ```shell
   docker compose start snuba-errors-consumer snuba-outcomes-consumer snuba-outcomes-billing-consumer snuba-replays-consumer snuba-profiling-profiles-consumer snuba-profiling-functions-consumer snuba-profiling-profile-chunks-consumer
   ```

##### Tips

* You can replace `snuba-consumers` with other consumer groups or `events` with other topics when needed.
* You can reset the offset to "earliest" instead of "latest" if you want to start from the beginning.
* If you have Kafka UI or Redpanda Console, you can reset the offsets through the web UI instead of the CLI.

#### [Another option](https://develop.sentry.dev/self-hosted/troubleshooting/kafka.md#another-option)

This option is as follows ([reported](https://github.com/getsentry/self-hosted/issues/1690) by [@gabn88](https://github.com/gabn88)):

1. Set offset to latest and execute:

   ```shell
   docker compose exec kafka kafka-consumer-groups --bootstrap-server kafka:9092 --all-groups --all-topics --reset-offsets --to-latest --execute
   ```

Unlike the proper solution, this involves resetting the offsets of all consumer groups and all topics.

#### [Nuclear option](https://develop.sentry.dev/self-hosted/troubleshooting/kafka.md#nuclear-option)

##### Warning

The *nuclear option* is to remove all Kafka-related volumes and recreate them. **This will cause data loss.** You'll lose any unprocessed events from the last 24 hours. Events that have already been processed and persisted in ClickHouse will remain safe.

1. Stop the instance:

   ```shell
   docker compose down --volumes
   ```

2. Remove the the Kafka volume:

   ```shell
   docker volume rm sentry-kafka
   ```

3. Run the install script again:

   ```shell
   ./install.sh
   ```

4. Start the instance:

   ```shell
   docker compose up --wait
   ```

## [Consumers Lagging Behind](https://develop.sentry.dev/self-hosted/troubleshooting/kafka.md#consumers-lagging-behind)

If you notice a very slow ingestion speed and consumers are lagging behind, it's likely that the consumers are not able to keep up with the producers. This can happen if the consumers are not able to keep up with the rate of messages being produced. To fix this, you can increase the number of partitions and increase the number of consumers.

1. For example, if you see `ingest-consumer` consumer group has a lot of lag, and you can see that it's subscribed to `ingest-events` topic, then you need to first increase the number of partitions for that topic.

   ```bash
   docker compose exec kafka kafka-topics --bootstrap-server kafka:9092 --alter --partitions 3 --topic ingest-events
   ```

2. Validate that the number of partitions for the topic is now 3.

   ```bash
   docker compose exec kafka kafka-topics --bootstrap-server kafka:9092 --describe --topic ingest-events
   ```

3. Then, you need to increase the number of consumers for the consumer group. You can see on the `docker-compose.yml` that the container that consumes `ingest-events` topic using `ingest-consumer` consumer group is `events-consumer` container. But we won't modify the `docker-compose.yml` directly, instead, we will create a new file called `docker-compose.override.yml` and add the following:

   ```yaml
   services:
     events-consumer:
       deploy:
         replicas: 3
   ```

   This will increase the number of consumers for the `ingest-consumer` consumer group to 3.

4. Finally, you need to refresh the `events-consumer` container. You can do so by running the following command:

   ```bash
   docker compose up -d --wait events-consumer
   ```

5. Observe the logs of `events-consumer`, you should not see any consumer errors. Let it run for a while (usually a few minutes until a few hours) and observe the Kafka topic lags.

##### Tip

The definition of "normal lag" varies depending on your system resources. If you are running a small instance, you can expect a normal lag of around hundreds of messages. If you are running a large instance, you can expect a normal lag of around thousands of messages.

## [Reducing disk usage](https://develop.sentry.dev/self-hosted/troubleshooting/kafka.md#reducing-disk-usage)

If you want to reduce the disk space used by Kafka, you'll need to carefully calculate how much data you are ingesting, how much data loss you can tolerate and then follow the recommendations on [this awesome StackOverflow post](https://stackoverflow.com/a/52970982/90297) or [this post on our community forum](https://forum.sentry.io/t/sentry-disk-cleanup-kafka/11337/2?u=byk).

You could, however, add these on the Kafka container's environment variables (by [@csvan](https://github.com/getsentry/self-hosted/issues/3389#issuecomment-2453567691)):

```yaml
services:
  kafka:
    # ...
    environment:
      KAFKA_LOG_RETENTION_HOURS: 24
      KAFKA_LOG_CLEANER_ENABLE: true
      KAFKA_LOG_CLEANUP_POLICY: delete
```