---
title: "Backend Telemetry Processor"
description: "Detailed backend telemetry processor design."
url: https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor/
---

# Backend Telemetry Processor

🚧 This document is work in progress.

This document uses key words such as "MUST", "SHOULD", and "MAY" as defined in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt) to indicate requirement levels.

For the common specification, refer to the [Telemetry Processor](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor.md) page. This page describes a backend-specific approach, which is optimized for high load. The key difference is that the telemetry scheduler pulls telemetry data from the telemetry buffers using **weighted round-robin scheduling**.

It's worth noting that this approach may not be suitable for SDKs needing to support multiple platforms, such as Java, because this approach doesn't work well with offline caching. Offline caches also need a priority based sending strategy and a priority based overflow strategy to avoid dropping critical data over high volume data. If the telemetry scheduler pulls data from the telemetry buffer and it supports an offline cache, it needs to balance items in the offline cache with items from the telemetry buffer. Each SDK should evaluate its requirements and decide whether to adopt the backend-specific pull-based approach or continue using a push-based model as defined in the [Telemetry Processor](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor.md) page, depending on its platform constraints and architectural needs.

## [Backend-Specific Design Decisions](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#backend-specific-design-decisions)

* **Weighted round-robin scheduling**: Backend applications often run under sustained high load. Weighted scheduling ensures critical telemetry (errors) gets sent even when flooded with high-volume data (logs, spans).
* **Signal-based scheduling**: The scheduler wakes when new data arrives rather than polling, reducing CPU overhead in idle periods.

### [Priorities](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#priorities)

* CRITICAL: Error, Feedback.
* HIGH: Session, CheckIn.
* MEDIUM: Transaction, ClientReport, Span.
* LOW: Log, Profile, ProfileChunk.
* LOWEST: Replay.

Configurable via weights.

### [Components](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#components)

#### [TelemetryBuffer](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#telemetrybuffer)

Unlike the [push-based approach](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor.md#push-based-approach), the [telemetry scheduler](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#telemetryscheduler) pulls data from the telemetry buffer by iterating over all buffers using weighted round-robin scheduling and flushing a buffer when it is ready. The telemetry buffer **MUST** still follow the common [telemetry buffer requirements](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor.md#telemetry-buffer). Here are the additional backend-specific requirements:

1. The telemetry buffer **SHOULD** drop older items as the overflow policy. It **MAY** also drop newer items to preserve what's already buffered.

On the backend, use the same size limits as the [common requirements](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor.md#telemetry-buffer), except for spans, where we recommend **1000** because span volume is higher.

##### [Span Buffer](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#span-buffer)

The span buffer must follow the common [telemetry span buffer requirements](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor.md#span-buffer).

##### [Trace Consistency Trade-offs](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#trace-consistency-trade-offs)

There still remains a small subset of cases that might result in partial traces, where either an old trace bucket was dropped and a new span with the same trace arrived, or we dropped an incoming span of this trace. The preferred overflow behavior in most cases should be `drop_oldest` since it results in the fewest incomplete traces from the two scenarios.

Buffers are mapped to [DataCategories](https://github.com/getsentry/relay/blob/master/relay-base-schema/src/data_category.rs), which determine their scheduling priority and rate limits.

#### [TelemetryScheduler](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#telemetryscheduler)

Unlike the [index spec's push-based approach](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor.md#push-based-or-pull-based-approach), where telemetry buffers push data to the scheduler when full or on timeout, the TelemetryScheduler here uses a **pull-based approach**: it runs as a background worker, actively pulls ready batches from telemetry buffers, and forwards them to the transport.

* **Initialization**: Constructs a weighted priority cycle (e.g., `[CRITICAL×5, HIGH×4, MEDIUM×3, ...]`) based on configured weights.

* **Event loop**: Wakes when explicitly signaled from the `captureX` methods on the client when new data is available (if the language does not support this, then a periodic ticker can be used).

* **Buffer selection**: Iterates through the priority cycle, checks whether a buffer is ready to flush and not rate limited, and then pulls the next batch from that buffer.

* **Rate limit coordination**: Queries the transport's rate limit state before attempting to send any category.

* **Envelope construction**: Converts buffered items into Sentry protocol envelopes.

  * Log items are batched together into a single envelope with multiple log entries.
  * Other categories typically send one item per envelope.

#### [Transport](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#transport)

The transport layer handles HTTP communication with Sentry's ingestion endpoints.

The only layer responsible for dropping events is the Buffer. In case that the transport is full, then the Buffer should drop the batch.

### [Configuration](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#configuration)

#### [Transport Options](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#transport-options)

* **Capacity**: 1000 items.

#### [Telemetry Buffer Options](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#telemetry-buffer-options)

* **Capacity**: 100 items for errors and check-ins, 10\*BATCH\_SIZE for logs, 1000 for transactions.
* **Overflow policy**: `drop_oldest`.
* **Batch size**: 1 for errors and check-ins (immediate send), 100 for logs.
* **Batch timeout**: 5 seconds for logs.

#### [Scheduler Options](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#scheduler-options)

* **Priority weights**: CRITICAL=5, HIGH=4, MEDIUM=3, LOW=2, LOWEST=1.

### [Implementation Example (Go)](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#implementation-example-go)

The `sentry-go` SDK provides a reference implementation of this architecture:

#### [Storage Interface](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#storage-interface)

```go
type Storage[T any] interface {
    // Core operations
    Offer(item T) bool
    Poll() (T, bool)
    PollBatch(maxItems int) []T
    PollIfReady() []T
    Drain() []T
    Peek() (T, bool)

    // State queries
    Size() int
    Capacity() int
    IsEmpty() bool
    IsFull() bool
    Utilization() float64

    // Flush management
    IsReadyToFlush() bool
    MarkFlushed()

    // Category/Priority
    Category() ratelimit.Category
    Priority() ratelimit.Priority
}


// Single item buffer
func (b *RingBuffer[T]) PollIfReady() []T {
	b.mu.Lock()
	defer b.mu.Unlock()

	if b.size == 0 {
		return nil
	}

	ready := b.size >= b.batchSize ||
		(b.timeout > 0 && time.Since(b.lastFlushTime) >= b.timeout)

	if !ready {
		return nil
	}

	itemCount := b.batchSize
	if itemCount > b.size {
		itemCount = b.size
	}

	result := make([]T, itemCount)
	var zero T

	for i := 0; i < itemCount; i++ {
		result[i] = b.items[b.head]
		b.items[b.head] = zero
		b.head = (b.head + 1) % b.capacity
		b.size--
	}

	b.lastFlushTime = time.Now()
	return result
}

// Bucketed buffer
func (b *BucketedBuffer[T]) PollIfReady() []T {
	b.mu.Lock()
	defer b.mu.Unlock()
	if b.bucketCount == 0 {
		return nil
	}
	// the batchSize is satisfied based on total items
	ready := b.totalItems >= b.batchSize || (b.timeout > 0 && time.Since(b.lastFlushTime) >= b.timeout)
	if !ready {
		return nil
	}
	// keep track of oldest bucket
	oldest := b.buckets[b.head]
	if oldest == nil {
		return nil
	}
	items := oldest.items
	if oldest.traceID != "" {
		delete(b.traceIndex, oldest.traceID)
	}
	b.buckets[b.head] = nil
	b.head = (b.head + 1) % b.bucketCapacity
	b.totalItems -= len(items)
	b.bucketCount--
	b.lastFlushTime = time.Now()
	return items
}
```

#### [TelemetryScheduler Processing](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#telemetryscheduler-processing)

```go
func (s *TelemetryScheduler) run() {
	for {
		s.mu.Lock()

		for !s.hasWork() && s.ctx.Err() == nil {
		  // signal the scheduler to sleep till we receive a signal for an added item.
			s.cond.Wait()
		}

		s.mu.Unlock()
		s.processNextBatch()
	}
}

func (s *TelemetryScheduler) hasWork() bool {
	for _, buffer := range s.buffers {
		if buffer.IsReadyToFlush() {
			return true
		}
	}
	return false
}

func (s *TelemetryScheduler) processNextBatch() {
	if len(s.currentCycle) == 0 {
		return
	}

	priority := s.currentCycle[s.cyclePos]
	s.cyclePos = (s.cyclePos + 1) % len(s.currentCycle)

	var bufferToProcess TelemetryBuffer[protocol.EnvelopeItemConvertible]
	var categoryToProcess ratelimit.Category
	for category, buffer := range s.buffers {
		if buffer.Priority() == priority && buffer.IsReadyToFlush() {
			bufferToProcess = buffer
			categoryToProcess = category
			break
		}
	}

	if bufferToProcess != nil {
		s.processItems(bufferToProcess, categoryToProcess, false)
	}
}
```

#### [Flushing](https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/backend-telemetry-processor.md#flushing)

```go
func (s *TelemetryScheduler) flush() {
  // should process all buffers and send to transport
  for category, buffer := range s.buffers {
		if !buffer.IsEmpty() {
			s.processItems(buffer, category, true)
		}
	}
}

// The Buffer exposes the flush method that calls both
func (b *Buffer) Flush(timeout time.Duration) {
  scheduler.flush()
  transport.flush(timeout)
}
```