Data · AI · Law · Ethics

The magazine for decision-makers

·

Databricks Takes Streaming to the Millisecond Level: Real-Time Mode Is Now Generally Available

On October 29, 2026, Databricks made Real-Time Mode generally available for Spark Declarative Pipelines: millisecond-latency streaming within the same framework used for ETL and batch.

by

·

·

4–6 minutes
Illustrazione astratta di flussi di dati veloci che attraversano una pipeline

On October 29, 2026, Databricks announced the general availability (GA) of Real-Time Mode (RTM) for Spark Declarative Pipelines (SDP). In practice, Databricks declarative pipelines can now process data streams with end-to-end latencies in the millisecond range, without having to run a specialized streaming engine alongside Spark.

What Real-Time Mode is

Real-Time Mode is an evolution of the Spark Structured Streaming engine, introduced by Databricks in 2025 for operational workloads. It works on three simple ideas: data is processed continuously, all stages of the query remain active at the same time, and each result is passed to the next stage as soon as it is ready, without waiting for a batch to close.

Micro-batch (traditional model) Incoming events Waiting for thenext batch Stage 1 Stage 2 Output Stages run one after another, in batches: data waits its turn. Real-Time Mode (SDP) Incoming events Stage 1 + Stage 2 active together continuous flow, data passed on immediately Sink Action Continuous streaming: end-to-end latencies in the millisecond range. Illustrative diagram by QrmServizi based on the description in the Databricks blog (October 29, 2026).
Micro-batch and Real-Time Mode compared (illustrative diagram).

Why inside Spark Declarative Pipelines

SDP has become the standard on Databricks for building streaming and ETL pipelines: it handles dependencies, incremental processing, orchestration, failure recovery and data quality checks on its own. With Real-Time Mode, low-latency use cases can also live within the same framework. According to Databricks, this means a single API, a single observability system and a single billing model.

On serverless compute there is no cluster to choose or size, and you pay only for the cores actually used. RTM flows also benefit from the versionless upgrades and low-downtime maintenance typical of SDP.

Before Sparkbatch and ETL Dedicated enginee.g. Flink, low latency Two codebases, fragmentedgovernance, more operational work. with RTM on SDP After Spark Declarative Pipelines Batch ETL Low-latency streaming (RTM) a single API, observability and billing A single framework for all pipelines. Illustrative diagram by QrmServizi. A summary of Databricks’ message, not an independent measurement.
From two separate engines to a single framework (illustrative diagram).

The figures claimed by Databricks

  • Latency: the release notes cite end-to-end latencies as low as five milliseconds, available on both classic and serverless compute.
  • Compared with micro-batch: up to 100 times faster than the traditional batch architecture.
  • Compared with Flink: on real-world feature computation workloads, RTM is up to 92% faster than specialized engines such as Apache Flink. The benchmark is published by Databricks on GitHub.
  • Feature Store: the same architecture enables the Databricks Feature Store to achieve p99 latency of 200 ms from Kafka to the online store.

These are vendor figures: before moving a workload from one engine to another, it is advisable to measure RTM on your own real-world cases.

How to enable it

The procedure described in the official blog involves three steps: switching the pipeline to continuous mode, enabling Real-Time Mode with the dedicated Spark configuration, and defining an update flow with a RealTime trigger toward a sink.

{
  "continuous": true,
  "spark_conf": {
    "spark.databricks.streaming.realTimeMode.enabled": "true"
  }
}

In code, the sink is defined with dp.create_sink() and the flow with the @dp.update_flow decorator, setting pipelines.trigger to "RealTime".

1. Sources 2. SDP pipeline 3. Sinks and usage Apache Kafka Amazon MSK Azure Event Hubs Kinesis (source only) Spark Declarative Pipeline mode: continuous Real-Time Mode enabled update flow with “RealTime” trigger orchestration, error recovery and quality checks included Kafka-compatible sink ForEach: Lakebase, Redis, OLTP Operational use fraud prevention, personalization, recommendations, real-time features Illustrative diagram by QrmServizi based on Databricks documentation (supported sources and sinks may vary: check the documentation).
The data path in a Real-Time Mode pipeline (illustrative diagram).

Use cases

RTM is designed to turn data into decisions within a few milliseconds: fraud detection, live personalization, recommendation systems and feature computation for machine learning models. Thanks to ForEach sinks, the pipeline can write directly to Lakebase, Redis or any other transactional database. Features reach the operational store a few milliseconds after being computed, with no synchronization steps between offline and online environments.

What to keep in mind

  • Sources and sinks: according to the documentation published during the preview phase, RTM works with message buses (Kafka, Amazon MSK, Azure Event Hubs; Kinesis only as a source), and Delta is not supported as a source or destination in real-time mode. The same pages stated that stream-stream joins and session windows were not supported and recommended a single RTM flow per pipeline. These limitations may have changed with GA: check the latest documentation.
  • Multiple flows per pipeline: since October 28, the ability to run multiple RTM flows in the same serverless pipeline (preview channel) has been in Beta, together with adaptive repartitioning of the source stage for Kafka and Kinesis.
  • Announced next steps: grouping multiple RTM flows to reduce costs, and a native Lakebase sink with batching, automatic retries and authentication managed by Unity Catalog.

In summary

With Real-Time Mode now generally available, Databricks aims to bring ETL, batch and low-latency operational streaming together in the same tool. For those currently maintaining Spark plus a dedicated engine for time-critical cases, it is worth evaluating a proof of concept on a real use case, comparing latency, costs and source and sink limitations against their own scenario.

Sources

Leave a Reply

Your email address will not be published. Required fields are marked *