On October 29, 2026, Databricks announced the general availability (GA) of Real-Time Mode (RTM) for Spark Declarative Pipelines (SDP). In practice, Databricks declarative pipelines can now process data streams with end-to-end latencies in the millisecond range, without having to run a specialized streaming engine alongside Spark.
What Real-Time Mode is
Real-Time Mode is an evolution of the Spark Structured Streaming engine, introduced by Databricks in 2025 for operational workloads. It works on three simple ideas: data is processed continuously, all stages of the query remain active at the same time, and each result is passed to the next stage as soon as it is ready, without waiting for a batch to close.
Why inside Spark Declarative Pipelines
SDP has become the standard on Databricks for building streaming and ETL pipelines: it handles dependencies, incremental processing, orchestration, failure recovery and data quality checks on its own. With Real-Time Mode, low-latency use cases can also live within the same framework. According to Databricks, this means a single API, a single observability system and a single billing model.
On serverless compute there is no cluster to choose or size, and you pay only for the cores actually used. RTM flows also benefit from the versionless upgrades and low-downtime maintenance typical of SDP.
The figures claimed by Databricks
- Latency: the release notes cite end-to-end latencies as low as five milliseconds, available on both classic and serverless compute.
- Compared with micro-batch: up to 100 times faster than the traditional batch architecture.
- Compared with Flink: on real-world feature computation workloads, RTM is up to 92% faster than specialized engines such as Apache Flink. The benchmark is published by Databricks on GitHub.
- Feature Store: the same architecture enables the Databricks Feature Store to achieve p99 latency of 200 ms from Kafka to the online store.
These are vendor figures: before moving a workload from one engine to another, it is advisable to measure RTM on your own real-world cases.
How to enable it
The procedure described in the official blog involves three steps: switching the pipeline to continuous mode, enabling Real-Time Mode with the dedicated Spark configuration, and defining an update flow with a RealTime trigger toward a sink.
{
"continuous": true,
"spark_conf": {
"spark.databricks.streaming.realTimeMode.enabled": "true"
}
}
In code, the sink is defined with dp.create_sink() and the flow with the @dp.update_flow decorator, setting pipelines.trigger to "RealTime".
Use cases
RTM is designed to turn data into decisions within a few milliseconds: fraud detection, live personalization, recommendation systems and feature computation for machine learning models. Thanks to ForEach sinks, the pipeline can write directly to Lakebase, Redis or any other transactional database. Features reach the operational store a few milliseconds after being computed, with no synchronization steps between offline and online environments.
What to keep in mind
- Sources and sinks: according to the documentation published during the preview phase, RTM works with message buses (Kafka, Amazon MSK, Azure Event Hubs; Kinesis only as a source), and Delta is not supported as a source or destination in real-time mode. The same pages stated that stream-stream joins and session windows were not supported and recommended a single RTM flow per pipeline. These limitations may have changed with GA: check the latest documentation.
- Multiple flows per pipeline: since October 28, the ability to run multiple RTM flows in the same serverless pipeline (preview channel) has been in Beta, together with adaptive repartitioning of the source stage for Kafka and Kinesis.
- Announced next steps: grouping multiple RTM flows to reduce costs, and a native Lakebase sink with batching, automatic retries and authentication managed by Unity Catalog.
In summary
With Real-Time Mode now generally available, Databricks aims to bring ETL, batch and low-latency operational streaming together in the same tool. For those currently maintaining Spark plus a dedicated engine for time-critical cases, it is worth evaluating a proof of concept on a real use case, comparing latency, costs and source and sink limitations against their own scenario.




Leave a Reply