Real-time mode limitations

This page describes known limitations for real-time mode in Structured Streaming.

Source limitations

For Kinesis, Databricks recommends Enhanced Fan-Out (EFO) mode for the lowest latency. Also, frequent repartitions might negatively impact latency.

Union limitations

The Union operator has some limitations:

• Self-union is not supported:
  • For Kafka, you can't use the same source data frame object and union derived data frames from it. As a workaround, use different DataFrames that read from the same source.
  • For Kinesis, you can't union data frames derived from the same Kinesis source with the same configuration. As a workaround, instead of using different DataFrames, you can assign a different consumerName option to each DataFrame.
• Stateful operators (for example, aggregate, deduplicate, transformWithState) can't be defined before the Union.
• Union with batch sources is not supported.

mapPartitions limitation

Real-time mode doesn't support mapPartitions in Scala or similar Python APIs (mapInPandas, mapInArrow). They take an iterator of the entire input partition and produce an iterator of the entire output, which, if not yielded eagerly, may block the output and increase latency. Their semantics also don't support watermark propagation well.

Use scalar UDFs combined with Transform complex data types or filter instead to achieve similar functionality.

transformWithStateInPandas not supported

The transformWithStateInPandas operator is not supported in real-time mode. If you need custom stateful processing in Python with real-time mode, use the row-based transformWithState API instead. The row-based API provides the same stateful processing capabilities using Row objects rather than pandas DataFrames.

See transformWithState in real-time mode for details on transformWithState behavior in real-time mode, and Real-time mode examples for a working Python example using the row-based API.