Azure Iot Hub device connected/disconnected event processing order

Question

Azure Iot Hub device connected/disconnected event processing order

LWB 0

Hello,

I'm looking forward to obtaining some guidance on how to properly handle device connected and disconnected events. My main concern is:

Ordering - How to ensure that our backend system always has the most updated version of the "connected"/"disconnected" status.

Our logic was built on a simple assumption: We only accept events with OperationTimestam > last processed event timestamp.

The logic described was solid, until recently when we saw an increased number of devices with the following behaviour:

0 - Device connected normally. 
1 - disconnected event  | OperationTimestamp: Aug 17, 2026 @ 14:09:34.214                  Sequence Number: 000000000000000001DD1E0F0EAC4F200000005600000000000000000000082B 
2 - connected event     | OperationTimestamp: Aug 17, 2026 @ 14:12:38.567                  Sequence Number: 000000000000000001DD1E0F0EAC4F200000005600000000000000000000082C 
3 - disconnected event  | OperationTimestamp: Aug 17, 2026 @ 14:14:09.913                  Sequence Number: 000000000000000001DD1E0F0EAC4F200000005600000000000000000000082D 
4 - connected event     | OperationTimestamp: Aug 17, 2026 @ 14:12:38.568 (Event dropped)  Sequence Number: 000000000000000001DD1E0F0EAC4F200000005600000000000000000000082E

The point here is that events 3 and 4 have consecutive sequence numbers (...082D and ...082E), but event 4 has an OperationTimestamp approximately 1 minute and 31 seconds older than event 3. Because our current processing logic uses OperationTimestamp to reject older events, event 4 is discarded even though its sequence number indicates that it follows event 3.

This raises the following questions:

Should SequenceNumber be considered the authoritative property for determining event ordering, or are there other properties/guarantees we should use?
What exactly does OperationTimestamp represent?
Is OperationTimestamp guaranteed to be increasing for events generated by the same device?
If not, under what circumstances can an event have an OperationTimestamp older than an event that has already been processed?
What is the recommended approach for ensuring that our backend always applies the latest device connection state when OperationTimestamp and SequenceNumber indicate different event orderings?
How does SequenceNumber behave under failover circumstances? Does it continue to be sequential or do we need to take any actions from our side?

We would appreciate any documentation or guidance on the expected ordering guarantees and the recommended event-processing strategy.

0 comments

1 answer

Your answer

Answer 1

Manish Deshpande 7,975 Microsoft External Staff Moderator

Hello @LWB

Your trace contains the answer, and it's a reassuring one: are four strictly increasing sequence numbers with no gaps. IoT Hub emitted these in the correct order, and event 4 genuinely is the newest state — that device's true final state is connected, while your backend has recorded disconnected. The ordering signal was present in every event; the comparison was reading the wrong field. This is a one-field change, not a redesign.

Q: Why is the device showing disconnected in the backend when the latest event indicates connected?

A: The events in your trace have sequentially increasing sequenceNumber values with no gaps, which confirms they were generated in the correct order. The latest event indicates the device is connected, so the backend should reflect that state. This suggests the application is likely using the wrong field for event ordering.

Q: Which field should be used for event ordering?

A: Use sequenceNumber as the authoritative ordering field.

It is the only field documented as strictly increasing.
Compare it as a string, not as a numeric value.
A higher sequenceNumber always represents a newer event.

Example:

bool isNewer = string.CompareOrdinal(incomingSeq, storedSeq) > 0;

Q: Can operationTimestamp be used to determine the latest event?

A: No.

operationTimestamp only indicates when the operation occurred. It is not guaranteed to increase sequentially and should not be used for state comparison or event ordering.

Q: Why can events appear out of timestamp order?

A: IoT Hub connection state events are generated from periodic 60-second snapshots, not from every individual connect or disconnect operation.

As a result:

Newer events can contain older operation timestamps.
Event delivery may be delayed or retried depending on the consumption path.
This behavior is expected and does not indicate an issue with IoT Hub.

Q: What is the recommended implementation?

A: Maintain device state using sequenceNumber.

Store the latest sequenceNumber per device.
Accept an incoming event only if its sequenceNumber is greater than the stored value.
Treat duplicate sequence numbers as no-ops.
Use operationTimestamp for logging and diagnostics only.

Q: What should the final device state be in this scenario?

A: Since the captured events have increasing sequenceNumber values and the last event is connected, the correct final device state should be connected. If the backend still shows disconnected, the event ordering logic should be reviewed to ensure it uses sequenceNumber rather than operationTimestamp.

LWB 0 Reputation points

2026-08-19T08:33:37.2133333+00:00

Hello, and what about fail-over? Does it continue to be sequential in that case or what will happen?
Manish Deshpande 7,975 Reputation points Microsoft External Staff Moderator

2026-08-20T20:04:18.1633333+00:00
Hello @LWB

Good point to dig into, and I want to correct something I said earlier: I said failover requires no action on your side, which was too broad. The strictly-greater comparison itself stays safe across a failover, but there are still things you need to handle. Here’s the accurate picture, with what’s documented separated from what isn’t.

What is documented, and what you must handle

Your consumer connection breaks. "The Event Hubs-compatible name and endpoint of your IoT hub's built-in events endpoint change after failover." If your reader uses the Event Hubs-compatible name/endpoint directly, it must be re-fetched. The way to avoid this entirely is to connect using the IoT hub's connection string rather than the Event Hubs-compatible endpoint — the docs state this "ensures that your back-end applications continue to work without requiring manual intervention after failover." If you're on Azure Functions or Stream Analytics, you must update the endpoint and then Restart, "because any event stream offsets become invalid after failover."

Any stored offset or checkpoint is invalid. Your last-processed sequence number per device is your own state and survives fine. But partition offsets / checkpoints from the event stream do not — plan to reset them, not resume from them.

Expect to lose events, and this is the part that matters most for connection state. The published RPO table lists Device-to-cloud messages: "All unread messages are lost." Anything not yet read from the built-in endpoint when failover occurs is gone. Practically: after any failover, your stored state for some devices will be stale, and no ordering logic can repair that.

If you route connection-state events via Event Grid instead, the position is better: "If you emit events through Azure Event Grid, they can be consumed via the same subscriptions that you configured earlier as long as those Event Grid subscriptions continue to be available. No further handling is required for custom endpoints." Worth considering if failover resilience is a hard requirement for this pipeline.

Timing. Manual failover duration scales with device count — "if you have 100,000 devices, it might take 15 minutes, but if you have five million devices, it might take an hour or longer." For a Microsoft-initiated failover: "Microsoft is unlikely to initiate failover except after a significant delay and on a best-effort basis" and it "can take several hours after the loss of the primary region, or even longer in some scenarios."

On your actual question — does the sequence number stay sequential?

I can’t guarantee something that isn’t published. There’s no documentation confirming whether the connection-state sequenceNumber counter keeps increasing through a regional failover or can restart in the paired region. I checked specifically, and I’d rather be transparent about that.

What that means in practice and this is a small change, not a redesign:

Keep the strictly-greater rule as your steady-state comparison. It is correct, and gaps are explicitly permitted, so a jump of more than 1 is never a problem.

Add one escape hatch for the failover case only. If an event for a device arrives with a sequence number lower than your stored value and the event is recent (compare eventTime/operationTimestamp against wall clock used here as a staleness heuristic, not as an ordering key), don't silently drop it. Log it and treat it as a reset signal: accept the state and rebase the stored sequence number to the new value. Under normal operation this branch should essentially never fire, so it doubles as an alarm.

Re-baseline deliberately after a known failover. Because unread events are lost, the reliable recovery is not to replay the stream — it's to re-establish ground truth. Query device connection state directly, or rely on your device heartbeat if you have one, and reset stored sequence numbers at that point.

References

Failover types, RPO table, post-failover configuration
https://learn.microsoft.com/en-us/azure/reliability/reliability-iot-hub

Failover/failback limits and duration
https://learn.microsoft.com/en-us/azure/iot-hub/tutorial-manual-failover

Using the IoT hub connection string to survive failover
https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-read-builtin#connect-to-the-built-in-endpoint

If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Thanks,
Manish.
Manish Deshpande 7,975 Reputation points Microsoft External Staff Moderator

2026-08-22T20:55:46.1733333+00:00

Hello @LWB

I wanted to check if my last response made sense. I’d be glad to assist further or explain anything in more detail and please accept as Yes and upvote if the answer is helpful so that it can help others in the community.