Data streaming and distrobution use case guide by splindsay-92 · Pull Request #2979 · ably/docs

splindsay-92 · 2025-11-25T18:12:11Z

Description

Added a pubsub guide that focus on data streaming and distrobution, with a focus on conflation, deltas and server-side batching.

FF-164

TODO: There are a few diagrams/images that might be useful, I will ask design once we are happy with the content.

Checklist

Commits have been rebased.
Linting has been run against the changed file(s).
The PR adheres to the writing style guide and contribution guide.

coderabbitai · 2025-11-25T18:12:18Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

🔍 Trigger a full review

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch chat/data-streaming-and-distrobution-use-case-guide

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

src/pages/docs/guides/pub-sub/data-streaming.mdx

david-hernandez-ably · 2025-12-05T17:58:11Z

src/pages/docs/guides/pub-sub/data-streaming.mdx

+* **Producer simplicity:** Publishers send complete state; Ably handles the optimization
+* **Lossless updates:** Subscribers still receive every update, just in a more efficient format
+
+[//]: # (Think it could be good to have some kind of visiual showing full message vs delta message size comparison so its not just walls of text)


Yes 100% need visuals

Request for visuals

Did we get the visual?

Not yet, I have a bunch of visuals that I'm going to request so I'll make sure they all get added. Some may be after merging though.

david-hernandez-ably · 2025-12-05T18:16:17Z

src/pages/docs/guides/pub-sub/data-streaming.mdx

+* **Cost efficiency:** Fewer outbound messages reduce billable message counts
+* **Granular control:** Publish rates can differ across conflation groups, but still be conflated on the same channel.
+
+[//]: # (Could be good here to have another visual showing messages published at mixed rates with differing conflation keys being conflated into a single batch message)


Request for visuals

src/pages/docs/guides/pub-sub/data-streaming.mdx

AndyTWF

This is a good start - I think we can do a bit more to call out Ably's USPs, what makes us distinct in the way we handle these tough problems (e.g. computing deltas server-side)?

src/data/nav/pubsub.ts

src/pages/docs/guides/pub-sub/data-streaming.mdx

AndyTWF · 2025-12-11T13:15:26Z

src/pages/docs/guides/pub-sub/data-streaming.mdx

+* **Producer simplicity:** Publishers send complete state; Ably handles the optimization
+* **Lossless updates:** Subscribers still receive every update, just in a more efficient format
+
+[//]: # (Think it could be good to have some kind of visiual showing full message vs delta message size comparison so its not just walls of text)


Yes 100% need visuals

src/pages/docs/guides/pub-sub/data-streaming.mdx

AndyTWF · 2025-12-11T13:36:54Z

src/pages/docs/guides/pub-sub/data-streaming.mdx

+How you structure channels significantly impacts performance, costs, and scalability:
+
+**Single channel with many subscribers:**
+This is the most common pattern for data streaming, and is recommended in most use-cases. Ably uses [consistent hashing](/docs/platform/architecture/platform-scalability) to distribute channel load across instances, enabling you to fan out to millions of subscribers on a single channel:


Consistent hashing is more about balancing inbound message load, rather than the fanout.

Clarified this, and mentioned we have a separate connection layer that can scale independently from the cores. 5f18fcd

src/pages/docs/guides/pub-sub/data-streaming.mdx

AndyTWF · 2025-12-11T13:38:50Z

src/pages/docs/guides/pub-sub/data-streaming.mdx

+
+Once conflation is configured as a channel rule, no consumer code changes are needed. Subscribers receive conflated updates transparently:
+
+<Code>


Ditto re more client-side languages

src/pages/docs/guides/pub-sub/data-streaming.mdx

m-hulbert

This is really great! I've mostly left minor comments throughout and then 2 asks on formatting:

There's some odd linting putting a line break after every sentence or piece of punctuation.
There's a lot of bold in use - especially in the lists. Can we pare this back a bit to only highlight critical or really important info please.

src/pages/docs/guides/pub-sub/data-streaming.mdx

m-hulbert · 2025-12-19T09:09:13Z

src/pages/docs/guides/pub-sub/data-streaming.mdx

+- **IoT sensor networks:** Aggregating and distributing data from thousands of sensors across industrial facilities, smart cities, or environmental monitoring systems.
+- **Live event platforms:** Managing reactions, chat, and activity feeds during concerts, conferences, or sporting events with thousands of simultaneous participants.
+- **Fleet and asset tracking:** Real-time position and status updates for vehicles, equipment, or goods in logistics and supply chain applications.
+- **Collaborative applications:** Synchronizing state and updates across users in shared documents, design tools, or gaming environments.


I actually disagree with the previous suggestion of mixing a Spaces use case into a Pub/Sub guide.

It does feel like there is some crossover here, state sync for things like match state is very much a pubsub use-case, but I'll have a think and might just remove it.

Edit: I removed it for now.

m-hulbert · 2025-12-19T09:12:01Z

src/pages/docs/guides/pub-sub/data-streaming.mdx

+
+## How do I reduce bandwidth and latency when data changes frequently but incrementally?
+
+**The challenge:** A sports games platform streams live match state to spectators and players as matches progress. The match state naturally grows throughout the game - starting with basic match info, then accumulating player statistics, scores, game events, and other statistics. By mid-game, the full state can be many kilobytes. Publishing the entire state with each update means transmitting increasingly large payloads repeatedly to every spectator, even though only small portions change between updates (a player's ranking shifts, a score increment or some game metric updates). This wastes massive bandwidth on redundant information, especially as the match state grows larger.


I'd suggest removing the bold emphasis on the 'problem' and 'what's needed' and just introducing the separate paragraps using words, e.g.

"An example of frequent, incremental state change is found in sports games platforms..."

src/pages/docs/guides/pub-sub/data-streaming.mdx

m-hulbert · 2025-12-19T09:33:37Z

src/pages/docs/guides/pub-sub/data-streaming.mdx

+**Realtime vs REST for publishing:**
+Choose the appropriate client type based on your publisher characteristics:
+- **Use Realtime SDK** when:
+  - Publishing messages at a very high volume.
+  - Need the lowest possible latency.
+  - Need bidirectional communication (publish and subscribe).
+  - Ordering of published messages is critical.
+
+- **Use REST API** when:
+  - You have stateless publishers (e.g., serverless functions).
+  - Publishing from environments where maintaining persistent connections is impractical.
+  - Publishing from some authoritative backend server, or on behalf of multiple users.
+  - Batch publishing many messages to different channels in a single API call.


This is quite generic advice on REST vs realtime - is there anything we should recommend specifially for these use cases here?

I could just remove this I suppose? The use-cases are more focused on the stream/outbound clients and I've not really discussed publisher characteristics (which could probably be a guide in and of itself), what do you think?

m-hulbert · 2025-12-19T09:34:23Z

src/pages/docs/guides/pub-sub/data-streaming.mdx

+
+### Platform capabilities
+
+Ably's architecture provides built-in guarantees:


Is this any different or rather more info in respect to the 4 pillars section?

There wasn't, so I've removed it :)

m-hulbert · 2025-12-19T09:35:14Z

src/pages/docs/guides/pub-sub/data-streaming.mdx

+* Choose appropriate optimization strategy for each channel or namespace.
+* Monitor statistics to validate configuration choices, start conservatively.
+* Test client applications to ensure they handle any added latency or batching behavior.
+* Review [platform limits](/docs/platform/pricing/limits) for your account tier.


I wonder if we should exclude this as theoretically this should be hard to hit for the majority of customers.

src/pages/docs/guides/pub-sub/data-streaming.mdx

splindsay-92 · 2025-12-22T14:34:45Z

This is really great! I've mostly left minor comments throughout and then 2 asks on formatting:

There's some odd linting putting a line break after every sentence or piece of punctuation.

There's a lot of bold in use - especially in the lists. Can we pare this back a bit to only highlight critical or really important info please.

I've corrected the linting, seems my local was misconfigured.

I've cut down the number of lists/use of bold - if it's still too much, I can do another round of culling

AndyTWF · 2026-01-22T09:55:16Z

src/pages/docs/guides/pub-sub/data-streaming.mdx

+- The optimization works transparently without code changes for producers or consumers
+- Creates predictable billing that scales linearly with user count rather than message volume
+
+[//]: # (Again, might be good to get a diagram or somethign in here so it feels less like a wall of text..)


Design have a diagram for SSB I'm pretty sure

AndyTWF

Happy with the technical solutions proposed here - will leave it to the folks in DevEd to cover the typography

- Use sports example instead of racing telemetry - Mention deltas are more useful for larger payloads

…ntion independent scaling layer for fanout.

splindsay-92 requested a review from a team November 28, 2025 10:10

splindsay-92 force-pushed the chat/data-streaming-and-distrobution-use-case-guide branch from 74a56b6 to 6881afb Compare November 28, 2025 10:10

splindsay-92 requested a review from mittulmadaan November 28, 2025 10:12

splindsay-92 added the review-app Create a Heroku review app label Nov 28, 2025

ably-ci temporarily deployed to ably-docs-chat-data-str-ccunkm November 28, 2025 10:13 Inactive

m-hulbert requested a review from AndyTWF November 28, 2025 10:19

splindsay-92 changed the title ~~chat/data-streaming-and-distrobution-use-case-guide~~ Data streaming and distrobution use case guide Dec 1, 2025

david-hernandez-ably self-requested a review December 3, 2025 14:41

splindsay-92 marked this pull request as ready for review December 3, 2025 15:08

david-hernandez-ably reviewed Dec 5, 2025

View reviewed changes

splindsay-92 force-pushed the chat/data-streaming-and-distrobution-use-case-guide branch 2 times, most recently from 193888b to cf51f30 Compare December 9, 2025 16:27

AndyTWF added review-app Create a Heroku review app and removed review-app Create a Heroku review app labels Dec 11, 2025

ably-ci temporarily deployed to ably-docs-chat-data-str-bckufm December 11, 2025 12:40 Inactive

AndyTWF reviewed Dec 11, 2025

View reviewed changes

ably-ci temporarily deployed to ably-docs-chat-data-str-bckufm December 11, 2025 17:53 Inactive

ably-ci temporarily deployed to ably-docs-chat-data-str-bckufm December 12, 2025 13:30 Inactive

owenpearson reviewed Dec 16, 2025

View reviewed changes

src/pages/docs/guides/pub-sub/data-streaming.mdx Outdated Show resolved Hide resolved

src/pages/docs/guides/pub-sub/data-streaming.mdx Outdated Show resolved Hide resolved

m-hulbert reviewed Dec 19, 2025

View reviewed changes

splindsay-92 force-pushed the chat/data-streaming-and-distrobution-use-case-guide branch from 682ec4c to 151751f Compare December 22, 2025 13:24

splindsay-92 requested a review from m-hulbert December 22, 2025 15:05

splindsay-92 force-pushed the chat/data-streaming-and-distrobution-use-case-guide branch from 7eed2db to a8e78f1 Compare January 7, 2026 17:40

splindsay-92 force-pushed the chat/data-streaming-and-distrobution-use-case-guide branch from a8e78f1 to f7089a1 Compare January 20, 2026 17:24

splindsay-92 requested a review from AndyTWF January 21, 2026 09:09

splindsay-92 force-pushed the chat/data-streaming-and-distrobution-use-case-guide branch from f7089a1 to b044538 Compare January 21, 2026 09:09

AndyTWF reviewed Jan 22, 2026

View reviewed changes

AndyTWF approved these changes Jan 22, 2026

View reviewed changes

splindsay-92 added 26 commits February 6, 2026 17:11

Docs: Split company intros

965beeb

Docs: Update checklist

045927f

Docs: Rework use-cases to be more generically app-focused

02efff9

Docs: Improve the sections on channel architecture.

29884a7

wip

528738a

Update Deltas section per Mittul's commnets

8c839f2

- Use sports example instead of racing telemetry - Mention deltas are more useful for larger payloads

Update Deltas section to note presence message limitations.

3def55a

Specifically call out how Ably's delta compression stats.

98f4c4e

fixing more PR comments

f4e73eb

Highlight the possible cost savings for SSB clearly.

49eb389

Clarify consistent hashing for horizontal channel scalability, and me…

ceb4fe5

…ntion independent scaling layer for fanout.

Add Ruby and Go code examples for each feature.

25fbbc9

Add missing punctuation.

c69679a

Clarify conflation key pattern in pub-sub data streaming guide.

5f63b4f

wip

9165b43

Reducing use of bullet point lists and bold font

e7b106e

Fixing some bad formatting around line breaks

0636a44

Remove bolded headers on How do I sections

57916bd

Remove duplicate four pillars section

e0e7ba3

Remove reference to platform limits in pub-sub guide

4c04730

Change wording - channel rule -> rule

4514296

Reformat rate limit note in pub-sub guide

44c20e7

Simplify formatting and section structure

6ec2a38

Rework text to be less bullet heavy

bf0caf6

fixing PR comments

6f450d2

fixing PR comments

28ce9a7

splindsay-92 force-pushed the chat/data-streaming-and-distrobution-use-case-guide branch from cc54037 to 28ce9a7 Compare February 6, 2026 17:11

splindsay-92 enabled auto-merge (squash) February 6, 2026 17:15

splindsay-92 merged commit 2b128eb into main Feb 6, 2026
7 checks passed

splindsay-92 deleted the chat/data-streaming-and-distrobution-use-case-guide branch February 6, 2026 17:18


		Once conflation is configured as a channel rule, no consumer code changes are needed. Subscribers receive conflated updates transparently:

		<Code>


		## How do I reduce bandwidth and latency when data changes frequently but incrementally?

		The challenge: A sports games platform streams live match state to spectators and players as matches progress. The match state naturally grows throughout the game - starting with basic match info, then accumulating player statistics, scores, game events, and other statistics. By mid-game, the full state can be many kilobytes. Publishing the entire state with each update means transmitting increasingly large payloads repeatedly to every spectator, even though only small portions change between updates (a player's ranking shifts, a score increment or some game metric updates). This wastes massive bandwidth on redundant information, especially as the match state grows larger.


		### Platform capabilities

		Ably's architecture provides built-in guarantees:

Conversation

splindsay-92 commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

coderabbitai bot commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AndyTWF left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

m-hulbert left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

splindsay-92 Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

splindsay-92 commented Dec 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

splindsay-92 commented Nov 25, 2025 •

edited

Loading

coderabbitai bot commented Nov 25, 2025 •

edited

Loading

splindsay-92 Dec 19, 2025 •

edited

Loading