Skip to content

feat: offline support - #1527

Merged
vishalnarkhede merged 69 commits into
v5-releasefrom
vishal/offline-support-with-hydration-1
Aug 2, 2022
Merged

vishalnarkhede merged 69 commits into
v5-releasefrom
vishal/offline-support-with-hydration-1

Conversation

@vishalnarkhede

@vishalnarkhede vishalnarkhede commented Jul 3, 2022 •

Copy link
Copy Markdown
Contributor

Whats Changed:

First version of offline support has been added to the SDK:

  • Store the channels data in sqlite database
  • Being able to view ChannelList and MessageList without any network
  • Usage of sync api and WS events to keep DB in sync with backend.

Screen Recordings

Following screen recordings display the loading time of chat on Slow 3G network

With Offline Support Without Offline Support
Screen.Recording.2022-07-15.at.20.32.37.mov
Screen.Recording.2022-07-15.at.20.34.15.mov

Description

JS client changes

To support offline storage, js client needs some tweaking as well. The changes are done in this PR - GetStream/stream-chat-js#1011

How to enable offline support

Chat component accepts a prop named enableOfflineSupport which can be used to toggle this feature.

When do we store data?

NOTE: Upsert queries to DB are batched as much as possible to lower the overhead of interaction between JS and sqlite.

How do we store data?

  • We are using react-native-quick-sqlite package which uses fast bindings via JSI.
  • We have QuickSqliteClient static class, which is basically an interface to deal with quick sqlite package.
  • You can find the entire DB schema in schema.ts. We store the data in 6 different normalised table - channels, messages, members, users, reactions, reads. Tables are basically mapped version of data/state received in API response from server. Mapper functions help with converting the API response to storable format e.g., mapMessageToStorable
  • We have two additional tables:
    • channelQueries table stores channel ids and corresponding stringified version of filters + sort
    • userSyncStatus table stores the datetime stamp of last time database was synced. This value is set after sync api is called. Please check this slack thread for discussion

How do we use offline stored data?

  • Our UI components such as MessageSimple, ChannelPreview etc expects an instance of Channel class from js client. Thus while loading UI components with offline data, we need to first create instances of Channel class from stored data. We do with following steps
    • Fetch channels data from database
    • Use Mapper functions to convert channels data to the format which is same as channels received from server as part of API response.
    • Use client.hydrateActiveChannels(channels) to hydrate the client with API response generated from local DB. This function will also return instances of Channel class, which can be used in our UI components.

How do we handle migration?

Well in short we don't, instead we simply drop and recreate all the tables if there are any changes to schema. We simply need to manually bump dbVersion on QuickSqliteClient everytime we make any changes to schema. This version number will be compared against user_version pragma to decide weather to recreate tables or not. You can find the logic here

Changes that customers will need on App level

  • Install react-native-quick-sqlite package
  • Currently we expect await client.connectUser(user, token) to succeed before rendering any Chat components. But now after this release, they will need to only call client.connectUser(user, token) but NOT wait for it to succeed, before rendering Chat component. But they must call connectUser before rendering components.
  • Call QuickSqliteClient.resetDatabase() after/before client.disconnectUser() call.
  • Customer can only have one Chat component in codebase.
  • Add enableOfflineSupport prop on Chat component

Tasks

  • Store channels, messages in sqlite database
  • Event handling to update the database
  • Typescript for database schema to ensure sanity
  • Unit tests
  • Usage of sync api to handle connection recovery
  • Setup migration strategy
  • Handling app settings
  • Upgrade watercooler app to use offline support for testing
  • Test expo messaging app. We won't be supporting offline support for Expo right now, but things should work normally as it is.

@vishalnarkhede
vishalnarkhede force-pushed the vishal/offline-support-with-hydration-1 branch 2 times, most recently from 999750e to 0a89925 Compare July 6, 2022 14:01
@vishalnarkhede
vishalnarkhede force-pushed the vishal/offline-support-with-hydration-1 branch from 0a89925 to 8742a26 Compare July 6, 2022 14:30
@vishalnarkhede
vishalnarkhede changed the base branch from develop to v5-release July 6, 2022 14:47
@vishalnarkhede
vishalnarkhede force-pushed the vishal/offline-support-with-hydration-1 branch from fc70d25 to 361a65e Compare July 7, 2022 17:42
santhoshvai and others added 6 commits July 25, 2022 15:07
…-react-native into vishal/offline-support-with-hydration-1

# Conflicts:
#	package/jest-setup.js
#	package/src/components/Attachment/Gallery.tsx
#	package/src/components/Avatar/Avatar.tsx
#	package/src/components/Avatar/GroupAvatar.tsx

@santhoshvai santhoshvai left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor comments.. awesome work @vishalnarkhede

Comment thread package/src/store/sqlite-utils/printRow.ts Outdated
Comment thread package/src/store/sqlite-utils/createUpdateQuery.ts
Comment thread package/src/store/sqlite-utils/appendWhereCluase.ts
Comment thread package/src/hooks/useLoadingImage.tsx
Comment thread package/src/store/apis/getLastSyncedAt.ts Outdated
Comment thread package/src/store/sqlite-utils/createDeleteQuery.ts
Comment thread package/src/store/sqlite-utils/createSelectQuery.ts
Comment thread package/src/store/sqlite-utils/createUpsertQuery.ts Outdated
Comment thread package/src/store/sqlite-utils/createUpsertQuery.ts Outdated
Comment thread .vscode/settings.json Outdated
{
"editor.formatOnSave": true
}
"editor.formatOnSave": false

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree.. temporary I guess?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes yes, only temporary!!

@khushal87

Copy link
Copy Markdown
Contributor

I believe we need to change the PR title and commit messages to something appropriate before merging?

@vishalnarkhede
vishalnarkhede force-pushed the vishal/offline-support-with-hydration-1 branch from e8dddab to 9aa56e2 Compare July 28, 2022 08:59
@vishalnarkhede
vishalnarkhede force-pushed the vishal/offline-support-with-hydration-1 branch from 35aaee1 to ea828c3 Compare July 28, 2022 11:18
@vishalnarkhede

Copy link
Copy Markdown
Contributor Author

Lint issues are fixed now.

@vanGalilea

Copy link
Copy Markdown
Contributor

Hey @vishalnarkhede, I think it will make sense to keep one of our sample apps (SA/TSM) without offline support to monitor regression.

@vanGalilea

Copy link
Copy Markdown
Contributor

Let's verify that we don't have any regression. This means that if the user will NOT opt-in to offline support and WILL NOT install react-native-quick-sqlite everything will work precisely the way it used to

@@ -0,0 +1,44 @@
import Database from 'better-sqlite3';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the idea behind creating a JS file over TS?

};

it('should create tables on first load if offline feature is enabled', async () => {
const { getByTestId } = render(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you also expect that the DB tables are not existing before the rendering?

</Chat>,
);

const expectCIDsOnUIToBeInDB = (queryAllByA11yRole) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move all expect funcs to a utils file?

/* eslint-disable no-underscore-dangle */
import React from 'react';
import { Text, View } from 'react-native';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know you might be annoying fighting TS in a test file, yet we agreed that when we create new test files, they'll be TS ones over JS

expect(matchingReadRows.length).toBe(1);
expect(matchingReadRows[0].unreadMessages).toBe(0);
});
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test file is way too long, can you consider splitting it:

  • by creating files that have to do with certain domains/topics etc.
  • moving setup/expectation funcs to a different utils file
  • using it.each where applicable (this will also reduce code duplications)

Comment on lines +102 to +116
uri:
!imageProp ||
imageProp.includes(randomImageBaseUrl) ||
imageProp.includes(randomSvgBaseUrl)
? imageProp?.includes(streamCDN)
? imageProp
: `${randomImageBaseUrl}${
name ? `?name=${getInitials(name)}&size=${size}` : ''
}`
: getResizedImageUrl({
height: size,
url: imageProp,
width: size,
}),
}}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we please refactor this in the future?

@vishalnarkhede
vishalnarkhede merged commit 9f5ea26 into v5-release Aug 2, 2022
@vishalnarkhede
vishalnarkhede deleted the vishal/offline-support-with-hydration-1 branch August 2, 2022 14:20
vishalnarkhede added a commit that referenced this pull request Aug 23, 2022
* feat: wip jump to a message after using the new around messages api

* refactor: scroll to first unread channel

* refactor: loading more recent messages should not rely on isUpToDate flag anymore

* fix: set noMoreMessages flag to false when we are jumping to a message

* fix: scroll to bottom button should not rely on isUpToDate flag

* fix: show scroll to bottom button using the new hasNoMoreRecentMessages prop

* fix: replace more isUpToDate calls with  hasNoMoreRecentMessagesToLoad

* feat: add documentation for hasNoMoreRecentMessagesToLoad

* feat: update stream-chat-js to 6.3.0

* chore: remove unused import of Text

* fix: readd the removed console log of error

* feat: add documentation for loadChannelAroundMessage

* chore: remove unneeded comment

* chore: move scrollToDebounceTimeoutRef to the top of messageList

* fix: remove unused variables

* fix: update jest snapshot for the new prop addition for messageList

* fix: remove unused useEffect in messageList

* fix: fix the failing message list test

* fix: channel unread indicator was showing up even when marked read

* fix: scroll to bottom indicator must not be always shown on open

* Revert "fix: fix the failing message list test"

This reverts commit d605744.

* fix: sometimes the channel was not loaded to the latest message

* feat: add maxToRenderPerBatch prop to the messageList

this reduces the scrollToIndex failure rate

* fix: when scrolling to a non rendered item, highlight was not shown

* fix: update snapshot to fix unit test

* fix: scroll to messageId didnt work if scrollToFirstUnreadMsg is set

* fix: ensure setTargetedMessage happens after messages array is copied

* fix: simplified channel logic of messageId scroll

* fix: remove unnecessary console log

* docs: create a new version of docs for v5 (#1596)

* docs: upgrade helper guide (#1605)

* docs: cut a new version of docs for v5

* docs: upgrade helper guide

* docs: drop v5 changes from v4 docs

* docs: add reason for cameraroll fork replacement

* docs: make removing cameraroll to be non-optional

Co-authored-by: Santhosh Vaiyapuri <santhoshvai@gmail.com>

* feat: offline support (#1527)

* feat: offline support

* refactor: drop un-used function pragmaUserVersion

* refactor: reverting changes to ChannelPreview

* fix: fixing migration code issue

* refactor: drop limits from queries

* refactor: handling of message.read events and reads entity

* refactor: added members to separate table

* refactor: relative explicit query with left join to enrich users

* refactor: cleanup and reorganizing

* refactor: added index and foreign key constraints

* refactor: changes to be compatible with js client

* test: mocking sqlite

* refactor: added missing fields on channels table

* refactor: changes to be in sync with event from js client

* refactor: renaming table and cleanup of columns

* refactor: moving all quick-sqlite logic to separate class

* test: added simple test for offline support

* refactor: handle member related events

* refactor: added usage of sync api

* refactor: cleaning up un-necessary packages

* refactor: fixed issue with update message foreign key constraint failure

* refactor: cleanup

* refactor: cleanup

* tests: writing tests for offline feature

* refactor: removed dev-menu package and usage

* refactor: remove unused initDevMenu import

* refactor: remove un-used flipper packages

* docs: adding code comments

* refactor: removing unnecessary generics

* refactor: cleanup of mapper functions

* refactor: renaming offlineChannelsActive to staticChannelsActive

* refactor: removed metro config changes

* buiild: fixed issue with manual release script

* refactor: cleanup of handleEventToSyncDB

* refactor: fixing lint issue

* fix: removed redundant usage of useStreami18n hook

* fix: issue with message not displayed in static state

* build: updating stream-chat version

* fix: issue with appendWhereClause and undefined value

* fix: issue with members state initialization

* build: do not format on save

* fix: issue mapping memberCount to api response

* build: upgrading stream-chat version

* fix: reposition image error indicator

* refactor: use network error handling to useLoadingImage

* fix: image loading issue after network is recovered

* fix: logic for sync api and lint issues

* feat: point cameraroll dependency to the stream fork version

* fix: message.new event handling

* refactor: remove unnecessary client as hook dep

* refactor: code review changes

* refactor: moving offline support logic to Chat component

* refactor: fix lint and typescript issues

* refactor: handle quick-sqlite not being installed in better way

* docs: lint fixes

Co-authored-by: Mads Røskar <madshvero@gmail.com>
Co-authored-by: stevegalili <galiliziv@gmail.com>
Co-authored-by: Santhosh Vaiyapuri <santhoshvai@gmail.com>

* build: fix the doc paths

* docs: update upgrade helper doc

* docs: update upgrade helper doc with logout logic changes

* docs: change rc to beta in version label

* docs: add pod install command to upgrade helper

* build: update target branch name

* fix: flipper database plugin setup

* fix: handling of createSelectQuery function

* build: upgrade better sqlite and quick sqlite dev dep

* fix: open the selected image when pressed in the image grid (#1585)

* refactor!: rename gallery state objects to match their data

* fix: open the selected image when pressed in the image grid

* style: reorder object keys to be alphabetic

* chore: rename image grid props

* fix: rename setters to match new names

* test: update snapshots

* test: move bettersqlite test utils to src/test-utils

* test: gitignore locally generated database files

* style: reorder imports

* style: remove redundant whitespace

* refactor: export store apis

* fix: set an initial image index for the ImageGallery (#1639)

* docs: update upgrade helper guide

* docs: rename setImages and setImage

* docs: fix broken mdx imports

* docs: remove unintended newline in import

* docs: update upgrade helper guide

* feat: upgrade stream-chat to 7.0.0-offline-support.4

* docs: update upgrade helper guide

* docs: offline support docs

* ignore intelliJ files.

* Update broken external URLs

* Update broken internal links

* docs: replace cameraroll to our fork in install steps

* docs: upgrading v4 dopcs

* build: connection docusaurus gh action to develop and main

* feat: remove cameraroll fork and use v5 original cameraroll

* feat: upgrade stream-chat to v7 stable

Co-authored-by: Santhosh Vaiyapuri <santhoshvai@gmail.com>
Co-authored-by: Vishal Narkhede <vishalnarkhede.iitd@gmail.com>
Co-authored-by: Mads Røskar <madshvero@gmail.com>
Co-authored-by: Mads Røskar <mads.roskar@getstream.io>
Co-authored-by: nash0x7e2 <mail@neevash.dev>
@github-actions github-actions Bot mentioned this pull request Aug 23, 2022
6 tasks
vishalnarkhede pushed a commit that referenced this pull request Aug 23, 2022
* feat: wip jump to a message after using the new around messages api

* refactor: scroll to first unread channel

* refactor: loading more recent messages should not rely on isUpToDate flag anymore

* fix: set noMoreMessages flag to false when we are jumping to a message

* fix: scroll to bottom button should not rely on isUpToDate flag

* fix: show scroll to bottom button using the new hasNoMoreRecentMessages prop

* fix: replace more isUpToDate calls with  hasNoMoreRecentMessagesToLoad

* feat: add documentation for hasNoMoreRecentMessagesToLoad

* feat: update stream-chat-js to 6.3.0

* chore: remove unused import of Text

* fix: readd the removed console log of error

* feat: add documentation for loadChannelAroundMessage

* chore: remove unneeded comment

* chore: move scrollToDebounceTimeoutRef to the top of messageList

* fix: remove unused variables

* fix: update jest snapshot for the new prop addition for messageList

* fix: remove unused useEffect in messageList

* fix: fix the failing message list test

* fix: channel unread indicator was showing up even when marked read

* fix: scroll to bottom indicator must not be always shown on open

* Revert "fix: fix the failing message list test"

This reverts commit d605744.

* fix: sometimes the channel was not loaded to the latest message

* feat: add maxToRenderPerBatch prop to the messageList

this reduces the scrollToIndex failure rate

* fix: when scrolling to a non rendered item, highlight was not shown

* fix: update snapshot to fix unit test

* fix: scroll to messageId didnt work if scrollToFirstUnreadMsg is set

* fix: ensure setTargetedMessage happens after messages array is copied

* fix: simplified channel logic of messageId scroll

* fix: remove unnecessary console log

* docs: create a new version of docs for v5 (#1596)

* docs: upgrade helper guide (#1605)

* docs: cut a new version of docs for v5

* docs: upgrade helper guide

* docs: drop v5 changes from v4 docs

* docs: add reason for cameraroll fork replacement

* docs: make removing cameraroll to be non-optional

Co-authored-by: Santhosh Vaiyapuri <santhoshvai@gmail.com>

* feat: offline support (#1527)

* feat: offline support

* refactor: drop un-used function pragmaUserVersion

* refactor: reverting changes to ChannelPreview

* fix: fixing migration code issue

* refactor: drop limits from queries

* refactor: handling of message.read events and reads entity

* refactor: added members to separate table

* refactor: relative explicit query with left join to enrich users

* refactor: cleanup and reorganizing

* refactor: added index and foreign key constraints

* refactor: changes to be compatible with js client

* test: mocking sqlite

* refactor: added missing fields on channels table

* refactor: changes to be in sync with event from js client

* refactor: renaming table and cleanup of columns

* refactor: moving all quick-sqlite logic to separate class

* test: added simple test for offline support

* refactor: handle member related events

* refactor: added usage of sync api

* refactor: cleaning up un-necessary packages

* refactor: fixed issue with update message foreign key constraint failure

* refactor: cleanup

* refactor: cleanup

* tests: writing tests for offline feature

* refactor: removed dev-menu package and usage

* refactor: remove unused initDevMenu import

* refactor: remove un-used flipper packages

* docs: adding code comments

* refactor: removing unnecessary generics

* refactor: cleanup of mapper functions

* refactor: renaming offlineChannelsActive to staticChannelsActive

* refactor: removed metro config changes

* buiild: fixed issue with manual release script

* refactor: cleanup of handleEventToSyncDB

* refactor: fixing lint issue

* fix: removed redundant usage of useStreami18n hook

* fix: issue with message not displayed in static state

* build: updating stream-chat version

* fix: issue with appendWhereClause and undefined value

* fix: issue with members state initialization

* build: do not format on save

* fix: issue mapping memberCount to api response

* build: upgrading stream-chat version

* fix: reposition image error indicator

* refactor: use network error handling to useLoadingImage

* fix: image loading issue after network is recovered

* fix: logic for sync api and lint issues

* feat: point cameraroll dependency to the stream fork version

* fix: message.new event handling

* refactor: remove unnecessary client as hook dep

* refactor: code review changes

* refactor: moving offline support logic to Chat component

* refactor: fix lint and typescript issues

* refactor: handle quick-sqlite not being installed in better way

* docs: lint fixes

Co-authored-by: Mads Røskar <madshvero@gmail.com>
Co-authored-by: stevegalili <galiliziv@gmail.com>
Co-authored-by: Santhosh Vaiyapuri <santhoshvai@gmail.com>

* build: fix the doc paths

* docs: update upgrade helper doc

* docs: update upgrade helper doc with logout logic changes

* docs: change rc to beta in version label

* docs: add pod install command to upgrade helper

* build: update target branch name

* fix: flipper database plugin setup

* fix: handling of createSelectQuery function

* build: upgrade better sqlite and quick sqlite dev dep

* fix: open the selected image when pressed in the image grid (#1585)

* refactor!: rename gallery state objects to match their data

* fix: open the selected image when pressed in the image grid

* style: reorder object keys to be alphabetic

* chore: rename image grid props

* fix: rename setters to match new names

* test: update snapshots

* test: move bettersqlite test utils to src/test-utils

* test: gitignore locally generated database files

* style: reorder imports

* style: remove redundant whitespace

* refactor: export store apis

* fix: set an initial image index for the ImageGallery (#1639)

* docs: update upgrade helper guide

* docs: rename setImages and setImage

* docs: fix broken mdx imports

* docs: remove unintended newline in import

* docs: update upgrade helper guide

* feat: upgrade stream-chat to 7.0.0-offline-support.4

* docs: update upgrade helper guide

* docs: offline support docs

* ignore intelliJ files.

* Update broken external URLs

* Update broken internal links

* docs: replace cameraroll to our fork in install steps

* docs: upgrading v4 dopcs

* build: connection docusaurus gh action to develop and main

* feat: remove cameraroll fork and use v5 original cameraroll

* feat: upgrade stream-chat to v7 stable

Co-authored-by: Santhosh Vaiyapuri <santhoshvai@gmail.com>
Co-authored-by: Vishal Narkhede <vishalnarkhede.iitd@gmail.com>
Co-authored-by: Mads Røskar <madshvero@gmail.com>
Co-authored-by: Mads Røskar <mads.roskar@getstream.io>
Co-authored-by: nash0x7e2 <mail@neevash.dev>

docs: update upgrade-helper for v5

docs: update stream-chat version in docs

docs: make v5 docs stable
@github-actions github-actions Bot mentioned this pull request Aug 31, 2022
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants