Replies: 5 comments 9 replies
-
A small update: I started experimenting in this repo: ctholho/qryo The readme in that repo provides additional information, challenges and a roadmap – and there is an example Nuxt application. I'd love some feedback here in discussion. Does my chosen path make any sense at all? I have the feeling it's going towards an opinionated implementation of tanstack/query with some additional stuff thrown in. Because I that means making many choices for devs, I would love some validation and criticism. |
Beta Was this translation helpful? Give feedback.
-
Love it! I'm wondering if this could "just" be a setting in the regular SDK, instead of having to be a separate package 🙂 I'd love to be able to use something like this for the admin app as well! I'm very curious to hear some more thoughts around the different conflict resolution techniques though. It sounds like that's something that you might have to be able to override method call to method call (eg what happens if you need a different resolution for |
Beta Was this translation helpful? Give feedback.
-
I was exploring RxDB and I think it is a really good piece of software. It supports many backends, one of which is graphql. I think supporting it would be really nice and would solve the need for offline-first. Hasura has a nice write-up about it. |
Beta Was this translation helpful? Give feedback.
-
Maybe check out replicache.dev for some offline first ideas. |
Beta Was this translation helpful? Give feedback.
-
Could something like this do the job? What would be the drawbacks? urql offline support documentation |
Beta Was this translation helpful? Give feedback.
-
Directus offers great tools for building RESTful APIs when your internet connection is great. But, building offline-first applications is still very hard. This RFC suggests creating an offline-first SDK named
Indirectus
. It can be a community-maintained project and does not depend on upstream changes.Objectives
The objective of the proposed offline-first SDK, Indirectus, is to help developers create dependable applications that can function offline or in areas with low connectivity:
SPA Framework agnostic (should work with react, vue and svelte … (maybe angular?))
Observability of a modifier queue
Immediate, optimistic updates on the local client
Safe by default: Due to the inherent insecurity of local storage and IndexedDB, the SDK encrypts data at rest. Optionally you can turn off encryption for public data.
Offer ways to create custom endpoints for reading, updating and deleting to CRDTs or implement OT conflict merging strategies more easily.
Allow overwriting HTTPs routes for specific resources if they need special handling in a Directus custom endpoint
GET /custom-crdt/:collection/:id
PATCH /items/:collection/:id
Drop-in replacement for the directus-sdk:
In existing projects this allows to gradually migrate to offline-first queries. See this example:
Notice:
.indirectus()
uses tanstack/query and is basically sugar for this query:Under consideration
createOne
orcreateMany
requests?idempotent_id
. Developers must remember to generate and useidempotent_id
on the client to make idempotentcreateOne
calls.created_at_client
that has to be unique for theuser_id
andcreated_at_client
. E.g.: in postgres:UNIQUE (user_id, created_at_client)
requireIdempotency
is true and developers callcreateOne
without a specific keyNon-objectives
done_by
user id with a simple foreign key or (iii) with a many to many relation or (iv) an enumeration type … among others.version
,updated_at
,status
columns in order to handle conflicts during updates and deletion.Architecture
Client side:
Basically: Indirectus client is a wrapper for tanstack/query with added client side encryption.
wear
Backend additions:
Examples
Create a new item:
By default with retries or saving to an offline mutation cache and automatic cache invalidation:
Notice, we’re not explicitly using an idempotency key. In this proposal, we’re generating that automagically.
Optimistic offline updates:
We can offer sugar without disabling the expressibility of tanstack/query
Disable client-side storage and encryption if offline:
You’ll be able to set tanstack/query options, directus SDK options and Indirectus options.
Conflict Resolution Approaches
This section outlines four methods for handling edit conflicts in Directus. Some of these methods can be implemented with Directus custom endpoints or flows. Generally, conflict resolution falls into four categories, with developers selecting the most suitable strategy. Performance and usability with CRDTs have come a long way since their inception 2011. For most small to mid-scale Directus use cases CRDTs will work fine. They will probably still be a rarely used data type.
Last Write Wins (LWW): This default method resolves conflicts by prioritizing the most recent write operation in a CRUD app. Earlier edits are overwritten.
articles.updateOne(15, { content: 'The looser takes it all' })
Operational Transformation (OT): This method can be implemented using a Directus flow operating on the
article_event
data, updating the articles collection accordingly.articleEvents.updateOne(15, { content: { index: 23, operation: 'add', value: '!?', timestamp: 1678814119 } })
Conflict-free Replicated Data Types (CRDTs): This method employs data structures that automatically merge, resolving conflicts. CRDT implementation would be facilitated by Directus' support for views in relational databases. However, developers should be be able to fully choose their preferred implementation, because developers may utilize resources like Yjs or Automerge.
articleDeltas.updateOne(15, Y.encodeStateAsUpdate(Y.Doc))
Return Conflict List to Last Writer or Arbitrator: This approach requires handling user responses to conflicts, similar to git. It places a significant burden on users and developers, and should only be considered as a last resort for specific use cases.
Conclusion
Offline-first web apps are great for ensuring a consistent user experience in areas with poor network connectivity. Despite the current challenges for application developers, basically all of the groundwork has already been laid.
I'd be glad to hear your comments, experiences, and feedback on the proposed API.
Beta Was this translation helpful? Give feedback.
All reactions