Yjs
Shared data types for building collaborative software
README
A CRDT framework with a powerful abstraction of shared data
Yjs is a CRDT implementation that exposes its internal
data structure as shared types. Shared types are common data types like Map
or Array with superpowers: changes are automatically distributed to other
peers and merged without merge conflicts.
Yjs is network agnostic (p2p!), supports many existing rich text
editors, offline editing, version snapshots, undo/redo and
shared cursors. It scales well with an unlimited number of users and is well
suited for even large documents.
Discuss: https://discuss.yjs.dev
Benchmark Yjs vs. Automerge:
Podcast ["Yjs Deep Dive into real time collaborative editing solutions":](https://www.tag1consulting.com/blog/deep-dive-real-time-collaborative-editing-solutions-tagteamtalk-001-0)
Podcast ["Google Docs-style editing in Gutenberg with the YJS framework":](https://publishpress.com/blog/yjs/)
:construction_worker_woman: If you are looking for professional (paid) support to
build collaborative or distributed applications ping us at
Sponsors
I'm currently looking for sponsors that allow me to be less dependent on
contracting work. These awesome backers already fund further development of
Yjs:
[<img src="https://user-images.githubusercontent.com/5553757/83337333-a7bcb380-a2ba-11ea-837b-e404eb35d318.png"
[](https://room.sh/) [![JourneyApps](https://github.com/journeyapps.png?size=60)](https://github.com/journeyapps) [![Adam Brunnmeier](https://github.com/adabru.png?size=60)](https://github.com/adabru) [![Nathanael Anderson](https://github.com/NathanaelA.png?size=60)](https://github.com/NathanaelA) [![Gremloon](https://github.com/gremloon.png?size=60)](https://github.com/gremloon)Who is using Yjs
Relm A collaborative gameworld for teamwork and
community. :star2:
Input A collaborative note taking app. :star2:
Room.sh A meeting application with integrated
collaborative drawing, editing, and coding tools. :star:
A collaborative wiki that is edited by thousands of different people to work
on a rapid and sophisticated response to the coronavirus outbreak and
subsequent impacts. :star:
Nimbus Note A note-taking app designed by
Nimbus Web.
JoeDocs An open collaborative wiki.
Pluxbox RadioManager A web-based app to
collaboratively organize radio broadcasts.
Cattaz A wiki that can run custom applications in the
wiki pages.
Table of Contents
Overview
This repository contains a collection of shared types that can be observed for
changes and manipulated concurrently. Network functionality and two-way-bindings
are implemented in separate modules.
Bindings
Name | Cursors | Binding | Demo |
---|---|---|---|
|---|:-:|---|---| | |||
[ProseMirror](https://prosemirror.net/) | ✔ | [y-prosemirror](http://github.com/yjs/y-prosemirror) | [demo](https://demos.yjs.dev/prosemirror/prosemirror.html) |
[Quill](https://quilljs.com/) | ✔ | [y-quill](http://github.com/yjs/y-quill) | [demo](https://demos.yjs.dev/quill/quill.html) |
[CodeMirror](https://codemirror.net/) | ✔ | [y-codemirror](http://github.com/yjs/y-codemirror) | [demo](https://demos.yjs.dev/codemirror/codemirror.html) |
[Monaco](https://microsoft.github.io/monaco-editor/) | ✔ | [y-monaco](http://github.com/yjs/y-monaco) | [demo](https://demos.yjs.dev/monaco/monaco.html) |
Providers
Setting up the communication between clients, managing awareness information,
and storing shared data for offline usage is quite a hassle. Providers
manage all that for you and are the perfect starting point for your
collaborative app.
- y-webrtc
- Propagates document updates peer-to-peer using WebRTC. The peers exchangesignaling data over signaling servers. Publically available signaling serversare available. Communication over the signaling servers can be encrypted byproviding a shared secret, keeping the connection information and the shareddocument private.
- y-websocket
- A module that contains a simple websocket backend and a websocket client thatconnects to that backend. The backend can be extended to persist updates in aleveldb database.
- y-indexeddb
- Efficiently persists document updates to the browsers indexeddb database.The document is immediately available and only diffs need to be synced through thenetwork provider.
- y-dat
- [WIP] Write document updates effinciently to the dat network usingmultifeed. Each client hasan append-only log of CRDT local updates (hypercore). Multifeed manages and synchypercores and y-dat listens to changes and applies them to the Yjs document.
Getting Started
Install Yjs and a provider with your favorite package manager:
- ```sh
- npm i yjs y-websocket
- ```
Start the y-websocket server:
- ```sh
- PORT=1234 node ./node_modules/y-websocket/bin/server.js
- ```
Example: Observe types
- ``` js
- const yarray = doc.getArray('my-array')
- yarray.observe(event => {
- console.log('yarray was modified')
- })
- // every time a local or remote client modifies yarray, the observer is called
- yarray.insert(0, ['val']) // => "yarray was modified"
- ```
Example: Nest types
Remember, shared types are just plain old data types. The only limitation is
that a shared type must exist only once in the shared document.
- ``` js
- const ymap = doc.getMap('map')
- const foodArray = new Y.Array()
- foodArray.insert(0, ['apple', 'banana'])
- ymap.set('food', foodArray)
- ymap.get('food') === foodArray // => true
- ymap.set('fruit', foodArray) // => Error! foodArray is already defined
- ```
Now you understand how types are defined on a shared document. Next you can jump
to the demo repository or continue reading
the API docs.
Example: Using and combining providers
Any of the Yjs providers can be combined with each other. So you can sync data
over different network technologies.
In most cases you want to use a network provider (like y-websocket or y-webrtc)
in combination with a persistence provider (y-indexeddb in the browser).
Persistence allows you to load the document faster and to persist data that is
created while offline.
For the sake of this demo we combine two different network providers with a
persistence provider.
- ``` js
- import * as Y from 'yjs'
- import { WebrtcProvider } from 'y-webrtc'
- import { WebsocketProvider } from 'y-websocket'
- import { IndexeddbPersistence } from 'y-indexeddb'
- const ydoc = new Y.Doc()
- // this allows you to instantly get the (cached) documents data
- const indexeddbProvider = new IndexeddbPersistence('count-demo', ydoc)
- idbP.whenSynced.then(() => {
- console.log('loaded data from indexed db')
- })
- // Sync clients with the y-webrtc provider.
- const webrtcProvider = new WebrtcProvider('count-demo', ydoc)
- // Sync clients with the y-websocket provider
- const websocketProvider = new WebsocketProvider(
- 'wss://demos.yjs.dev', 'count-demo', ydoc
- )
- // array of numbers which produce a sum
- const yarray = ydoc.getArray('count')
- // observe changes of the sum
- yarray.observe(event => {
- // print updates when the data changes
- console.log('new sum: ' + yarray.toArray().reduce((a,b) => a + b))
- })
- // add 1 to the sum
- yarray.push([1]) // => "new sum: 1"
- ```
API
- ``` js
- import * as Y from 'yjs'
- ```
Shared Types
Y.Array
A shareable Array-like type that supports efficient insert/delete of elements
at any position. Internally it uses a linked list of Arrays that is split when
necessary.
const yarray = new Y.Array()
- Insert content at index. Note that content is an array of elements.I.e.
array.insert(0, [1])
splices the list and inserts 1 atposition 0. - Copies the content of this YArray to a new Array.
- Copies the content of this YArray to a new Array. It transforms all child typesto JSON using their
toJSON
method. - Returns an YArray Iterator that contains the values for each index in the array.
for (let value of yarray) { .. }
- Adds an event listener to this type that will be called synchronously every timethis type is modified. In the case this type is modified in the event listener,the event listener will be called again after the current event listener returns.
- Removes an
observe
event listener from this type. - Adds an event listener to this type that will be called synchronously every timethis type or any of its children is modified. In the case this type is modifiedin the event listener, the event listener will be called again after the currentevent listener returns. The event listener receives all Events created by itselfor any of its children.
- Removes an
observeDeep
event listener from this type.
insert(index:number, content:Array<object|boolean|Array|string|number|Uint8Array|Y.Type>)
push(Array<Object|boolean|Array|string|number|Uint8Array|Y.Type>)
unshift(Array<Object|boolean|Array|string|number|Uint8Array|Y.Type>)
delete(index:number, length:number)
get(index:number)
length:number
forEach(function(value:object|boolean|Array|string|number|Uint8Array|Y.Type, index:number, array: Y.Array))
map(function(T, number, YArray):M):Array<M>
toArray():Array<object|boolean|Array|string|number|Uint8Array|Y.Type>
toJSON():Array<Object|boolean|Array|string|number>
[Symbol.Iterator]
observe(function(YArrayEvent, Transaction):void)
unobserve(function(YArrayEvent, Transaction):void)
observeDeep(function(Array<YEvent>, Transaction):void)
unobserveDeep(function(Array<YEvent>, Transaction):void)
Y.Map
A shareable Map type.
const ymap = new Y.Map()
- Copies the
[key,value]
pairs of this YMap to a new Object.Ittransforms all child types to JSON using theirtoJSON
method. - Execute the provided function once for every key-value pair.
- Returns an Iterator of
[key, value]
pairs.for (let [key, value] of ymap) { .. }
- Returns an Iterator of
[key, value]
pairs. - Returns an Iterator of all values.
- Returns an Iterator of all keys.
- Adds an event listener to this type that will be called synchronously every timethis type is modified. In the case this type is modified in the event listener,the event listener will be called again after the current event listener returns.
- Removes an
observe
event listener from this type. - Adds an event listener to this type that will be called synchronously every timethis type or any of its children is modified. In the case this type is modifiedin the event listener, the event listener will be called again after the currentevent listener returns. The event listener receives all Events created by itselfor any of its children.
- Removes an
observeDeep
event listener from this type.
get(key:string):object|boolean|string|number|Uint8Array|Y.Type
set(key:string, value:object|boolean|string|number|Uint8Array|Y.Type)
delete(key:string)
has(key:string):boolean
get(index:number)
toJSON():Object<string, Object|boolean|Array|string|number|Uint8Array>
forEach(function(value:object|boolean|Array|string|number|Uint8Array|Y.Type, key:string, map: Y.Map))
[Symbol.Iterator]
entries()
values()
keys()
observe(function(YMapEvent, Transaction):void)
unobserve(function(YMapEvent, Transaction):void)
observeDeep(function(Array<YEvent>, Transaction):void)
unobserveDeep(function(Array<YEvent>, Transaction):void)
Y.Text
A shareable type that is optimized for shared editing on text. It allows to
assign properties to ranges in the text. This makes it possible to implement
rich-text bindings to this type.
This type can also be transformed to the
delta format. Similarly theYTextEvents compute changes as deltas.
const ytext = new Y.Text()
- Insert a string at index and assign formatting attributes to it.
ytext.insert(0, 'bold text', { bold: true })
- Assign formatting attributes to a range in the text
- See Quill DeltaCan set options for preventing remove ending newLines, default is true.
ytext.applyDelta(delta, { sanitize: false })
- Transforms this type, without formatting options, into a string.
- See
toString
- Transforms this type to a Quill Delta
- Adds an event listener to this type that will be called synchronously every timethis type is modified. In the case this type is modified in the event listener,the event listener will be called again after the current event listener returns.
- Removes an
observe
event listener from this type. - Adds an event listener to this type that will be called synchronously every timethis type or any of its children is modified. In the case this type is modifiedin the event listener, the event listener will be called again after the currentevent listener returns. The event listener receives all Events created by itselfor any of its children.
- Removes an
observeDeep
event listener from this type.
insert(index:number, content:string, [formattingAttributes:Object<string,string>])
delete(index:number, length:number)
format(index:number, length:number, formattingAttributes:Object<string,string>)
applyDelta(delta, opts:Object<string,any>)
length:number
toString():string
toJSON():string
toDelta():Delta
observe(function(YTextEvent, Transaction):void)
unobserve(function(YTextEvent, Transaction):void)
observeDeep(function(Array<YEvent>, Transaction):void)
unobserveDeep(function(Array<YEvent>, Transaction):void)
Y.XmlFragment
A container that holds an Array of Y.XmlElements.
const yxml = new Y.XmlFragment()
- Copies the children to a new Array.
- Transforms this type and all children to new DOM elements.
- Get the XML serialization of all descendants.
- See
toString
. - Adds an event listener to this type that will be called synchronously every timethis type is modified. In the case this type is modified in the event listener,the event listener will be called again after the current event listener returns.
- Removes an
observe
event listener from this type. - Adds an event listener to this type that will be called synchronously every timethis type or any of its children is modified. In the case this type is modifiedin the event listener, the event listener will be called again after the currentevent listener returns. The event listener receives all Events created by itselfor any of its children.
- Removes an
observeDeep
event listener from this type.
insert(index:number, content:Array<Y.XmlElement|Y.XmlText>)
delete(index:number, length:number)
get(index:number)
length:number
toArray():Array<Y.XmlElement|Y.XmlText>
toDOM():DocumentFragment
toString():string
toJSON():string
observe(function(YXmlEvent, Transaction):void)
unobserve(function(YXmlEvent, Transaction):void)
observeDeep(function(Array<YEvent>, Transaction):void)
unobserveDeep(function(Array<YEvent>, Transaction):void)
Y.XmlElement
A shareable type that represents an XML Element. It has a nodeName
,
attributes, and a list of children. But it makes no effort to validate its
content and be actually XML compliant.
const yxml = new Y.XmlElement()
- Copies the children to a new Array.
- Transforms this type and all children to a new DOM element.
- Get the XML serialization of all descendants.
- See
toString
. - Adds an event listener to this type that will be called synchronously everytime this type is modified. In the case this type is modified in the eventlistener, the event listener will be called again after the current eventlistener returns.
- Removes an
observe
event listener from this type. - Adds an event listener to this type that will be called synchronously every timethis type or any of its children is modified. In the case this type is modifiedin the event listener, the event listener will be called again after the currentevent listener returns. The event listener receives all Events created by itselfor any of its children.
- Removes an
observeDeep
event listener from this type.
insert(index:number, content:Array<Y.XmlElement|Y.XmlText>)
delete(index:number, length:number)
get(index:number)
length:number
setAttribute(attributeName:string, attributeValue:string)
removeAttribute(attributeName:string)
getAttribute(attributeName:string):string
getAttributes(attributeName:string):Object<string,string>
toArray():Array<Y.XmlElement|Y.XmlText>
toDOM():Element
toString():string
toJSON():string
observe(function(YXmlEvent, Transaction):void)
unobserve(function(YXmlEvent, Transaction):void)
observeDeep(function(Array<YEvent>, Transaction):void)
unobserveDeep(function(Array<YEvent>, Transaction):void)
Y.Doc
- ``` js
- const doc = new Y.Doc()
- ```
- A unique id that identifies this client. (readonly)
- Whether garbage collection is enabled on this doc instance. Set doc.gc = falsein order to disable gc and be able to restore old content. See https://github.com/yjs/yjs#yjs-crdt-algorithmfor more information about gc in Yjs.
- Every change on the shared document happens in a transaction. Observer calls andthe
update
event are called after each transaction. You shouldbundle changes into a single transaction to reduce the amount of eventcalls. I.e.doc.transact(() => { yarray.insert(..); ymap.set(..) })
triggers a single change event.
You can specify an optionalorigin
parameter that is stored ontransaction.origin
andon('update', (update, origin) => ..)
. - Define a shared type.
- Define a shared Y.Array type. Is equivalent to
y.get(string, Y.Array)
. - Define a shared Y.Map type. Is equivalent to
y.get(string, Y.Map)
. - Define a shared Y.XmlFragment type. Is equivalent to
y.get(string, Y.XmlFragment)
. - Register an event listener on the shared type
- Unregister an event listener from the shared type
clientID
gc
transact(function(Transaction):void [, origin:any])
get(string, Y.[TypeClass]):[Type]
getArray(string):Y.Array
getMap(string):Y.Map
getXmlFragment(string):Y.XmlFragment
on(string, function)
off(string, function)
Y.Doc Events
- Listen to document updates. Document updates must be transmitted to all otherpeers. You can apply document updates in any order and multiple times.
- Emitted before each transaction.
- Emitted after each transaction.
on('update', function(updateMessage:Uint8Array, origin:any, Y.Doc):void)
on('beforeTransaction', function(Y.Transaction, Y.Doc):void)
on('afterTransaction', function(Y.Transaction, Y.Doc):void)
Document Updates
Changes on the shared document are encoded into document updates. Document
updates are commutative and idempotent. This means that they can be applied
in any order and multiple times.
Example: Listen to update events and apply them on remote client
- ``` js
- const doc1 = new Y.Doc()
- const doc2 = new Y.Doc()
- doc1.on('update', update => {
- Y.applyUpdate(doc2, update)
- })
- doc2.on('update', update => {
- Y.applyUpdate(doc1, update)
- })
- // All changes are also applied to the other document
- doc1.getArray('myarray').insert(0, ['Hello doc2, you got this?'])
- doc2.getArray('myarray').get(0) // => 'Hello doc2, you got this?'
- ```
Yjs internally maintains a state vector that denotes the next
expected clock from each client. In a different interpretation it holds the
number of structs created by each client. When two clients sync, you can either
exchange the complete document structure or only the differences by sending the
state vector to compute the differences.
Example: Sync two clients by exchanging the complete document structure
- ``` js
- const state1 = Y.encodeStateAsUpdate(ydoc1)
- const state2 = Y.encodeStateAsUpdate(ydoc2)
- Y.applyUpdate(ydoc1, state2)
- Y.applyUpdate(ydoc2, state1)
- ```
Example: Sync two clients by computing the differences
This example shows how to sync two clients with the minimal amount of exchanged
data by computing only the differences using the state vector of the remote
client. Syncing clients using the state vector requires another roundtrip, but
can safe a lot of bandwidth.
- ``` js
- const stateVector1 = Y.encodeStateVector(ydoc1)
- const stateVector2 = Y.encodeStateVector(ydoc2)
- const diff1 = Y.encodeStateAsUpdate(ydoc1, stateVector2)
- const diff2 = Y.encodeStateAsUpdate(ydoc2, stateVector1)
- Y.applyUpdate(ydoc1, diff2)
- Y.applyUpdate(ydoc2, diff1)
- ```
- Apply a document update on the shared document. Optionally you can specify
transactionOrigin
that will be stored ontransaction.origin
andydoc.on('update', (update, origin) => ..)
. - Encode the document state as a single update message that can be applied on theremote document. Optionally specify the target state vector to only write thedifferences to the update message.
- Computes the state vector and encodes it into an Uint8Array.
Y.applyUpdate(Y.Doc, update:Uint8Array, [transactionOrigin:any])
Y.encodeStateAsUpdate(Y.Doc, [encodedTargetStateVector:Uint8Array]):Uint8Array
Y.encodeStateVector(Y.Doc):Uint8Array
Relative Positions
This API is not stable yet
This feature is intended for managing selections / cursors. When working with
other users that manipulate the shared document, you can't trust that an index
position (an integer) will stay at the intended location. A relative position
is fixated to an element in the shared document and is not affected by remote
changes. I.e. given the document "a|c", the relative position is attached to
c. When a remote user modifies the document by inserting a character before
the cursor, the cursor will stay attached to the character c. `insert(1,
'x')("a|c") = "ax|c"`. When the relative position is set to the end of the
document, it will stay attached to the end of the document.
Example: Transform to RelativePosition and back
- ``` js
- const relPos = Y.createRelativePositionFromTypeIndex(ytext, 2)
- const pos = Y.createAbsolutePositionFromRelativePosition(relPos, doc)
- pos.type === ytext // => true
- pos.index === 2 // => true
- ```
Example: Send relative position to remote client (json)
- ``` js
- const relPos = Y.createRelativePositionFromTypeIndex(ytext, 2)
- const encodedRelPos = JSON.stringify(relPos)
- // send encodedRelPos to remote client..
- const parsedRelPos = JSON.parse(encodedRelPos)
- const pos = Y.createAbsolutePositionFromRelativePosition(parsedRelPos, remoteDoc)
- pos.type === remoteytext // => true
- pos.index === 2 // => true
- ```
Example: Send relative position to remote client (Uint8Array)
- ``` js
- const relPos = Y.createRelativePositionFromTypeIndex(ytext, 2)
- const encodedRelPos = Y.encodeRelativePosition(relPos)
- // send encodedRelPos to remote client..
- const parsedRelPos = Y.decodeRelativePosition(encodedRelPos)
- const pos = Y.createAbsolutePositionFromRelativePosition(parsedRelPos, remoteDoc)
- pos.type === remoteytext // => true
- pos.index === 2 // => true
- ```
Y.createRelativePositionFromTypeIndex(Uint8Array|Y.Type, number)
Y.createAbsolutePositionFromRelativePosition(RelativePosition, Y.Doc)
Y.encodeRelativePosition(RelativePosition):Uint8Array
Y.decodeRelativePosition(Uint8Array):RelativePosition
Y.UndoManager
Yjs ships with an Undo/Redo manager for selective undo/redo of of changes on a
Yjs type. The changes can be optionally scoped to transaction origins.
- ``` js
- const ytext = doc.getText('text')
- const undoManager = new Y.UndoManager(ytext)
- ytext.insert(0, 'abc')
- undoManager.undo()
- ytext.toString() // => ''
- undoManager.redo()
- ytext.toString() // => 'abc'
- ```
- Accepts either single type as scope or an array of types.
- Register an event that is called when a
StackItem
is added to theundo- or the redo-stack. - Register an event that is called when a
StackItem
is popped fromthe undo- or the redo-stack.
constructor(scope:Y.AbstractType|Array<Y.AbstractType> [, {captureTimeout:number,trackedOrigins:Set<any>,deleteFilter:function(item):boolean}])
undo()
redo()
stopCapturing()
on('stack-item-added', { stackItem: { meta: Map<any,any> }, type: 'undo'| 'redo' })
on('stack-item-popped', { stackItem: { meta: Map<any,any> }, type: 'undo'| 'redo' })
Example: Stop Capturing
UndoManager merges Undo-StackItems if they are created within time-gap
smaller than options.captureTimeout. Call um.stopCapturing() so that the next
StackItem won't be merged.
- ``` js
- // without stopCapturing
- ytext.insert(0, 'a')
- ytext.insert(1, 'b')
- undoManager.undo()
- ytext.toString() // => '' (note that 'ab' was removed)
- // with stopCapturing
- ytext.insert(0, 'a')
- undoManager.stopCapturing()
- ytext.insert(0, 'b')
- undoManager.undo()
- ytext.toString() // => 'a' (note that only 'b' was removed)
- ```
Example: Specify tracked origins
Every change on the shared document has an origin. If no origin was specified,
it defaults to null. By specifying trackedOrigins you can
selectively specify which changes should be tracked by UndoManager. The
UndoManager instance is always added to trackedOrigins.
- ``` js
- class CustomBinding {}
- const ytext = doc.getText('text')
- const undoManager = new Y.UndoManager(ytext, {
- trackedOrigins: new Set([42, CustomBinding])
- })
- ytext.insert(0, 'abc')
- undoManager.undo()
- ytext.toString() // => 'abc' (does not track because origin `null` and not part
- // of `trackedTransactionOrigins`)
- ytext.delete(0, 3) // revert change
- doc.transact(() => {
- ytext.insert(0, 'abc')
- }, 42)
- undoManager.undo()
- ytext.toString() // => '' (tracked because origin is an instance of `trackedTransactionorigins`)
- doc.transact(() => {
- ytext.insert(0, 'abc')
- }, 41)
- undoManager.undo()
- ytext.toString() // => '' (not tracked because 41 is not an instance of
- // `trackedTransactionorigins`)
- ytext.delete(0, 3) // revert change
- doc.transact(() => {
- ytext.insert(0, 'abc')
- }, new CustomBinding())
- undoManager.undo()
- ytext.toString() // => '' (tracked because origin is a `CustomBinding` and
- // `CustomBinding` is in `trackedTransactionorigins`)
- ```
Example: Add additional information to the StackItems
When undoing or redoing a previous action, it is often expected to restore
additional meta information like the cursor location or the view on the
document. You can assign meta-information to Undo-/Redo-StackItems.
- ``` js
- const ytext = doc.getText('text')
- const undoManager = new Y.UndoManager(ytext, {
- trackedOrigins: new Set([42, CustomBinding])
- })
- undoManager.on('stack-item-added', event => {
- // save the current cursor location on the stack-item
- event.stackItem.meta.set('cursor-location', getRelativeCursorLocation())
- })
- undoManager.on('stack-item-popped', event => {
- // restore the current cursor location on the stack-item
- restoreCursorLocation(event.stackItem.meta.get('cursor-location'))
- })
- ```
Miscellaneous
Typescript Declarations
Yjs has type descriptions. But until [this
ticket](https://github.com/Microsoft/TypeScript/issues/7546) is fixed, this is
how you can make use of Yjs type declarations.
- ``` json
- {
- "compilerOptions": {
- "allowJs": true,
- "checkJs": true,
- },
- "maxNodeModuleJsDepth": 5
- }
- ```
Yjs CRDT Algorithm
Conflict-free replicated data types (CRDT) for collaborative editing are an
alternative approach to operational transformation (OT). A very simple
differenciation between the two approaches is that OT attempts to transform
index positions to ensure convergence (all clients end up with the same
content), while CRDTs use mathematical models that usually do not involve index
transformations, like linked lists. OT is currently the de-facto standard for
shared editing on text. OT approaches that support shared editing without a
central source of truth (a central server) require too much bookkeeping to be
viable in practice. CRDTs are better suited for distributed systems, provide
additional guarantees that the document can be synced with remote clients, and
do not require a central source of truth.
Yjs implements a modified version of the algorithm described in [this
paper](https://www.researchgate.net/publication/310212186_Near_Real-Time_Peer-to-Peer_Shared_Editing_on_Extensible_Data_Types).
I will eventually publish a paper that describes why this approach works so well
in practice. Note: Since operations make up the document structure, we prefer
the term struct now.
CRDTs suitable for shared text editing suffer from the fact that they only grow
in size. There are CRDTs that do not grow in size, but they do not have the
characteristics that are benificial for shared text editing (like intention
preservation). Yjs implements many improvements to the original algorithm that
diminish the trade-off that the document only grows in size. We can't garbage
collect deleted structs (tombstones) while ensuring a unique order of the
structs. But we can 1. merge preceeding structs into a single struct to reduce
the amount of meta information, 2. we can delete content from the struct if it
is deleted, and 3. we can garbage collect tombstones if we don't care about the
order of the structs anymore (e.g. if the parent was deleted).
Examples:
1. If a user inserts elements in sequence, the struct will be merged into a
single struct. E.g. array.insert(0, ['a']), array.insert(0, ['b']); is
first represented as two structs (`[{id: {client, clock: 0}, content: 'a'},
{id: {client, clock: 1}, content: 'b'}`) and then merged into a single
struct: [{id: {client, clock: 0}, content: 'ab'}].
2. When a struct that contains content (e.g. ItemString) is deleted, the
struct will be replaced with an ItemDeleted that does not contain content
anymore.
3. When a type is deleted, all child elements are transformed to GC structs. A
GC struct only denotes the existence of a struct and that it is deleted.
GC structs can always be merged with other GC structs if the id's are
adjacent.
Especially when working on structured content (e.g. shared editing on
ProseMirror), these improvements yield very good results when
benchmarking random document edits.
In practice they show even better results, because users usually edit text in
sequence, resulting in structs that can easily be merged. The benchmarks show
that even in the worst case scenario that a user edits text from right to left,
Yjs achieves good performance even for huge documents.
State Vector
Yjs has the ability to exchange only the differences when syncing two clients.
We use lamport timestamps to identify structs and to track in which order a
client created them. Each struct has an `struct.id = { client: number, clock:
number} that uniquely identifies a struct. We define the next expected clock`
by each client as the state vector. This data structure is similar to the
version vectors data structure.
But we use state vectors only to describe the state of the local document, so we
can compute the missing struct of the remote client. We do not use it to track
causality.
License and Author
Yjs and all related projects are [MIT licensed](./LICENSE).
Yjs is based on my research as a student at the [RWTH
i5](http://dbis.rwth-aachen.de/). Now I am working on Yjs in my spare time.
Fund this project by donating on Patreon or
hiring me for professional support.