Docs
Benchmarks

Benchmarks

The benchmarks demonstrate the loading performance and sizes of raw CRDT documents (Automerge & Yjs) compared to Secsync snapshot and updates.

The secsync version use encryption/decryption and don't do any of the optional signature checks. The raw CRDT documents are base64 encoded which adds about 30% to the size of the document compared to storing a binary document. This was chosen since often this data is transferred over the network and encoded using base64.

Yjs

10k Changes

This benchmark uses a document with 10k changes and the results show 3 different scenarios:

  1. The whole document as separate changes
  2. 9k changes as a snapshot and 1k changes as a separate changes
  3. The whole document as a snapshot

The results demonstrate that only encrypting changes increases the loading times of a Yjs document and also due the lost compression the size of the document increases. The loading time of a snapshot is significantly faster than loading changes and the size of the document is smaller.

When introducing Secsync the the impact is even larger. When combining Snapshots with Updates it's possible to get to acceptable loading times and sizes.

Yjs Loading time comparison Yjs Size comparison
Loading time (seconds)Size (KB)
Yjs (Changes)63421
Secsync + Yjs (Changes)11304451
Yjs (Snapshot of 9k changes + 1k changes)25055
Secsync + Yjs (Snapshot of 9k changes + 1k changes)370459
Yjs (Snapshot)0.812.9
Secsync + Yjs (Snapshot)1.513.4

Raw Data

Recorded on an Macbook Pro M1, 2020, 16 GB Memory

Snapshot + Updates Decryption: 116.48012506961823 milliseconds.
Snapshot + Updates Doc: 253.57566595077515 milliseconds.
Snapshot + Updates Decryption + Doc: 370.0557910203934 milliseconds.
Changes Decryption: 1072.7890410423279 milliseconds.
Changes Decryption + Doc: 57.17595899105072 milliseconds.
Changes Decryption + Doc: 1129.9650000333786 milliseconds.
Snapshot Decryption: 0.4355419874191284 milliseconds.
Snapshot Doc: 1.0230000019073486 milliseconds.
Snapshot Decryption + Doc: 1.458541989326477 milliseconds.
---
Snapshot with 1000 Changes Base64: 8.481458067893982 milliseconds.
Snapshot with 1000 Changes Doc: 250.2777919769287 milliseconds.
Changes Base64: 27.473582983016968 milliseconds.
Changes Doc: 62.76724994182587 milliseconds.
Snapshot Base64: 1.2406660318374634 milliseconds.
Snapshot Doc: 0.8804590702056885 milliseconds.

250k Changes

Loading time (seconds)Size (KB)
Yjs (Changes)161511234
Secsync + Yjs (Changes)29476116317
Yjs (Snapshot of 249k changes + 1k changes)151252
Secsync + Yjs (Snapshot of 249k changes + 1k changes)277659
Yjs (Snapshot)18213
Secsync + Yjs (Snapshot)35214

Raw Data

Recorded on an Macbook Pro M1, 2020, 16 GB Memory

> node loadYjs2.js

Snapshot with 1000 Changes Base64: 23.62625002861023 milliseconds.
Snapshot with 1000 Changes Doc: 151.11641597747803 milliseconds.
Changes Base64: 718.525083065033 milliseconds.
Changes Doc: 1615.3544169664383 milliseconds.
Snapshot Base64: 15.500166058540344 milliseconds.
Snapshot Doc: 18.124833941459656 milliseconds.

> node loadSecsyncYjs2.js

Snapshot + Updates Decryption: 125.45704102516174 milliseconds.
Snapshot + Updates Doc: 151.64654195308685 milliseconds.
Snapshot + Updates Decryption + Doc: 277.1035829782486 milliseconds.
Changes Decryption: 27888.64362502098 milliseconds.
Changes Decryption + Doc: 1587.4889999628067 milliseconds.
Changes Decryption + Doc: 29476.132624983788 milliseconds.
Snapshot Decryption: 3.7959580421447754 milliseconds.
Snapshot Doc: 31.969749927520752 milliseconds.
Snapshot Decryption + Doc: 35.76570796966553 milliseconds.

Automerge

10k Changes

This benchmark uses a document with 10k changes and the results show 3 different scenarios:

  1. The whole document as separate changes
  2. 9k changes as a snapshot and 1k changes as a separate changes
  3. The whole document as a snapshot

The results are similar to the Yjs benchmarks. The loading time of a snapshot is significantly faster than loading changes and the size of the document is smaller.

Automerge Loading time comparison Automerge Size comparison
Loading time (seconds)Size (KB)
Automerge (Changes)681442
Secsync + Automerge (Changes)11625471
Automerge (Snapshot of 9k changes + 1k changes)72152
Secsync + Automerge (Snapshot of 9k changes + 1k changes)188557
Automerge (Snapshot)39.99.3
Secsync + Automerge (Snapshot)38.29.8

Raw Data

Recorded on an Macbook Pro M1, 2020, 16 GB Memory

Snapshot + Updates Decryption: 120.0347501039505 milliseconds.
Snapshot + Updates Doc: 68.33024990558624 milliseconds.
Snapshot + Updates Decryption + Doc: 188.36500000953674 milliseconds.
Changes Decryption: 1095.1798330545425 milliseconds.
Changes Decryption + Doc: 67.0505839586258 milliseconds.
Changes Decryption + Doc: 1162.2304170131683 milliseconds.
Snapshot Decryption: 0.36016595363616943 milliseconds.
Snapshot Doc: 39.89745903015137 milliseconds.
Snapshot Decryption + Doc: 40.25762498378754 milliseconds.
---
Snapshot with 1000 Changes Base64: 0.6732079982757568 milliseconds.
Snapshot with 1000 Changes Doc: 71.84425008296967 milliseconds.
Changes Base64: 4.018791913986206 milliseconds.
Changes Doc: 67.62387502193451 milliseconds.
Snapshot Base64: 0.031916022300720215 milliseconds.
Snapshot Doc: 38.19574999809265 milliseconds.

250k Changes

Loading time (seconds)Size (KB)
Automerge (Changes)180638725
Secsync + Automerge (Changes)31119143835
Automerge (Snapshot of 9k changes + 1k changes)1115323
Secsync + Automerge (Snapshot of 9k changes + 1k changes)1233729
Automerge (Snapshot)1026172
Secsync + Automerge (Snapshot)1014173

Raw Data

Recorded on an Macbook Pro M1, 2020, 16 GB Memory

> node loadAutomerge.js

Snapshot with 1000 Changes Base64: 0.8246250152587891 milliseconds.
Snapshot with 1000 Changes Doc: 1115.5433340072632 milliseconds.
Changes Base64: 67.34670794010162 milliseconds.
Changes Doc: 1806.1128751039505 milliseconds.
Snapshot Base64: 0.13020896911621094 milliseconds.
Snapshot Doc: 1026.205708026886 milliseconds.

> node loadSecsyncAutomerge.js

Snapshot + Updates Decryption: 127.65812504291534 milliseconds.
Snapshot + Updates Doc: 1105.7897909879684 milliseconds.
Snapshot + Updates Decryption + Doc: 1233.4479160308838 milliseconds.
Changes Decryption: 29083.603749990463 milliseconds.
Changes Decryption + Doc: 2035.4658749103546 milliseconds.
Changes Decryption + Doc: 31119.069624900818 milliseconds.
Snapshot Decryption: 3.142209053039551 milliseconds.
Snapshot Doc: 1011.0410829782486 milliseconds.
Snapshot Decryption + Doc: 1014.1832920312881 milliseconds.

Conclusion

The benchmarks show that the impact of Secsync on loading times and sizes of a document is insignificant when using snapshots. The more CRDT document changes are stored as separate Secsync updates the larger the impact by the CRDT libraries Yjs and Automerge, but even more so by Secsync.

Having up to 1000 changes per snapshot still lead to reasonable numbers for loading times and sizes, but a lower number like 200-500 changes would be even better and is recommended.

Running the benchmarks

The source code for the benchmarks is available at https://github.com/serenity-kit/secsync/tree/main/benchmarks/snapshots (opens in a new tab)

The README.md (opens in a new tab) in this directory contains further details on the setup and how to run the benchmarks yourself.

Further ideas for Benchmarking

  • Benchmarking the Secsync sync engine and analyse the impact of the state machine
  • Benchmarking a real editor integration e.g. Prosemirror (loading the document, write performance)