Skip to content

feat(2883): Error handling improvements #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
islathehut merged 11 commits into from
Jun 13, 2025
Merged

feat(2883): Error handling improvements #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
8552083
Error handling improvements
islathehut Feb 13, 2025
8dacc62
User abortable-iterator package
islathehut Feb 13, 2025
e5ca22a
Update package-lock.json
islathehut May 16, 2025
850261f
Delete old homegrown abortable iterator file
islathehut May 19, 2025
536590a
Consolidate test databases under single parent directory + linting
islathehut Jun 11, 2025
1d6e07d
Fix sending join event that was breaking tests
islathehut Jun 11, 2025
a077906
Linting fix
islathehut Jun 11, 2025
9049ca5
Separate out linter so tests always run
islathehut Jun 11, 2025
6dc9201
Update run-linter.yml
islathehut Jun 11, 2025
184a0b0
Merge branch 'main' into feat/error-handling-improvements
islathehut Jun 12, 2025
c4eac5d
Linting fix
islathehut Jun 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/api-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ jobs:
node-version: 'lts/*'
registry-url: https://registry.npmjs.org/
- run: npm ci
- run: npm run lint
- run: npm run build:docs
- uses: actions/upload-pages-artifact@v3
with:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/npm-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ jobs:
node-version: 'lts/*'
registry-url: https://registry.npmjs.org/
- run: npm ci
- run: npm run lint
- run: npm test
- run: npm publish
env:
Expand Down
18 changes: 18 additions & 0 deletions .github/workflows/run-linter.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
name: Run Linter

on: [push, pull_request]

jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 'lts/*'
registry-url: https://registry.npmjs.org/
- name: Install dependencies
run: npm ci
- name: Run linter
run: npm run lint
4 changes: 0 additions & 4 deletions .github/workflows/run-test-node-min-ver.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ jobs:
registry-url: https://registry.npmjs.org/
- name: Install dependencies
run: npm ci
- name: Run linter
run: npm run lint
- name: Run tests
run: npm run test:ci
test-browser:
Expand All @@ -28,8 +26,6 @@ jobs:
registry-url: https://registry.npmjs.org/
- name: Install dependencies
run: npm ci
- name: Run linter
run: npm run lint
- name: Run a webrtc relay in the background
run: npm run webrtc:background
- name: Run browser tests
Expand Down
4 changes: 0 additions & 4 deletions .github/workflows/run-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ jobs:
registry-url: https://registry.npmjs.org/
- name: Install dependencies
run: npm ci
- name: Run linter
run: npm run lint
- name: Run tests
run: npm run test:ci
test-browser:
Expand All @@ -28,8 +26,6 @@ jobs:
registry-url: https://registry.npmjs.org/
- name: Install dependencies
run: npm ci
- name: Run linter
run: npm run lint
- name: Run a webrtc relay in the background
run: npm run webrtc:background
- name: Run browser tests
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ test/ipfs/
test/browser/ipfs/
test/browser/bundle.js*

# Don't track test databases
test-data/
benchmark-data/

# Don't track builds in git
# Distribution builds are available via npm
dist/
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/log-append.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ const queryLoop = async () => {

console.log('Benchmark duration is ' + benchmarkDuration + ' seconds')

await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/orbitdb')

const identities = await Identities()
const testIdentity = await identities.createIdentity({ id: 'userA' })
Expand Down Expand Up @@ -59,7 +59,7 @@ const queryLoop = async () => {
}
if (seconds >= benchmarkDuration) {
clearInterval(interval)
await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/orbitdb')
process.exit(0)
}
console.log(`${queriesPerSecond} queries per second, ${totalQueries} queries in ${seconds} seconds`)
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/log-iterator.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import { rimraf as rmrf } from 'rimraf'
;(async () => {
console.log('Starting benchmark...')

await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/orbitdb')

const identities = await Identities()
const testIdentity = await identities.createIdentity({ id: 'userA' })
Expand Down Expand Up @@ -55,7 +55,7 @@ import { rimraf as rmrf } from 'rimraf'

console.log(`Iterating ${all.length} entries took ${duration2} ms, ${operationsPerSecond2} ops/s, ${millisecondsPerOp2} ms/op`)

await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/orbitdb')

process.exit(0)
})()
8 changes: 4 additions & 4 deletions benchmarks/orbitdb-documents.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ EventEmitter.defaultMaxListeners = 10000

const entryCount = 1000

await rmrf('./ipfs')
await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/ipfs')
await rmrf('./benchmarks/test-data/orbitdb')

const ipfs = await createHelia()
const orbitdb = await createOrbitDB({ ipfs })
Expand Down Expand Up @@ -61,8 +61,8 @@ EventEmitter.defaultMaxListeners = 10000
await orbitdb.stop()
await ipfs.stop()

await rmrf('./ipfs')
await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/ipfs')
await rmrf('./benchmarks/test-data/orbitdb')

process.exit(0)
})()
8 changes: 4 additions & 4 deletions benchmarks/orbitdb-events.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ EventEmitter.defaultMaxListeners = 10000

const entryCount = 1000

await rmrf('./ipfs')
await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/ipfs')
await rmrf('./benchmarks/test-data/orbitdb')

const ipfs = await createHelia()
const orbitdb = await createOrbitDB({ ipfs })
Expand Down Expand Up @@ -53,8 +53,8 @@ EventEmitter.defaultMaxListeners = 10000
await orbitdb.stop()
await ipfs.stop()

await rmrf('./ipfs')
await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/ipfs')
await rmrf('./benchmarks/test-data/orbitdb')

process.exit(0)
})()
8 changes: 4 additions & 4 deletions benchmarks/orbitdb-keyvalue.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ EventEmitter.defaultMaxListeners = 10000

const entryCount = 1000

await rmrf('./ipfs')
await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/ipfs')
await rmrf('./benchmarks/test-data/orbitdb')

const ipfs = await createHelia()
const orbitdb = await createOrbitDB({ ipfs })
Expand Down Expand Up @@ -53,8 +53,8 @@ EventEmitter.defaultMaxListeners = 10000
await orbitdb.stop()
await ipfs.stop()

await rmrf('./ipfs')
await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/ipfs')
await rmrf('./benchmarks/test-data/orbitdb')

process.exit(0)
})()
8 changes: 4 additions & 4 deletions benchmarks/orbitdb-kv-indexed.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ EventEmitter.defaultMaxListeners = 10000

const entryCount = 1000

await rmrf('./ipfs')
await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/ipfs')
await rmrf('./benchmarks/test-data/orbitdb')

const ipfs = await createHelia()
const orbitdb = await createOrbitDB({ ipfs })
Expand Down Expand Up @@ -53,8 +53,8 @@ EventEmitter.defaultMaxListeners = 10000
await orbitdb.stop()
await ipfs.stop()

await rmrf('./ipfs')
await rmrf('./orbitdb')
await rmrf('./benchmarks/test-data/ipfs')
await rmrf('./benchmarks/test-data/orbitdb')

process.exit(0)
})()
12 changes: 6 additions & 6 deletions benchmarks/orbitdb-replicate.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@ EventEmitter.defaultMaxListeners = 10000

const entryCount = 1000

await rmrf('./orbitdb1')
await rmrf('./orbitdb2')
await rmrf('./benchmarks/test-data/orbitdb1')
await rmrf('./benchmarks/test-data/orbitdb2')

const [ipfs1, ipfs2] = await Promise.all([createHelia(), createHelia()])

const orbitdb1 = await createOrbitDB({ ipfs: ipfs1, directory: './orbitdb1' })
const orbitdb2 = await createOrbitDB({ ipfs: ipfs2, directory: './orbitdb2' })
const orbitdb1 = await createOrbitDB({ ipfs: ipfs1, directory: './benchmarks/test-data/orbitdb1' })
const orbitdb2 = await createOrbitDB({ ipfs: ipfs2, directory: './benchmarks/test-data/orbitdb2' })

await connectPeers(ipfs1, ipfs2)

Expand Down Expand Up @@ -67,8 +67,8 @@ EventEmitter.defaultMaxListeners = 10000
await ipfs1.stop()
await ipfs2.stop()

await rmrf('./orbitdb1')
await rmrf('./orbitdb2')
await rmrf('./benchmarks/test-data/orbitdb1')
await rmrf('./benchmarks/test-data/orbitdb2')

process.exit(0)
})()
14 changes: 12 additions & 2 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
"dependencies": {
"@ipld/dag-cbor": "^9.0.6",
"@libp2p/crypto": "^5.0.5",
"abortable-iterator": "5.1.0",
"it-pipe": "^3.0.1",
"level": "^8.0.0",
"lru": "^3.1.0",
Expand Down
53 changes: 32 additions & 21 deletions src/database.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ const defaultCacheSize = 1000
* @return {module:Databases~Database} An instance of Database.
* @instance
*/
const Database = async ({ ipfs, identity, address, name, access, directory, meta, headsStorage, entryStorage, indexStorage, referencesCount, syncAutomatically, onUpdate }) => {
const Database = async ({ ipfs, identity, address, name, access, directory, meta, headsStorage, entryStorage, indexStorage, referencesCount, syncAutomatically, onUpdate, events }) => {
/**
* @namespace module:Databases~Database
* @description The instance returned by {@link module:Database~Database}.
Expand Down Expand Up @@ -93,24 +93,27 @@ const Database = async ({ ipfs, identity, address, name, access, directory, meta
meta = meta || {}
referencesCount = Number(referencesCount) > -1 ? referencesCount : defaultReferencesCount

events = events || new EventEmitter()
Copy link
Collaborator

@adrastaea adrastaea Jun 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we care about passing in our own event emitter?

Copy link
Collaborator Author

@islathehut islathehut Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was just so we could share event emitters between different elements (e.g. storage and database instance).


entryStorage = entryStorage || await ComposedStorage(
await LRUStorage({ size: defaultCacheSize }),
await IPFSBlockStorage({ ipfs, pin: true })
await IPFSBlockStorage({ ipfs, pin: true }),
events
)

headsStorage = headsStorage || await ComposedStorage(
await LRUStorage({ size: defaultCacheSize }),
await LevelStorage({ path: pathJoin(directory, '/log/_heads/') })
await LevelStorage({ path: pathJoin(directory, '/log/_heads/') }),
events
)

indexStorage = indexStorage || await ComposedStorage(
await LRUStorage({ size: defaultCacheSize }),
await LevelStorage({ path: pathJoin(directory, '/log/_index/') })
await LevelStorage({ path: pathJoin(directory, '/log/_index/') }),
events
)

const log = await Log(identity, { logId: address, access, entryStorage, headsStorage, indexStorage })

const events = new EventEmitter()
const log = await Log(identity, { logId: address, access, entryStorage, headsStorage, indexStorage, events })

const queue = new PQueue({ concurrency: 1 })

Expand All @@ -125,13 +128,17 @@ const Database = async ({ ipfs, identity, address, name, access, directory, meta
*/
const addOperation = async (op) => {
const task = async () => {
const entry = await log.append(op, { referencesCount })
await sync.add(entry)
if (onUpdate) {
await onUpdate(log, entry)
try {
const entry = await log.append(op, { referencesCount })
await sync.add(entry)
if (onUpdate) {
await onUpdate(log, entry)
}
events.emit('update', entry)
return entry.hash
} catch (e) {
events.emit('error', e)
}
events.emit('update', entry)
return entry.hash
}
const hash = await queue.add(task)
await queue.onIdle()
Expand All @@ -140,15 +147,19 @@ const Database = async ({ ipfs, identity, address, name, access, directory, meta

const applyOperation = async (bytes) => {
const task = async () => {
const entry = await Entry.decode(bytes)
if (entry) {
const updated = await log.joinEntry(entry)
if (updated) {
if (onUpdate) {
await onUpdate(log, entry)
try {
const entry = await Entry.decode(bytes)
if (entry) {
const updated = await log.joinEntry(entry)
if (updated) {
if (onUpdate) {
await onUpdate(log, entry)
}
events.emit('update', entry)
}
events.emit('update', entry)
}
} catch (e) {
events.emit('error', e)
}
}
await queue.add(task)
Expand Down Expand Up @@ -241,7 +252,7 @@ const Database = async ({ ipfs, identity, address, name, access, directory, meta
* @memberof module:Databases~Database
* @instance
*/
access,
access
}
}

Expand Down
4 changes: 2 additions & 2 deletions src/databases/documents.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ const DefaultOptions = { indexBy: '_id' }
* @return {module:Databases.Databases-Documents} A Documents function.
* @memberof module:Databases
*/
const Documents = ({ indexBy } = DefaultOptions) => async ({ ipfs, identity, address, name, access, directory, meta, headsStorage, entryStorage, indexStorage, referencesCount, syncAutomatically, onUpdate }) => {
const database = await Database({ ipfs, identity, address, name, access, directory, meta, headsStorage, entryStorage, indexStorage, referencesCount, syncAutomatically })
const Documents = ({ indexBy } = DefaultOptions) => async ({ ipfs, identity, address, name, access, directory, meta, headsStorage, entryStorage, indexStorage, referencesCount, syncAutomatically, onUpdate, events }) => {
const database = await Database({ ipfs, identity, address, name, access, directory, meta, headsStorage, entryStorage, indexStorage, referencesCount, syncAutomatically, events })

const { addOperation, log } = database

Expand Down
Loading