feat(hermes): send OnNewDoc event #691

no30bit · 2025-11-25T11:14:51Z

Description

Send OnNewDoc event to guests.

Related Issue(s)

Closes #629

Description of Changes

Tracking app subscriptions
Separate from regular ipfs subscriptions

Breaking Changes

Describe any breaking changes and the impact.

Screenshots

If applicable, add screenshots to help explain your changes.

Related Pull Requests

If applicable, list any related pull requests.

Please confirm the following checks

My code follows the style guidelines of this project
I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
Any dependent changes have been merged and published in downstream module

Replace async hermes-ipfs library with synchronous WIT bindings (file_add, file_pin, pubsub_publish, pubsub_subscribe). Add actual channel subscription in SyncChannel::new() and document complete pub/sub flow with PR #691 infrastructure. Changes: - Use WIT IPFS functions directly instead of async library - Call pubsub_subscribe() to register DocSync subscriptions - Document how on_new_doc events are triggered by PR #691 - Remove conditional compilation and async dependencies - Show clear 4-step workflow: add → pin → validate → publish

Add comprehensive comments explaining PR #691 requirement for subscription event routing. Document the complete pub/sub flow, what works now vs what needs PR #691, and how to integrate when it merges

Fix compilation error and clarify that publishing to PubSub works now; PR #691 is only needed to route incoming messages to event handlers.

* feat: implement doc-sync channel::post API for issue #628 - Add channel::post(document_bytes) API as requested - Integrate hermes-ipfs library for IPFS operations - Add HTTP endpoints for testing (/api/doc-sync/*) - Configure HTTP gateway routing - Implement IPFS add, pin, and PubSub publish workflow * feat: implement doc-sync channel::post API for issue #628 - Add channel::post(document_bytes) API as requested - Integrate hermes-ipfs library for IPFS operations - Add HTTP endpoints for testing (/api/doc-sync/*) - Configure HTTP gateway routing - Implement IPFS add, pin, and PubSub publish workflow * hermes ipfs version * fix: enable WASM compilation for doc-sync module and add host stubs WASM compilation fixes: - Made tokio runtime features conditional (rt-multi-thread only for non-WASM) - Added separate WASM/native implementations using futures::executor for WASM - Conditionally compile Runtime usage and block_on calls Host implementation: - Replaced panicking todo!() with warning messages and stub return values - Added Resource stub creation for SyncChannel::new() - Functions now print warnings but don't crash runtime This allows the doc-sync module to compile for wasm32-wasip2 targets and run without panicking, though full functionality requires proper host implementation. * Remove all cfg attributes and consolidate WASM/non-WASM code paths into single implementations using futures::executor. Simplify HTTP handlers and reduce complexity to clearly demonstrate the 4-step workflow. * refactor(doc-sync): simplify for demo workflow Remove conditional compilation, OnceLock pattern, and unnecessary complexity to clearly show the 4-step IPFS PubSub workflow. * refactor(doc-sync): use WIT bindings directly for demo Replace async hermes-ipfs library with direct WIT function calls (file_add, file_pin, pubsub_publish). Remove conditional compilation and async dependencies to simplify the 4-step workflow demo. * refactor(doc-sync): use WIT bindings directly for demo Replace async hermes-ipfs library with direct WIT function calls (file_add, file_pin, pubsub_publish). Remove conditional compilation and async dependencies to simplify the 4-step workflow demo. * refactor(doc-sync): integrate with PR #691 subscription flow Replace async hermes-ipfs library with synchronous WIT bindings (file_add, file_pin, pubsub_publish, pubsub_subscribe). Add actual channel subscription in SyncChannel::new() and document complete pub/sub flow with PR #691 infrastructure. Changes: - Use WIT IPFS functions directly instead of async library - Call pubsub_subscribe() to register DocSync subscriptions - Document how on_new_doc events are triggered by PR #691 - Remove conditional compilation and async dependencies - Show clear 4-step workflow: add → pin → validate → publish * docs(doc-sync): document PR #691 integration requirements Add comprehensive comments explaining PR #691 requirement for subscription event routing. Document the complete pub/sub flow, what works now vs what needs PR #691, and how to integrate when it merges * fix(doc-sync): import GuestSyncChannel trait and clarify PR #691 comment Fix compilation error and clarify that publishing to PubSub works now; PR #691 is only needed to route incoming messages to event handlers. * update docs * refactor(doc-sync): Focus module on publishing workflow only Remove subscription logic and simplify documentation. Module now demonstrates only the 4-step publishing workflow: file_add, file_pin, pre-publish validation, and pubsub_publish. * fmt * refactor(doc-sync): Move post logic to host Execute the 4-step publishing workflow (file_add, file_pin, pre-publish, pubsub_publish) on the host side instead of in the WASM module. Reduces boundary crossings from 6 to 2 for better performance. * fmt * fmt * fix(doc-sync): Fix compilation errors - Fix SyncChannel resource import and usage - Update channel::post() to call host implementation correctly * refactor(doc-sync): Replace eprintln with tracing Use tracing macros (info/warn/error) instead of eprintln for logging in doc-sync host implementation. * refactor(doc-sync): Replace eprintln with tracing Use tracing macros (info/warn/error) instead of eprintln for logging in doc-sync host implementation. * fmt * refactor(doc-sync): Extract constants and improve error logging - Add DOC_SYNC_TOPIC and DOC_SYNC_CHANNEL constants - Add error logging to id_for() method * refactor(doc-sync): Improve error logging and remove redundant conversions - Log actual error details instead of discarding with - Remove redundant .to_string() on CID response - Add error logging for failed post operations * fmt

hermes/bin/src/runtime_extensions/hermes/doc_sync/mod.rs

hermes/bin/src/runtime_extensions/hermes/doc_sync/host.rs

* feat: implement doc-sync channel::post API for issue #628 - Add channel::post(document_bytes) API as requested - Integrate hermes-ipfs library for IPFS operations - Add HTTP endpoints for testing (/api/doc-sync/*) - Configure HTTP gateway routing - Implement IPFS add, pin, and PubSub publish workflow * feat: implement doc-sync channel::post API for issue #628 - Add channel::post(document_bytes) API as requested - Integrate hermes-ipfs library for IPFS operations - Add HTTP endpoints for testing (/api/doc-sync/*) - Configure HTTP gateway routing - Implement IPFS add, pin, and PubSub publish workflow * hermes ipfs version * fix: enable WASM compilation for doc-sync module and add host stubs WASM compilation fixes: - Made tokio runtime features conditional (rt-multi-thread only for non-WASM) - Added separate WASM/native implementations using futures::executor for WASM - Conditionally compile Runtime usage and block_on calls Host implementation: - Replaced panicking todo!() with warning messages and stub return values - Added Resource stub creation for SyncChannel::new() - Functions now print warnings but don't crash runtime This allows the doc-sync module to compile for wasm32-wasip2 targets and run without panicking, though full functionality requires proper host implementation. * Remove all cfg attributes and consolidate WASM/non-WASM code paths into single implementations using futures::executor. Simplify HTTP handlers and reduce complexity to clearly demonstrate the 4-step workflow. * refactor(doc-sync): simplify for demo workflow Remove conditional compilation, OnceLock pattern, and unnecessary complexity to clearly show the 4-step IPFS PubSub workflow. * refactor(doc-sync): use WIT bindings directly for demo Replace async hermes-ipfs library with direct WIT function calls (file_add, file_pin, pubsub_publish). Remove conditional compilation and async dependencies to simplify the 4-step workflow demo. * refactor(doc-sync): use WIT bindings directly for demo Replace async hermes-ipfs library with direct WIT function calls (file_add, file_pin, pubsub_publish). Remove conditional compilation and async dependencies to simplify the 4-step workflow demo. * refactor(doc-sync): integrate with PR #691 subscription flow Replace async hermes-ipfs library with synchronous WIT bindings (file_add, file_pin, pubsub_publish, pubsub_subscribe). Add actual channel subscription in SyncChannel::new() and document complete pub/sub flow with PR #691 infrastructure. Changes: - Use WIT IPFS functions directly instead of async library - Call pubsub_subscribe() to register DocSync subscriptions - Document how on_new_doc events are triggered by PR #691 - Remove conditional compilation and async dependencies - Show clear 4-step workflow: add → pin → validate → publish * docs(doc-sync): document PR #691 integration requirements Add comprehensive comments explaining PR #691 requirement for subscription event routing. Document the complete pub/sub flow, what works now vs what needs PR #691, and how to integrate when it merges * fix(doc-sync): import GuestSyncChannel trait and clarify PR #691 comment Fix compilation error and clarify that publishing to PubSub works now; PR #691 is only needed to route incoming messages to event handlers. * update docs * refactor(doc-sync): Focus module on publishing workflow only Remove subscription logic and simplify documentation. Module now demonstrates only the 4-step publishing workflow: file_add, file_pin, pre-publish validation, and pubsub_publish. * fmt * refactor(doc-sync): Move post logic to host Execute the 4-step publishing workflow (file_add, file_pin, pre-publish, pubsub_publish) on the host side instead of in the WASM module. Reduces boundary crossings from 6 to 2 for better performance. * fmt * fmt * fix(doc-sync): Fix compilation errors - Fix SyncChannel resource import and usage - Update channel::post() to call host implementation correctly * refactor(doc-sync): Replace eprintln with tracing Use tracing macros (info/warn/error) instead of eprintln for logging in doc-sync host implementation. * refactor(doc-sync): Replace eprintln with tracing Use tracing macros (info/warn/error) instead of eprintln for logging in doc-sync host implementation. * fmt * refactor(doc-sync): Extract constants and improve error logging - Add DOC_SYNC_TOPIC and DOC_SYNC_CHANNEL constants - Add error logging to id_for() method * refactor(doc-sync): Improve error logging and remove redundant conversions - Log actual error details instead of discarding with - Remove redundant .to_string() on CID response - Add error logging for failed post operations * fmt * fmt * refactor(doc-sync): Replace stub implementations with todo!() Replace placeholder implementations with explicit todo!() markers: - id_for: Remove incorrect file_add call with side effects - sync-channel::new: Remove placeholder resource ID (42) - sync-channel::close: Remove stub return value These are out of scope for post() implementation and should panic if called. * Update hermes/bin/src/ipfs/task.rs Co-authored-by: Rafał Chabowski <rafal.chabowski@iohk.io> * docs(doc-sync): Move crate docs to README and remove duplication Move module documentation from lib.rs to a crate-level README.md for better visibility and remove duplicated content between crate-level and channel module docs. * docs(doc-sync): Move crate docs to README and remove duplication Move module documentation from lib.rs to a crate-level README.md for better visibility and remove duplicated content between crate-level and channel module docs. * fix(doc-sync): Fix markdown linting errors in README Split long sentences to one per line and ensure lines stay under 132 character limit. * fix(doc-sync): Replace todo!() with placeholder implementations Replace 5 todo!() macros in doc_sync host with minimal placeholder implementations to pass clippy lint checks. Functions now return empty/default values instead of panicking. * fix(doc-sync): Resolve all clippy pedantic lints Fixes documentation formatting, format string inlining, unnecessary value passing, wildcard imports, and missing error documentation to satisfy clippy::pedantic requirements * Log pin result * IPFS file add operation now also returns CID * Fixes after the merge * Remove the pin step * Remove the pin step * Cleanup * Cleanup * DHT provide on the host side * Wire the `IpfsCommand::DhtProvide` command * Add `is_pre_publish_completed()` * Implement basic timeout * Implement DHT get providers * Use proper peer ID * Update test component * Temp switch to `main` branch in libs * Cleanup * Split `fn post()` into auxiliary functions * Update `hermes-ipfs` * Update comment * Fix typo * Fix typo * Revert unnecessary comment change * Use correct error in `dht_provide()` * Use proper error in `IpfsCommand::DhtGetProviders()` --------- Co-authored-by: cong-or <conchubhar.gannon@gmail.com>

hermes/bin/tests/integration/tests/serial/ipfs_subscribe.rs

aido-mth · 2025-12-11T09:07:52Z

hermes/bin/src/ipfs/mod.rs

+    doc_sync_topic_subscriptions: DashMap<PubsubTopic, HashSet<ApplicationName>>,
    /// Collection of stream join handles per topic subscription.
    subscriptions_streams: DashMap<PubsubTopic, JoinHandle<()>>,
+    /// Collection of stream join handles per topic subscription (Doc Sync).
+    doc_sync_subscriptions_streams: DashMap<PubsubTopic, JoinHandle<()>>,


Suggested change

doc_sync_topic_subscriptions: DashMap<PubsubTopic, HashSet<ApplicationName>>,

/// Collection of stream join handles per topic subscription.

subscriptions_streams: DashMap<PubsubTopic, JoinHandle<()>>,

/// Collection of stream join handles per topic subscription (Doc Sync).

doc_sync_subscriptions_streams: DashMap<PubsubTopic, JoinHandle<()>>,

doc_sync_topic_subscriptions: DashMap<PubsubTopic, StructWithHashSetAndHandler>,

/// Collection of stream join handles per topic subscription.

subscriptions_streams: DashMap<PubsubTopic, JoinHandle<()>>,

Does such change make sense? Or there is a reason to have multiple dash maps?

For a single topic we can have both doc sync subscription and regular ipfs subscription.
The kind would have to go as a key, but it's less of an intrusive change with multiple maps, I think.

@no30bit I'm talking about doc_sync_topic_subscriptions and doc_sync_subscriptions_streams which are both for doc sync

There can be 2 tasks – one sending DocSync events, and the other sending regular IPFS events

no30bit added 2 commits November 25, 2025 13:11

init

7913af9

draft

917dd07

minikin assigned no30bit Nov 25, 2025

no30bit added this to Catalyst Nov 25, 2025

github-project-automation bot moved this to New in Catalyst Nov 25, 2025

no30bit moved this from New to 🏗 In progress in Catalyst Nov 25, 2025

cong-or added a commit that referenced this pull request Nov 25, 2025

fix(doc-sync): import GuestSyncChannel trait and clarify PR #691 comment

5ba20ba

Fix compilation error and clarify that publishing to PubSub works now; PR #691 is only needed to route incoming messages to event handlers.

pin

3be91e3

no30bit marked this pull request as ready for review November 27, 2025 13:23

no30bit requested review from aido-mth, bkioshn, cong-or, rafal-ch and stevenj as code owners November 27, 2025 13:23

Merge remote-tracking branch 'origin/main' into feat/629/send-on-new-doc

434b9c8

bkioshn reviewed Dec 1, 2025

View reviewed changes

hermes/bin/src/runtime_extensions/hermes/doc_sync/mod.rs Outdated Show resolved Hide resolved

hermes/bin/src/runtime_extensions/hermes/doc_sync/host.rs Show resolved Hide resolved

hermes/bin/src/runtime_extensions/hermes/doc_sync/host.rs Outdated Show resolved Hide resolved

no30bit added 6 commits December 1, 2025 09:55

Merge remote-tracking branch 'origin/main' into feat/629/send-on-new-doc

2dd2492

update name mappings

f4f27fc

fix test build

0eabeff

fix fmt

b0b4f13

use .new channel instead of map functions

60d50b1

Merge remote-tracking branch 'origin/main' into feat/629/send-on-new-doc

713e374

Merge remote-tracking branch 'origin/main' into feat/629/send-on-new-doc

f757880

This was referenced Dec 2, 2025

feat(hermes): .new and .diff decoding and encoding #715

Closed

feat(rust/hermes-ipfs): Doc Sync .new and .diff payloads input-output-hk/catalyst-libs#681

Merged

Merge remote-tracking branch 'origin' into feat/629/send-on-new-doc

d8cdc34

no30bit added 3 commits December 9, 2025 17:14

Accept .new payload

cd0f7fe

fix integration tests

03be725

fix doc sync test

34e940f

aido-mth reviewed Dec 11, 2025

View reviewed changes

no30bit added 3 commits December 11, 2025 14:11

add delay to playground

86bba7f

wip

4fd6489

Remove new ipfs test

0312d50

no30bit requested a review from aido-mth December 15, 2025 14:05

remove dbg comments

fafe0be

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(hermes): send OnNewDoc event #691

feat(hermes): send OnNewDoc event #691

Uh oh!

no30bit commented Nov 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aido-mth Dec 11, 2025

Uh oh!

no30bit Dec 11, 2025

Uh oh!

aido-mth Dec 11, 2025

Uh oh!

no30bit Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat(hermes): send OnNewDoc event #691

Are you sure you want to change the base?

feat(hermes): send OnNewDoc event #691

Uh oh!

Conversation

no30bit commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Description of Changes

Breaking Changes

Screenshots

Related Pull Requests

Please confirm the following checks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aido-mth Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

no30bit Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

aido-mth Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

no30bit Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

no30bit commented Nov 25, 2025 •

edited

Loading