mod_websub

WebSub resource synchronization, following https://www.w3.org/TR/websub/.

Integration points

Enable mod_websub on the publisher and importing site. Anyone who can view an original resource can subscribe to it, including anonymous visitors for public resources. Private subscriptions use explicit HTTP authorization and current resource access. The use mod_websub permission controls administration.

• controller_websub handles hub requests, callback verification, and deliveries.
• controller_websub_topic serves the complete, fixed JSON topic representation.
• The admin Content menu links to a subscription overview, filtered by direction and local resource ID. It requires use mod_websub; edit pages show the active incoming subscriber count and a filtered overview link.
• m_websub provides the ACL-checked status/start/stop API and delivery/import queues.
• z_websub_subscription persists subscriber intent, leases, and renewal work.
• z_websub_discovery and z_websub_http handle discovery and outbound fetch policy.

The resource_headers and rsc_export_done observers advertise discovery links. Full resource exports also include websub.hub and websub.topic for clients subscribing from exported JSON. This optional JSON extension is present only on authoritative resources and agrees with the standard discovery links. The semantic resource identifier remains uri. rsc_import_fetch_result exposes subscription availability to the import dialog; rsc_import_done starts an explicitly requested subscription. rsc_update_done queues publication. Second ticks consult the module server's dirty flag and next due time; ten-minute ticks reconcile with the database. Enqueues signal after commit, and the daily tick performs maintenance. The protocol adapters are documented separately from the resource importer: external peers need no Zotonic-specific WebSub fields.

Resource identity versus the WebSub topic

A resource's identity is its language-less m_rsc:uri/2, including /id/name. Imports store this in source_uri and the resource's uri. It is never replaced by a page, representation, or hub URL. Discovery advertises rel=self pointing to the fixed JSON websub_topic dispatch, and rel=hub to the local hub. The WebSub topic is stored separately in topic_url. This distinction lets /id retain semantic-web content negotiation while a WebSub topic has one media type.

Flow between two Zotonic systems

1. On importing a connected resource from A into B, B fetches A's /id URI with Accept: application/json. The JSON export retains A's semantic resource URI. HTTP Link headers (or the JSON links extension) advertise A's JSON topic and hub.
2. If the editor opts into automatic updates, B persists the source URI, local resource, editor, and desired subscription state. Recursive imports do not opt in.
3. B rediscovers the source and POSTs a standard URL-encoded subscription request to the advertised hub. hub.topic is the discovered self URL, not necessarily /id. B supplies a unique callback, random secret, and requested lease.
4. A bounds and persists verification work and replies 202. Independently, A checks the subscribing user's resource access and GETs B's callback with the topic, mode, random challenge, and lease. B verifies pending intent and echoes the challenge as plain text with nosniff. A records the verified subscription.
5. Changes on A queue the resource's newest version. A rechecks access and lease, then POSTs the complete JSON topic representation, Link hub/self headers, and an HMAC-SHA256 signature to B. Private topics also send full authorized JSON; there is no notification-only or Zotonic-specific wire format.
6. B verifies the signature and callback, queues work, and quickly acknowledges. Public imports consume the signed export; credentialed imports refetch A's semantic URI using the editor's source credentials. Inside the import transaction B checks that the stored source, current resource URI, and payload URI agree, then applies newer content with the editor's current permissions and saved options.
7. B renews before lease expiry, rediscovering the hub/topic and rotating its callback and secret. The old callback remains active until the new one verifies. Stopping immediately disables imports and queues verified remote unsubscription. Confirmation also queues a catch-up fetch to recover missed updates.

Collections and connected resources

Edge insertions, removals, and reordering advance the authoritative subject's version and publish its updated export. Receivers synchronize edges using saved import depth and filters, then fetch newly referenced placeholders after commit. Existing imported members are reused, not refreshed by the collection's delivery.

The optional is_subscribe_connections import setting subscribes imported connected resources across all predicates, up to the saved connection depth. It defaults to false. Connected imports inherit the option with decremented depth, so later additions follow the same bound. Subscriptions remain independent: removing a connection or stopping the parent does not unsubscribe them. See the module README for the complete collection lifetime and retry behavior.

Interoperation with other WebSub implementations

No peer-brand detection or private protocol extension is required. Any subscriber can discover the fixed JSON topic from headers or HTML, subscribe using standard form fields, answer verification, and receive its full application/json body. The body's result.uri is the semantic identity; the delivery Link self names the WebSub topic. Private subscriptions require explicit HTTP authorization and current access at A's hub (cookies alone are insufficient); granting one authorizes full content delivery to the verified callback.

A non-Zotonic hub can relay the same JSON topic to B without changes. A non-Zotonic publisher can be imported if it supplies the resource-export JSON understood by Zotonic's resource importer. WebSub is media-type agnostic, but this application adapter imports resources, not arbitrary Atom/RSS/HTML documents. It does not claim that those document formats can be imported as Zotonic resources.

Standard subscription verification, signatures, leases, 307/308 hub redirects, and unsubscription apply to all peers. A 202 alone never activates a subscription. Delivery retry exhaustion drops only that notification, preserving the lease for future updates. Semantic resource identity is independent of WebSub topic migration.

Security and operation

All WebSub requests use z_fetch with automatic redirects disabled. Destinations are checked for non-public addresses on each hop; private, loopback, link-local, and reserved networks are rejected. Development sites may use .test peers on loopback/private networks with self-signed TLS. Other destinations keep verified TLS and public-address checks. HTTPS downgrades are refused. Cross-origin GET redirects permanently drop user credentials for that chain. Callback requests are anonymous. z_fetch integrates mod_oauth2 consumer tokens using the original HTTPS URL, hostname, and subscribing user; the source's same-origin hub can use that token as well. Token lookup and renewal remain owned by mod_oauth2.

TODO: add validated-address pinning to z_fetch/z_url_fetch, preserving the original Host header and TLS hostname. The current DNS preflight does not prevent DNS rebinding between validation and connection establishment.

Hub admission deduplicates identical requests and limits each site to 120 new verification tasks per minute and 1000 queued tasks. Admission is serialized in the database across nodes; overload returns 503 before accepting work. Token group restrictions are retained when recreating delivery ACL contexts. Admin start, stop, and status require edit permission. Callback tokens and secrets are not exposed by status. Imports serialize identity and permission checks with the update.

The unique per-site sidejob processes renewal, delivery, and import queues. Schema version 3 separates source identity from topic and adds durable admission accounting. See README.md for deployment details and the regression suite.