mod_import_feed

Discover RSS 2.0, RSS 1.0/RDF, Atom, JSON Feed 1/1.1 and iCalendar feeds and automatically import articles and media.

Administrator accounts cannot run imports. Select an enabled non-administrator account with only the required import permissions. This is checked when saving settings and before polling, including for existing feeds whose account is later promoted to admin.

Setup and administration

Enable Import feeds in Modules and open Content > Import feeds. The module requires mod_admin and mod_cron. Enter a website or feed URL, select one of the discovered feeds, and review three example entries beside the import settings. Discovery uses HTML alternate links and HTTP Link headers.

Managing feeds requires use permission for mod_import_feed. New feeds default to the current editor. Administrators and users with use permission for mod_admin_identity can select another enabled account with Select user. Saving records that account and its user groups for background imports; ordinary settings edits preserve the selected account and saved groups. The worker checks that the editor is still enabled and uses the editor's current, bounded permissions. Resource, media and edge operations enforce their normal ACLs; imported HTML is sanitized.

Per-feed settings

Settings are stored in the feed table. There are no m_config options.

Imports create published resources and media placeholder resources. Inline images become z-media embeds referring to these stable resource IDs. After saving the article and its connections, the importer queues persistent z_pivot_rsc tasks for the downloads. Attached media use the article itself as their placeholder. The tasks use m_media to identify each downloaded file and select its media category; downloads do not hold a database transaction.

Each task rechecks the feed's current import account and fetch credentials, resource permissions, pending URL and existing medium. Deleted resources and locally replaced media are not overwritten. Download failures retry up to sixteen times after the initial attempt. Delays double from one minute up to a fixed twelve-hour cap: 1, 2, 4, 8, 16, 32, 64, 128, 256 and 512 minutes, followed by six twelve-hour waits (89 hours 3 minutes total, excluding request execution and scheduling delays). A longer Retry-After takes precedence. These limits are fixed, not module configuration. Permission failures stop immediately. The feed activity log shows queued, started, retry, success and error events. A terminal failure leaves an empty placeholder so the article remains imported and a later reimport can retry. Inline images can use data: URLs, including base64 and percent-encoded data. They follow the same deferred media import path and become z-media embeds. A SHA-256 identity of the source URL keeps payloads out of resource URI indexes and activity logs, and lets repeated imports reuse the same resource. Embedded media have the same 20 MiB size limit as downloaded media.

Native Zotonic resource feeds already use the core importer's persistent media tasks.

For each inline image, the importer selects one variant, preferring the largest srcset width or density, with src as fallback. WordPress featured-image variants reuse the original image. In attached mode, other inline images retain their source URLs.

Fetch authentication and development

Discovery can use the current user's registered OAuth2 consumer credentials. Saved feeds use their selected import user's credentials for previews and polling. Tokens are resolved per destination host and port, only over HTTPS, and are never stored in feed settings. Missing credentials leave a request anonymous. Native imports save their fetch preference for core reimports and medium downloads.

Development sites allow local destinations and self-signed TLS certificates. Transport, redirects and certificate checks use the shared z_fetch routines; browser input cannot change the server's environment. OAuth2 credentials are never sent over HTTP. Anonymous fetching remains anonymous across redirects.

Polling and identity

Polling stores the source's ETag and Last-Modified after a successful import and sends both validators on subsequent requests. HTTP 304 is a successful unchanged poll: it does not parse a body, import entries, or remove existing content, and preserves the publisher's polling interval. Validator updates are protected by the poll lease and settings version. Failed imports, settings changes, manual reimport and restoring deleted resources require a fresh response. Previews and individual-resource fetches remain unconditional.

Atom and RSS rel=next links and JSON Feed next_url are followed. The initial import reads only the first page unless Import all pages on the initial import is enabled (import_all_pages, default false). This option can import large amounts of articles and media.

Later polls process each complete page and stop following links when that page contains a previously successfully imported entry. Deleted tracked entries also count as seen. This fetches new published entries, but is not a history or change synchronization mechanism: older edits and backdated articles may be missed.

Continuation requests use the same URL, credential and response-size checks, without first-page HTTP validators. Failed pages resume from their saved URL. Each worker processes at most 100 pages, then schedules a continuation. Loops are rejected within a traversal batch. Settings saves restart from the first page. Validators are saved only for completed single-page polls; pending or failed traversals cannot be hidden by a first-page HTTP 304.

Polling uses poll_minutes as a minimum interval (default 60 minutes; range 5–10080). RSS ttl and namespace-qualified syndication updatePeriod / updateFrequency hints can lengthen it. When both are present, the longer hint wins. Missing syndication frequency defaults to 1; a frequency without a period uses daily. Monthly/yearly periods conservatively use 31/366 days, and division rounds up to whole seconds. Invalid hints are ignored. Schedule alignment via updateBase, RSS skipHours/skipDays, and HTTP cache freshness are not used.

Consecutive feed-fetch or parse failures double the configured interval on each failure, with a fixed 12-hour backoff cap. Neither that cap nor backoff can shorten poll_minutes or the last valid publisher interval. A successful fetch and parse resets the failure counter and replaces the stored publisher hint, restoring the normal frequency even if individual entries fail to import. Counters, publisher hints, and next-poll times survive restarts. There is no automatic give-up deadline.

HTTP 429/503 responses honor Retry-After as seconds or an HTTP date. The next poll waits for the longest of the normal interval, failure backoff, and that delay. Past dates mean no additional delay; malformed, duplicate or out-of-range values are ignored. Parsed delays must fit 0–2147483647 seconds. Entry-fetch errors can also extend the next poll; media download retries honor the same header while retaining their sixteen-retry limit. Headers and response bodies are not saved in error messages. Explicit Poll now remains an immediate override. No new module configuration is required; poll_minutes remains the per-feed control.

The minute tick starts up to five background workers per site and Erlang node. Each worker polls up to five due feeds in sequence. Database leases prevent two workers from polling the same feed concurrently. The overview shows article counts, cumulative import/update/skip/error statistics and poll times. Each feed has settings, pause/resume, test/preview, poll-now, reimport and paginated history for articles and dependent media.

Stable source URIs and checksums allow repeated polls without duplicates. Reimport processes only entries currently present in the feed, preserving resource IDs. Deleted resources remain deleted. Media already imported from the same URL is reused. Feeds with more than 100 entries are rejected, rather than silently importing a partial feed. WebSub hub links are recognized in feed data; feed fetching uses polling.

Save & refetch deleted saves the settings and requests one fetch that can restore deleted articles and their media when present in the current feed. It also fetches a paused feed once, without resuming its regular polling. The request is consumed when a worker claims the feed, including if fetching fails; subsequent polls preserve deleted resources as usual. Missing entries are never restored.

Imports lock their source URIs across connected Erlang nodes, so independent feeds can progress concurrently while shared articles and media are serialized. Network requests and the complete article import do not hold a database transaction; resource and media APIs manage their own write transactions. The rsc_update observer writes ordinary feed resources' source mappings and feed membership in that same transaction, including authoritative copies and media placeholders. Completion checksums are recorded only after the entry finishes. A failed import can leave successfully imported resources in place for reuse on retry.

Deleting imported content checks each resource's delete permission and protection. Dependent cleanup follows the normal resource deletion rules; a dependent does not prevent deletion of its parent.

iCalendar feeds

Discover text/calendar links or enter an .ics feed URL directly. VEVENTs must have a UID, UTC DTSTAMP and valid DTSTART. Invalid events are skipped; malformed calendar structure is rejected. Dates, durations and exclusive all-day end dates are normalized to Zotonic event properties. UTC and known IANA TZIDs are supported; floating times are interpreted as UTC. Custom VTIMEZONE rules are not evaluated, and events with unknown TZIDs are skipped. Cancelled events are unpublished. With category guessing enabled, calendar entries use event; otherwise the configured category applies.

Recurrence rules and recurrence IDs are retained as ical_rrule and ical_recurrence_id; occurrences are not expanded. Exceptions have separate identities derived from UID and RECURRENCE-ID. Alarms are ignored.

Zotonic calendar exports advertise X-ZOTONIC-RESOURCE, using the same native resource import and optional per-resource WebSub subscriptions as Atom/RSS/JSON Feed. Generic calendar feeds continue to use polling; feed-level WebSub push subscriptions are not provided. X-ZOTONIC-NEXT is followed using the existing first-page/initial-import-all-pages and stop-at-seen-entry policies.

Zotonic resource feeds

Atom and RSS entries from mod_export advertise a JSON resource link with relation https://zotonic.com/rel/resource. JSON Feed items advertise the same endpoint in _zotonic.resource_url. These use m_rsc_import, preserving native properties, categories, media embeds and connections. Connection import depth defaults to 1 and accepts 0–10. Content-group/copyright defaults and cleanup patterns apply; category guessing and media-layout choices do not reshape native resources. Choosing no media disables native medium downloads.

With mod_websub enabled, optional subscriptions cover imported articles and, when selected, their connections within the import depth. Both options default to false and require non-authoritative imports and a supporting source. Existing subscriptions are managed separately through WebSub administration. Native resources use the core Fetch new version flow and saved import options, without requiring the entry to remain in the feed. These refreshes use the core import notification with native resource source metadata.

Fetch new version

For ordinary feed entries, the rsc_import_fetch observer refreshes known imported articles from their current feed, using the saved settings and import account. The caller needs feed-management and article-edit permission. Paused feeds can be refreshed manually without resuming polling. This bypasses unchanged checksums and still runs the import notification. An article no longer present in the feed returns feed_entry_not_found; unknown URIs fall through to the standard resource importer.

Import notification

Before writing an entry or its media, the importer dispatches the core #import_resource{item, related, source, options, raw} notification. The source is #{importer => mod_import_feed, feed => Settings}. Include zotonic_core/include/zotonic.hrl, declare zotonic_observer, and implement observe_import_resource/2 or its optional PID-aware variant. See the core observer documentation for the payload, permissions and result contract.

Unchanged and deleted entries are skipped before notifying. Native feed entries notify once; their delegation to m_rsc_import preserves this scope, while recursively imported resources may generate separate notifications.