mod_filestore

Store uploaded media and generated previews on S3-compatible storage, FTP/FTPS, or WebDAV. Zotonic keeps a database registry of remote files and serves them through a shared local cache when they are no longer on the site's disk.

Set up remote storage

1. Enable mod_filestore for the site.
2. Open System → Cloud File Store in the admin. Changing settings and starting bulk moves require the use mod_admin_config permission.
3. Enter the base URL and credentials for your storage service using the table below. For S3, enter the provider's region to enable Signature V4, or leave it empty to use legacy Signature V2.
4. Select Upload new media files to the cloud file store. Decide whether to Keep local files after upload and choose the remote deletion delay.
5. Save the settings. Zotonic writes, reads back, and deletes a temporary -zotonic-filestore-test-file- beneath the base URL. Settings are saved only when this test succeeds.
6. Upload a test image, allow the upload queue to run, and check both the original and a resized preview. Use the bulk action to upload existing media when ready.

A successful credential test checks these operations at the configured location; check normal media delivery as well. The test needs delete permission even when remote deletion is set to Never.

Supported services and URLs

The admin derives the service from the URL scheme. http: and https: select S3, not WebDAV. For WebDAV use webdav: or webdavs:; dav: and davs: are aliases. Use webdavs: or davs: to protect the WebDAV credentials with HTTPS.

Both ftp: and ftps: select the FTP backend. The server must support TLS: the client uses passive FTP with explicit TLS by default, or implicit TLS when port 990 is specified, for example ftps://files.example.com:990/mysite. Allow the server's passive data connections through the firewall. SFTP (SSH file transfer) is not supported by this backend.

FTP and WebDAV create missing directories during upload. Their accounts need permission to create directories and write, read, and delete files in the chosen location. For S3, the admin can try to create a private bucket if it is missing; this requires permission to create a bucket. The checkbox is a setup action, not a saved configuration option.

S3 signature version and region

The S3 region field (s3region) selects the signature version automatically:

Leading and trailing whitespace is removed. Use the region supplied by your storage provider; Zotonic does not infer it from the endpoint URL. There is no separate signature-version setting and no automatic fallback to V2 if a V4 request fails. Providers requiring V4 need a nonempty, correct region.

This selection applies to settings tests and all S3 file operations, including uploads, downloads, streaming, deletion, and bucket-creation requests. FTP and WebDAV ignore the region.

An unset site region inherits the system-wide s3region default. An explicitly empty site region overrides that default and selects V2. When configuration is locked, the system-wide region determines the signature version. Existing installations without a region continue to use V2.

S3 permissions

Use a bucket or prefix dedicated to the site. Grant bucket listing where required by the provider and grant the following object permissions beneath the base URL:

Other applications using the filestore may write additional paths. Give those paths the corresponding permissions. Files do not need public-read access: Zotonic retrieves them using the configured credentials.

Configuration reference

Site settings are stored under mod_filestore in m_config. The historical s3 prefixes also apply to FTP and WebDAV, except s3region, which only applies to S3. An explicitly empty site region selects V2 instead of inheriting a global region. An unset site region inherits the global setting; locked configuration always uses the global setting.

System-wide defaults and locked settings

The same keys can be set in the zotonic_mod_filestore application environment in the system configuration. Nonempty site settings override these defaults unless is_config_locked is true. For s3region, an explicitly empty site setting also overrides the default, selecting Signature V2. This lock is a system-wide option: it makes sites use the application settings and prevents editing them through the filestore form.

is_read_only is an exception to this override rule: setting it to true in the global zotonic_mod_filestore configuration makes storage read-only for every site, including sites with their own endpoints or credentials. A site cannot turn it off with false. This restriction works independently of is_config_locked; locking configuration is not required to enforce read-only storage. A site can also enable read-only mode when the global value is false.

For example, merge this application entry into the system's Erlang configuration list, using your own endpoint and credentials:

{zotonic_mod_filestore, [
    {service, <<"webdav">>},
    {s3url, <<"webdavs://files.example.com/zotonic/{{site}}">>},
    {s3key, <<"storage-user">>},
    {s3secret, <<"replace-with-password">>},
    {is_read_only, false},
    {is_upload_enabled, true},
    {is_local_keep, true},
    {delete_interval, <<"false">>},
    {is_config_locked, true}
]}

The example permits uploads. Set {is_read_only, true} in the same application entry to make all sites download-only. It takes precedence over {is_upload_enabled, true} and any delete_interval: uploads and remote deletes are blocked without changing those saved settings. This is useful when a staging or recovery installation needs to read existing storage without modifying it.

In a system-configured base URL, {{site}} is replaced with the site name. If the placeholder is absent, Zotonic appends the site name as a directory. A base URL saved in the site's admin is used as entered, without this expansion. Configure service alongside the URL: URL-based service detection happens when saving the admin form, not when reading application configuration.

Read-only storage

To allow downloads while preventing remote writes for all sites, merge this entry into the Zotonic global Erlang configuration list (or add the key to its existing zotonic_mod_filestore entry):

{zotonic_mod_filestore, [
    {is_read_only, true}
]}

For one site, add this to its site configuration:

{mod_filestore, [
    {is_read_only, true}
]}

These examples show the entry to merge into the existing configuration list. The setting can also be stored as mod_filestore.is_read_only in m_config. Restrictions are additive: a global or site-file true remains effective even if another source says false, or the service configuration is locked. The default is false. To restore writes, remove or disable every active read-only restriction; changing only a site setting cannot lift a global restriction.

Read-only mode blocks background uploads, remote deletion, direct filestore upload/delete requests and the admin's write-based credential test. Direct writes return {error, read_only}. Existing upload and deletion queues are retained and can resume when read-only mode is disabled. Local registry deletion marks can still be recorded. Already dispatched remote operations cannot be revoked by changing configuration; enable read-only mode before starting the site when that matters.

Normal downloads, cached reads and copying remote files to local disk remain available. The admin shows the effective mode and disables upload/test buttons. Settings can be saved without a write test; verify credentials by downloading an existing file. Read-only accounts need only the provider's read permissions for those objects. No test object or bucket is created in this mode.

Uploads, local copies, and the cache

New media files and previews are queued for asynchronous upload. Queue entries become eligible after one minute. Processing runs on a minute tick, with bounded batches, database-load checks, and backoff, so completion can take longer.

With is_local_keep enabled, remote storage holds an additional copy of local media. Otherwise, successfully uploaded files move from the site's files directory into filezcache, where they can be evicted. When a remote-only file is requested, Zotonic downloads it into the cache and can serve it while the download proceeds.

The filezcache application is shared by all sites. Its max_bytes application setting controls cache capacity (default 10 GiB). Cache files are disposable; retain the remote files and database registry. Keeping remote media copies does not replace a backup of the site's database, configuration, and application code.

Read failures and retries

A failed remote cache read remains in the file registry and is retried when the file is requested again. This includes forbidden and enoent: a permission or account problem can be temporary, and S3 can return forbidden for missing files.

Each file has its own exponential retry delay: nominally 5, 10, 20, 40 seconds, and so on, capped at ten minutes. Jitter varies each delay between 80% and 100% of that value. During the cooldown, requests do not start another remote read; an existing local or cached copy remains usable. A successful remote read clears the error and resets the delay. There is no automatic permanent cutoff, and files that are not requested generate no read retries.

Retry eligibility is not a scheduled download time. Zotonic's file-serving negative cache can add up to 60 seconds before another lookup occurs. Retry state survives restarts. An expiring database claim limits concurrent attempts across nodes; abandoned claims expire after ten minutes.

This policy applies to remote reads into filezcache. Background queue processing has separate batch-size backoff. Direct #filestore_request{} calls do not use this persistent read-retry policy.

Deletion and moving existing files

Zotonic normally retains deleted media for five weeks to allow recovery. The filestore's delete_interval adds a delay after a file is marked for remote deletion. 0 means no extra delay, not deletion at the instant a page is removed. false keeps remote files indefinitely; it does not make the remote service itself immutable.

Use the admin's bulk actions to queue existing local media for upload or to move remote files back to the server's disk. These operations run in the background; watch the queues and logs. Moving files to disk requires enough disk space and working credentials for their existing locations. Disable uploads when bringing files back for a storage migration.

Before switching to a different service or account, bring the files back locally and verify that the download queue has completed. Then configure and test the new location and queue uploads. Changing the base URL alone does not copy existing remote objects: the registry retains their original service and location. The default reverse lookup uses credentials for the currently configured service. Remote deletion also checks that an object's URL matches the configured base URL.

Statistics and troubleshooting

The admin shows registered media, estimated local file counts and sizes, remote file counts and sizes, and upload, download, and delete queue counts. These figures come from the site's database; they are not a scan or integrity check of remote storage. Local totals are estimates, not a scan of the files directory.

If uploads do not progress, check that uploads are enabled, the module is active, and the URL and credentials pass the settings test. Inspect logs for connection, permission, TLS, or storage errors. For FTP, check passive data connections; for WebDAV, check the full collection path and the webdavs: scheme. Allow for the queue delay and backoff before assuming a newly queued file is stuck.

Health warnings and error details

The admin dashboard and filestore configuration page warn administrators when:

• At least five remote cache reads failed and failures account for at least 20% of completed reads in the current and preceding fourteen minute buckets.
• At least one active, unresolved read error is older than 24 hours.

The read-error rate is shared across nodes. It counts completed remote attempts, not cache hits or requests suppressed by the cooldown. The 24-hour age runs from the first failure in the current unresolved sequence; repeated failures do not restart that clock.

Follow View errors to the filestore page for details. The list shows 50 files per page, oldest first, with Previous/Next navigation. Each row shows the relative file path, storage service, latest recorded error, consecutive failure count, failing-since time, and retry eligibility. Only errors older than 24 hours are listed, including entries explicitly marked unrecoverable. Remote URLs and credentials are not included in the list.

Marking a file unrecoverable

Only the administrator account, user 1, sees the Mark unrecoverable action. After confirmation, the server rechecks both the user ID and that the entry still has an unresolved error older than 24 hours. Other administrators cannot perform this action, and a recovered or recently failed entry is not eligible.

Marking an entry unrecoverable stops automatic remote cache-read retries and removes it from active unresolved-error counts. Its row remains in the error list with an Unrecoverable label. The file registry record and error details are retained; no local or remote files are deleted. Historical read-rate statistics are unchanged, so a high-error-rate warning can remain until those attempts leave the reporting window. Late callbacks from an attempt already in flight cannot clear the unrecoverable state. A new successful upload of the path resets it.

Integration points

mod_filestore handles these notifications:

• #media_update_done{} queues inserted or updated media files.
• #filestore{} handles file lookup, upload, and deletion through the file registry and cache.
• #filestore_request{} handles direct storage upload, download, and deletion requests.
• #filestore_credentials_lookup{} maps a local path and optional resource ID to remote credentials and location.
• #filestore_credentials_revlookup{} resolves credentials for an existing remote service and location.
• #admin_menu{} adds the admin menu entry.

Applications can provide credential lookup observers to route files to different services. See zotonic_file.hrl for the notification records and model#filestore for the configuration and statistics model. Storage requests use s3filez, ftpfilez, or webdavfilez; filezcache manages the shared download cache.