# mod\_filestore

Source: https://zotonic.com/docs/1329/mod_filestore

URI: https://zotonic.com/id/1329

Language: en



Store uploaded media and generated previews on S3-compatible storage, FTP/FTPS, or WebDAV. Zotonic keeps a database registry of remote files and serves them through a shared local cache when they are no longer on the site's disk.

Set up remote storage
---------------------

1. Enable `mod_filestore` for the site.
2. Open **System → Cloud File Store** in the admin. Changing settings and starting bulk moves require the `use mod_admin_config` permission.
3. Enter the base URL and credentials for your storage service using the table below. For S3, enter the provider's region to enable Signature V4, or leave it empty to use legacy Signature V2.
4. Select **Upload new media files to the cloud file store**. Decide whether to **Keep local files after upload** and choose the remote deletion delay.
5. Save the settings. Zotonic writes, reads back, and deletes a temporary `-zotonic-filestore-test-file-` beneath the base URL. Settings are saved only when this test succeeds.
6. Upload a test image, allow the upload queue to run, and check both the original and a resized preview. Use the bulk action to upload existing media when ready.

A successful credential test checks these operations at the configured location; check normal media delivery as well. The test needs delete permission even when remote deletion is set to **Never**.

### Supported services and URLs

| Service           | Base URL example                                                | Credentials               | `service` value |
| ----------------- | --------------------------------------------------------------- | ------------------------- | --------------- |
| S3-compatible     | `https://mybucket.s3.amazonaws.com/mysite`                      | Access key and secret key | `s3`            |
| FTP over TLS      | `ftps://files.example.com/mysite`                               | Username and password     | `ftp`           |
| WebDAV over HTTPS | `webdavs://files.example.com/remote.php/dav/files/alice/mysite` | Username and password     | `webdav`        |

The admin derives the service from the URL scheme. `http:` and `https:` select S3, not WebDAV. For WebDAV use `webdav:` or `webdavs:`; `dav:` and `davs:` are aliases. Use `webdavs:` or `davs:` to protect the WebDAV credentials with HTTPS.

Both `ftp:` and `ftps:` select the FTP backend. The server must support TLS: the client uses passive FTP with explicit TLS by default, or implicit TLS when port 990 is specified, for example `ftps://files.example.com:990/mysite`. Allow the server's passive data connections through the firewall. SFTP (SSH file transfer) is not supported by this backend.

FTP and WebDAV create missing directories during upload. Their accounts need permission to create directories and write, read, and delete files in the chosen location. For S3, the admin can try to create a private bucket if it is missing; this requires permission to create a bucket. The checkbox is a setup action, not a saved configuration option.

### S3 signature version and region

The **S3 region** field (`s3region`) selects the signature version automatically:

| Effective region                        | Signature version                    |
| --------------------------------------- | ------------------------------------ |
| Empty, including whitespace-only values | Signature V2 (legacy)                |
| Nonempty, for example `eu-west-1`       | Signature V4, signed for that region |

Leading and trailing whitespace is removed. Use the region supplied by your storage provider; Zotonic does not infer it from the endpoint URL. There is no separate signature-version setting and no automatic fallback to V2 if a V4 request fails. Providers requiring V4 need a nonempty, correct region.

This selection applies to settings tests and all S3 file operations, including uploads, downloads, streaming, deletion, and bucket-creation requests. FTP and WebDAV ignore the region.

An unset site region inherits the system-wide `s3region` default. An explicitly empty site region overrides that default and selects V2. When configuration is locked, the system-wide region determines the signature version. Existing installations without a region continue to use V2.

### S3 permissions

Use a bucket or prefix dedicated to the site. Grant bucket listing where required by the provider and grant the following object permissions beneath the base URL:

| Path                            | Permissions                                                                      |
| ------------------------------- | -------------------------------------------------------------------------------- |
| `-zotonic-filestore-test-file-` | `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject`                                |
| `archive/*`                     | `s3:GetObject`, `s3:PutObject`; `s3:DeleteObject` when remote deletion is enabled |
| `preview/*`                     | `s3:GetObject`, `s3:PutObject`; `s3:DeleteObject` when remote deletion is enabled |

Other applications using the filestore may write additional paths. Give those paths the corresponding permissions. Files do not need public-read access: Zotonic retrieves them using the configured credentials.

Configuration reference
-----------------------

Site settings are stored under `mod_filestore` in `m_config`. The historical `s3` prefixes also apply to FTP and WebDAV, except `s3region`, which only applies to S3. An explicitly empty site region selects V2 instead of inheriting a global region. An unset site region inherits the global setting; locked configuration always uses the global setting.

| Key                 | Meaning                                                                          |
| ------------------- | -------------------------------------------------------------------------------- |
| `service`           | Backend identifier: `s3`, `ftp`, or `webdav`. Set it explicitly outside the admin; an empty value falls back to `s3`. TLS variants are URL schemes, not backend identifiers. |
| `s3url`             | Base URL, including the bucket or directory and optional site prefix.            |
| `s3region`          | S3 signing region, for example `eu-west-1`. A nonempty region enables Signature V4; empty uses legacy Signature V2. |
| `s3key`             | S3 access key, or FTP/WebDAV username.                                           |
| `s3secret`          | S3 secret key, or FTP/WebDAV password.                                           |
| `is_read_only`      | Download-only mode. Blocks uploads, remote deletion and write-based credential tests for every backend. Default `false`. A global or site-file `true` cannot be overridden by other settings. |
| `is_upload_enabled` | Allow background uploads. Set explicitly when configuring storage outside the admin. Disabling this does not disable reads, queued downloads, or remote deletion. |
| `is_local_keep`     | Keep local files after successful upload. When false, uploaded files move into the evictable cache. |
| `delete_interval`   | Extra delay before deleting files marked for remote deletion: `0` (the default, no extra delay), `false` (never), seconds, or a value such as `1 week`, `2 days`, or `3 months`. |
| `tls_options`       | Erlang list of TLS options passed to the selected storage client. Not an admin form field; an empty list uses the client's defaults. |

### System-wide defaults and locked settings

The same keys can be set in the `zotonic_mod_filestore` application environment in the system configuration. Nonempty site settings override these defaults unless `is_config_locked` is true. For `s3region`, an explicitly empty site setting also overrides the default, selecting Signature V2. This lock is a **system-wide** option: it makes sites use the application settings and prevents editing them through the filestore form.

`is_read_only` is an exception to this override rule: setting it to `true` in the **global `zotonic_mod_filestore` configuration makes storage read-only for every site**, including sites with their own endpoints or credentials. A site cannot turn it off with `false`. This restriction works independently of `is_config_locked`; locking configuration is not required to enforce read-only storage. A site can also enable read-only mode when the global value is false.

For example, merge this application entry into the system's Erlang configuration list, using your own endpoint and credentials:

```erlang
{zotonic_mod_filestore, [
    {service, <<"webdav">>},
    {s3url, <<"webdavs://files.example.com/zotonic/{{site}}">>},
    {s3key, <<"storage-user">>},
    {s3secret, <<"replace-with-password">>},
    {is_read_only, false},
    {is_upload_enabled, true},
    {is_local_keep, true},
    {delete_interval, <<"false">>},
    {is_config_locked, true}
]}
```

The example permits uploads. Set `{is_read_only, true}` in the same application entry to make all sites download-only. It takes precedence over `{is_upload_enabled, true}` and any `delete_interval`: uploads and remote deletes are blocked without changing those saved settings. This is useful when a staging or recovery installation needs to read existing storage without modifying it.

In a **system-configured** base URL, `{{site}}` is replaced with the site name. If the placeholder is absent, Zotonic appends the site name as a directory. A base URL saved in the site's admin is used as entered, without this expansion. Configure `service` alongside the URL: URL-based service detection happens when saving the admin form, not when reading application configuration.

### Read-only storage

To allow downloads while preventing remote writes for **all sites**, merge this entry into the Zotonic global Erlang configuration list (or add the key to its existing `zotonic_mod_filestore` entry):

```erlang
{zotonic_mod_filestore, [
    {is_read_only, true}
]}
```

For one site, add this to its site configuration:

```erlang
{mod_filestore, [
    {is_read_only, true}
]}
```

These examples show the entry to merge into the existing configuration list. The setting can also be stored as `mod_filestore.is_read_only` in `m_config`. Restrictions are additive: a global or site-file `true` remains effective even if another source says `false`, or the service configuration is locked. The default is `false`. To restore writes, remove or disable every active read-only restriction; changing only a site setting cannot lift a global restriction.

Read-only mode blocks background uploads, remote deletion, direct filestore upload/delete requests and the admin's write-based credential test. Direct writes return `{error, read_only}`. Existing upload and deletion queues are retained and can resume when read-only mode is disabled. Local registry deletion marks can still be recorded. Already dispatched remote operations cannot be revoked by changing configuration; enable read-only mode before starting the site when that matters.

Normal downloads, cached reads and copying remote files to local disk remain available. The admin shows the effective mode and disables upload/test buttons. Settings can be saved without a write test; verify credentials by downloading an existing file. Read-only accounts need only the provider's read permissions for those objects. No test object or bucket is created in this mode.

Uploads, local copies, and the cache
------------------------------------

New media files and previews are queued for asynchronous upload. Queue entries become eligible after one minute. Processing runs on a minute tick, with bounded batches, database-load checks, and backoff, so completion can take longer.

With `is_local_keep` enabled, remote storage holds an additional copy of local media. Otherwise, successfully uploaded files move from the site's files directory into `filezcache`, where they can be evicted. When a remote-only file is requested, Zotonic downloads it into the cache and can serve it while the download proceeds.

The `filezcache` application is shared by all sites. Its `max_bytes` application setting controls cache capacity (default 10 GiB). Cache files are disposable; retain the remote files and database registry. Keeping remote media copies does not replace a backup of the site's database, configuration, and application code.

Read failures and retries
-------------------------

A failed remote cache read remains in the file registry and is retried when the file is requested again. This includes `forbidden` and `enoent`: a permission or account problem can be temporary, and S3 can return `forbidden` for missing files.

Each file has its own exponential retry delay: nominally 5, 10, 20, 40 seconds, and so on, capped at ten minutes. Jitter varies each delay between 80% and 100% of that value. During the cooldown, requests do not start another remote read; an existing local or cached copy remains usable. A successful remote read clears the error and resets the delay. There is no automatic permanent cutoff, and files that are not requested generate no read retries.

Retry eligibility is not a scheduled download time. Zotonic's file-serving negative cache can add up to 60 seconds before another lookup occurs. Retry state survives restarts. An expiring database claim limits concurrent attempts across nodes; abandoned claims expire after ten minutes.

This policy applies to remote reads into `filezcache`. Background queue processing has separate batch-size backoff. Direct `#filestore_request{}` calls do not use this persistent read-retry policy.

Deletion and moving existing files
----------------------------------

Zotonic normally retains deleted media for five weeks to allow recovery. The filestore's `delete_interval` adds a delay after a file is marked for remote deletion. `0` means no **extra** delay, not deletion at the instant a page is removed. `false` keeps remote files indefinitely; it does not make the remote service itself immutable.

Use the admin's bulk actions to queue existing local media for upload or to move remote files back to the server's disk. These operations run in the background; watch the queues and logs. Each download request walks a fixed range of registry IDs once, including entries previously recorded as local so missing local files can be recovered. Keeping the remote copy does not cause completed entries to be queued again. Files registered after the walk starts require a new request. Moving files to disk requires enough disk space and working credentials for their existing locations. Disable uploads when bringing files back for a storage migration.

Before switching to a different service or account, bring the files back locally and verify that the download queue has completed. Then configure and test the new location and queue uploads. Changing the base URL alone does not copy existing remote objects: the registry retains their original service and location. The default reverse lookup uses credentials for the currently configured service. Remote deletion also checks that an object's URL matches the configured base URL.

Statistics and troubleshooting
------------------------------

The admin shows registered media, estimated local file counts and sizes, remote file counts and sizes, and upload, download, and delete queue counts. These figures come from the site's database; they are not a scan or integrity check of remote storage. Local totals are estimates, not a scan of the files directory.

If uploads do not progress, check that uploads are enabled, the module is active, and the URL and credentials pass the settings test. Inspect logs for connection, permission, TLS, or storage errors. For FTP, check passive data connections; for WebDAV, check the full collection path and the `webdavs:` scheme. Allow for the queue delay and backoff before assuming a newly queued file is stuck.

### Health warnings and error details

The admin dashboard and filestore configuration page warn administrators when:

- At least five remote cache reads failed and failures account for at least 20% of completed reads in the current and preceding fourteen minute buckets.
- At least one active, unresolved read error is older than 24 hours.

The read-error rate is shared across nodes. It counts completed remote attempts, not cache hits or requests suppressed by the cooldown. The 24-hour age runs from the first failure in the current unresolved sequence; repeated failures do not restart that clock.

Follow **View errors** to the filestore page for details. The list shows **50 files per page**, oldest first, with Previous/Next navigation. Each row shows the relative file path, storage service, latest recorded error, consecutive failure count, failing-since time, and retry eligibility. Only errors older than 24 hours are listed, including entries explicitly marked unrecoverable. Remote URLs and credentials are not included in the list.

### Marking a file unrecoverable

Only the administrator account, **user 1**, sees the **Mark unrecoverable** action. After confirmation, the server rechecks both the user ID and that the entry still has an unresolved error older than 24 hours. Other administrators cannot perform this action, and a recovered or recently failed entry is not eligible.

Marking an entry unrecoverable stops automatic remote cache-read retries and removes it from active unresolved-error counts. Its row remains in the error list with an **Unrecoverable** label. The file registry record and error details are retained; no local or remote files are deleted. Historical read-rate statistics are unchanged, so a high-error-rate warning can remain until those attempts leave the reporting window. Late callbacks from an attempt already in flight cannot clear the unrecoverable state. A new successful upload of the path resets it.

Integration points
------------------

`mod_filestore` handles these notifications:

- `#media_update_done{}` queues inserted or updated media files.
- `#filestore{}` handles file lookup, upload, and deletion through the file registry and cache.
- `#filestore{action=cafs_adopt}` registers a known-checksum remote original under its canonical CAFS path without downloading. `cafs_retire` retires a redundant legacy path once the canonical remote copy is healthy. Adoption retargets the registry entry while keeping its URL. Both use the caller's transaction.
- `#filestore{action=purge_previews}` removes old generated-preview queue entries and schedules deletion of remote `preview/` files. Archive preview originals are kept.
- `#filestore{action=delete_status}` requests deletion and returns `absent`, `pending`, or an error. Exact paths and prefixes include queued uploads and remote entries; repeated requests preserve the deletion retention deadline.
- `#filestore{action=cancel_delete}` cancels an undispatched deletion in the caller’s transaction. It returns `true` when safe to reuse the path, and `false` once deletion has been dispatched. No handler means cancellation is unconfirmed.
- `#filestore_request{}` handles direct storage upload, download, and deletion requests.
- `#filestore_credentials_lookup{}` maps a local path and optional resource ID to remote credentials and location.
- `#filestore_credentials_revlookup{}` resolves credentials for an existing remote service and location.
- `#admin_menu{}` adds the admin menu entry.

Applications can provide credential lookup observers to route files to different services. See `zotonic_file.hrl` for the notification records and <a href="/id/doc_model_model_filestore" class="doc-reference doc-reference-model"><code>model#filestore</code></a> for the configuration and statistics model. Storage requests use `s3filez`, `ftpfilez`, or `webdavfilez`; `filezcache` manages the shared download cache.

License: 

Attribution: 

Related resources: https://zotonic.com/api/agent/related/1329?language=en


## API access

- [API help](<https://zotonic.com/api-help>)
- [OpenAPI specification](<https://zotonic.com/openapi.json>)
- [This resource as JSON](<https://zotonic.com/api/agent/resource/1329?language=en>)
