{"result":{"depiction_url":null,"edges":{"refers":{"objects":[{"created":"2026-11-05T18:43:22Z","object_id":{"id":1284,"is_a":["text","documentation","reference","model"],"name":"doc_model_model_media","title":"media","uri":"https:\/\/zotonic.com\/id\/1284"},"seq":1000000},{"created":"2026-11-05T18:43:22Z","object_id":{"id":1329,"is_a":["text","documentation","reference","module"],"name":"doc_module_mod_filestore","title":"mod_filestore","uri":"https:\/\/zotonic.com\/id\/1329"},"seq":1000000},{"created":"2026-11-05T18:43:22Z","object_id":{"id":1521,"is_a":["text","documentation","reference","module"],"name":"doc_module_mod_backup","title":"mod_backup","uri":"https:\/\/zotonic.com\/id\/1521"},"seq":1000000},{"created":"2026-11-05T18:43:22Z","object_id":{"id":2140,"is_a":["text","documentation","reference","model"],"name":"doc_model_model_filestore","title":"filestore","uri":"https:\/\/zotonic.com\/id\/2140"},"seq":1000000},{"created":"2026-11-05T18:43:22Z","object_id":{"id":3141,"is_a":["text","documentation","reference","model"],"name":"doc_model_model_cafs","title":"cafs","uri":"https:\/\/zotonic.com\/id\/3141"},"seq":1000000}],"predicate":{"id":2409,"is_a":["meta","predicate"],"name":"refers","title":{"_type":"trans","tr":{"en":"Refers"}},"uri":"https:\/\/zotonic.com\/id\/refers"}},"subject":{"objects":[{"created":"2026-11-05T18:43:22Z","object_id":{"id":2556,"is_a":["categorization","keyword","keyword_information_type"],"name":"zotonic_topic_explanation","title":"Explanation","uri":"https:\/\/zotonic.com\/id\/2556"},"seq":1},{"created":"2026-11-05T18:43:22Z","object_id":{"id":2564,"is_a":["categorization","keyword","keyword_audience"],"name":"zotonic_topic_backend_developer","title":"Backend developer","uri":"https:\/\/zotonic.com\/id\/2564"},"seq":2},{"created":"2026-11-05T18:43:22Z","object_id":{"id":2566,"is_a":["categorization","keyword","keyword_audience"],"name":"zotonic_topic_operator","title":"Operator","uri":"https:\/\/zotonic.com\/id\/2566"},"seq":3},{"created":"2026-11-05T18:43:22Z","object_id":{"id":2576,"is_a":["categorization","keyword","keyword_domain"],"name":"zotonic_topic_media_management","title":"Media management","uri":"https:\/\/zotonic.com\/id\/2576"},"seq":4},{"created":"2026-11-05T18:43:22Z","object_id":{"id":2581,"is_a":["categorization","keyword","keyword_domain"],"name":"zotonic_topic_file_storage","title":"File storage","uri":"https:\/\/zotonic.com\/id\/2581"},"seq":5},{"created":"2026-11-05T18:43:22Z","object_id":{"id":2602,"is_a":["categorization","keyword","keyword_domain"],"name":"zotonic_topic_backup_and_restore","title":"Backup and restore","uri":"https:\/\/zotonic.com\/id\/2602"},"seq":6},{"created":"2026-11-05T18:43:22Z","object_id":{"id":2630,"is_a":["categorization","keyword","keyword_architecture"],"name":"zotonic_topic_cache","title":"Cache","uri":"https:\/\/zotonic.com\/id\/2630"},"seq":7}],"predicate":{"id":308,"is_a":["meta","predicate"],"name":"subject","title":{"_type":"trans","tr":{"en":"Keyword"}},"uri":"http:\/\/purl.org\/dc\/elements\/1.1\/subject"}}},"id":3142,"is_a":["text","documentation","developerguide"],"links":[{"rel":"self","target":"https:\/\/zotonic.com\/.zotonic\/websub\/topic\/3142"},{"rel":"hub","target":"https:\/\/zotonic.com\/.zotonic\/websub"}],"medium":null,"medium_url":null,"name":"doc_guide_file_storage","page_url":{"en":"https:\/\/zotonic.com\/docs\/3142\/how-zotonic-stores-and-collects-media-files","x-default":"https:\/\/zotonic.com\/docs\/3142\/how-zotonic-stores-and-collects-media-files"},"preview_url":null,"resource":{"version":20,"pivot_location_lat":null,"title":"How Zotonic stores and collects media files","is_authoritative":true,"body":"

How Zotonic stores and collects media files<\/h1>\n

Zotonic separates a media resource's identity from its file bytes. A resource has a\nmedium record with a logical filename and metadata. CAFS<\/strong>, the content-addressable\nfile store, lets several logical filenames share the same bytes. Filestore<\/strong> can\nmove those bytes to remote storage and retrieve them when needed.<\/p>\n

These layers work together. CAFS decides which content is shared and still needed;\nfilestore manages where the bytes are available. The mediarunner<\/a>\nadds an optional processing service, with its own temporary cache. It does not\nreplace the site's media store.<\/p>\n

From a medium record to bytes<\/h2>\n

\"Two<\/p>\n
Layer<\/th>What it represents<\/th><\/tr><\/thead>
Resource and medium<\/td>Ownership, metadata, access rules and logical filenames<\/td><\/tr>
CAFS mapping<\/td>The SHA-256 content hash behind a logical filename<\/td><\/tr>
Physical storage<\/td>The local file or its remote filestore object<\/td><\/tr>
File cache<\/td>A local copy used while serving or processing remotely stored bytes<\/td><\/tr><\/tbody><\/table>\n

A content hash is an identity for bytes, not an authorization token. Two resources\ncan share content while retaining different permissions. Public media delivery still\nuses the requesting resource's access rules and the normal signed preview URLs.\nApplications should use Zotonic's media and file APIs rather than construct public\nURLs from physical hashes.<\/p>\n

Where files live<\/h2>\n

Paths below are relative to the site's files directory, normally beneath\n<data_dir>\/sites\/<site>\/files\/<\/code>:<\/p>\n
Path<\/th>Purpose<\/th><\/tr><\/thead>
archive\/cafs\/sha256\/ab\/cd\/<hash><\/code><\/td>A CAFS object: an original, poster, shared preview or retained task input<\/td><\/tr>
archive\/<logical-filename><\/code><\/td>An older file that has not been promoted to CAFS<\/td><\/tr>
preview\/<preview-filename><\/code><\/td>Legacy previews and previews on database-free sites<\/td><\/tr><\/tbody><\/table>\n

Before a browser uploads a file, Zotonic hashes it in a worker and checks for a\nmatching current medium that the user can view. A match skips the browser transfer;\na miss uploads normally. Older originals with a known matching checksum are\nlazily promoted to CAFS; files without a checksum upload normally. Reuse still runs the normal validation and media processing,\nand may fetch remote-only bytes on the server. See m_media<\/a> for the\nupload preflight API and its short-lived, user-bound token.<\/p>\n

The CAFS path uses cafs<\/code> without a leading dot<\/strong>. Its first two directory levels\ncome from the first four hash characters. The complete hash is 64 lowercase\nhexadecimal characters, and the object path has no file extension. MIME type and\nthe logical filename remain part of the media metadata.<\/p>\n

New files are hashed when staged, then attached to their logical filenames. Fresh\nprivate upload copies are hashed once, and the medium digest reuses that hash.\nLegacy files and other potentially mutable sources are hashed before copying and\nagain afterward, rejecting changes during copying. Copying\nan existing CAFS-backed medium can share the mapping without copying or downloading\nits bytes. A medium copy also brings its owned original and poster into CAFS where\nneeded; it preserves independent resource metadata and permissions. See\nthe media model<\/a> for the copy APIs and the CAFS model<\/a>\nfor staging, attachment and temporary access to local bytes.<\/p>\n

Existing installations do not need a mass migration. Older files, with or without\nan existing SHA-2 checksum, continue to work. Available bytes can be hashed and\npromoted during an operation such as copying. An existing checksum alone is not a\nsubstitute for a CAFS mapping. Installation does not download all remote files to\ncalculate hashes. An operation that actually needs a legacy file may still need to\nretrieve that particular file.<\/p>\n

Shared previews<\/h2>\n

A preview is determined by its original content and the effective rendering\noperations. For example, two resources using the same original and the same resize\ncan share one generated result. Changing operation order can change the output,\nso operation order is part of the identity.<\/p>\n

CAFS identifies a derivative using the source hash, effective operations and\nrenderer identity. The preview pipeline also includes the rendering metadata and\noutput format that affect the result. Concurrent requests share generation work.\nThe resulting preview bytes are stored as another CAFS object.<\/p>\n

Zotonic tracks uses of a preview<\/strong> separately from the generated bytes. A use\nrecords the source and the relevant mediaclass and image-edit revision. This lets\nan old URL become obsolete while another resource still uses the same output.<\/p>\n

When changing rendering dependencies outside the operations themselves, such as\nfonts or policy, update the site's site.cafs_renderer_version<\/code> and invalidate\nrunning file entries with z_file_sup:refresh\/0<\/code>. If a mediarunner is used, review its\nseparate mediarunner_cache_version<\/code> too; both caches must distinguish the new output.<\/p>\n

Adding remote filestore<\/h2>\n

Enable mod_filestore<\/a> and open System → Cloud File Store<\/strong>\nin the admin. Configure a site-specific storage location and credentials, enable\nuploading, and choose whether to keep local copies and how long to delay remote\ndeletion. Saving the configuration tests writing, reading and deleting a temporary\nobject. Follow it with a normal image upload and preview request.<\/p>\n

Filestore supports S3-compatible storage, FTP over TLS and WebDAV. Use the module's\nsetup documentation for URL schemes, S3 regions and required permissions. Keep the\nremote location private; Zotonic can serve files through its normal access checks.<\/p>\n

For CAFS objects, filestore uses the same physical key, including the\narchive\/cafs\/sha256\/…<\/code> path. Shared previews therefore also live under archive\/<\/code>\nin remote storage. Older preview files can still use preview\/<\/code>. Storage policies\nand permissions need to allow both layouts during migration.<\/p>\n

After upload, a site can retain the local copy or let it be removed. A later read\nof a remote-only object obtains a local cached copy. Processing holds the required\nfile protection while reading, so cache eviction does not remove an active input.\nThe registry and retry queues survive temporary storage failures. Bulk downloads\nverify the recorded size and, for CAFS objects, the SHA-256 hash before publishing\nthe local file or scheduling removal of the remote copy. Failed downloads remain\nqueued with backoff; restarting the bulk operation is not required.<\/p>\n

Use Zotonic's supported bulk upload and move operations when changing storage.\nAvoid moving or deleting objects by hand: database mappings, shared references,\nremote locations and pending deletions must stay consistent. The\nfilestore model<\/a> and module documentation describe those operations.<\/p>\n

What garbage collection removes<\/h2>\n

Deleting one medium does not necessarily make its bytes disposable. Another\nlogical filename, a preview relationship or a processing task may still need them.\nA pin<\/strong> is a retention reference held while an operation needs an object. Staging\npins expire; longer-lived tasks, such as video conversion, retain their inputs until\ncompletion or reconciliation.<\/p>\n

\"Removing<\/p>\n

There are several retention stages, rather than one universal deletion timer:<\/p>\n
Event<\/th>What happens<\/th><\/tr><\/thead>
A medium is deleted or replaced<\/td>Its old filename remains available for the existing five-week recovery period before cleanup retires the mapping.<\/td><\/tr>
All preview sources are removed<\/td>Their preview uses become obsolete; shared uses and processing references can still retain the output.<\/td><\/tr>
A URL's mediaclass checksum or image-edit revision is superseded<\/td>That preview use becomes obsolete and enters the preview grace period.<\/td><\/tr>
No required references remain<\/td>Bounded maintenance can collect the CAFS object when its collection deadline permits.<\/td><\/tr>
A remotely stored object is collected<\/td>Filestore applies its remote deletion policy and retains deletion state until the operation is confirmed.<\/td><\/tr><\/tbody><\/table>\n

Obsolete preview uses have a default 90-day grace period<\/strong>. Requests to an\nobsolete URL do not keep extending that deadline. Reverting settings can make a use\ncurrent again. Cleanup also handles tracked legacy preview paths. Existing\nunversioned URLs use a first-observed edit baseline, so upgrading does not invalidate\nall old previews at once.<\/p>\n

Collecting the last source object can also release its derivatives after the source\nrecovery period; the preview grace period does not keep an otherwise unreferenced\noriginal alive.<\/p>\n

Collection is lazy and runs in batches. The grace periods, remaining references,\nmaintenance schedule, storage outages and remote deletion policy all affect when\nbytes finally disappear. Do not expect every associated preview to vanish\nimmediately when the last visible medium is deleted. A filestore setting of\nNever<\/strong> deliberately prevents remote deletion.<\/p>\n

Private preview and download artifacts are a separate cleanup concern. Worker\nmonitors remove temporary files after failure, and a bounded sweep removes eligible\nold orphan artifacts after a restart. This does not treat ordinary previews as\nexpired merely because their files are old.<\/p>\n

Configuration and maintenance<\/h2>\n

The main CAFS controls are per-site m_config<\/code> settings<\/strong>, under the site<\/code>\nnamespace:<\/p>\n
Key<\/th>Default<\/th>Purpose<\/th><\/tr><\/thead>
site.cafs_preview_retention_days<\/code><\/td>90<\/code><\/td>Grace period for obsolete preview uses<\/td><\/tr>
site.cafs_renderer_version<\/code><\/td>1<\/code><\/td>Additional rendering identity after changes outside the recorded operations<\/td><\/tr><\/tbody><\/table>\n

For example:<\/p>\n

m_config:set_value(site, cafs_preview_retention_days, 180, Context).\n<\/code><\/pre>\n
These are different from system-level mediarunner connection settings and\nmod_filestore<\/code>'s storage configuration. CAFS is installed by Zotonic's core schema;\nthere is no separate CAFS module to activate. See m_cafs<\/a> for accepted\nvalues, maintenance APIs and reference handling.<\/p>\n
For capacity planning, account for retained originals, obsolete previews during\ntheir grace period, remote download caches and active processing files. If storage\nkeeps growing, inspect upload\/deletion backlogs, retained processing tasks, preview\nretention and the remote deletion policy before removing files manually.<\/p>\n
Back up the database and the authoritative media bytes together. A local cache may\ncontain only a fraction of remotely stored media, and the mediarunner cache is not\na backup. Keep the remote credentials and storage configuration needed for recovery.\nSee backup and restore<\/a> for the supported workflow.<\/p>\n
Delete all generated previews<\/h2>\n
On Maintenance → Media<\/strong>, use Delete generated previews<\/strong> to queue a complete\ncleanup of resized image variants. The action requires permission to use\nmod_admin_config<\/code> and runs in bounded background batches, visible in the task queue.<\/p>\n
It clears CAFS derivative references and legacy files in the generated preview\/<\/code>\ndirectory, including library-image variants. It also removes pending uploads of\nold legacy variants and queues their remote deletion. Filestore retention and\nread-only settings still apply; an unavailable backend is retried.<\/p>\n
Uploaded originals and all medium preview_filename<\/code> images (movie posters, PDF\ncovers, embed thumbnails, etc.) are preserved. Those stored preview originals\nbelong to the archive, including archive\/preview\/<\/code>, rather than the generated\npreview directory. CAFS objects shared with originals, stored previews, other\nreferences or active processing remain protected.<\/p>\n
Previews regenerate on demand. The task uses a fixed cutoff, so it does not keep\ncollecting newly created variants while visitors browse. Active CAFS generators\nfinish before their old derivative references are removed. Cleanup preserves URL\nobsolescence records, so old invalid mediaclass\/edit URLs do not become valid again.<\/p>\n
Backend deletion coordination<\/h2>\n
Remote deletion is coordinated through #filestore{action=delete_status}<\/code>. The\nbackend reports absent<\/code> only after both its remote records and queued uploads\nare gone; otherwise cleanup retains its tombstone and retries. This also applies\nto legacy files and preview prefixes. Repeated checks preserve the backend's\nretention deadline.<\/p>\n
The internal site.filestore_used<\/code> marker records whether a backend has ever been\ninstalled. It is maintained automatically, including an upgrade backfill for\npreviously installed filestore modules; it is not an administrator setting and\nmust not be reset when disabling filestore. Without a handler, sites carrying\nthis marker defer cleanup. Sites that have never used a backend can still clean\nup local files. Backend implementations must set this marker before accepting\nfiles and implement the deletion-status notification.<\/p>\n
Sites without a database<\/h2>\n
Sites such as zotonic_site_status<\/code>, configured with dbdatabase = none<\/code>, skip CAFS.\nTheir occasional library previews are written to the ordinary preview\/<\/code> directory\nand need no preview garbage collection. They require neither CAFS tables nor remote\nfilestore to generate those previews.<\/p>\n
This behavior is selected by site configuration. A temporary database outage on\na database-backed site does not switch its files into a different storage model.<\/p>\n
Deduplicating existing media<\/h2>\n
Use Maintenance → Media → Deduplicate media files<\/strong>, or call\nm_media:migrate_to_cafs(Context)<\/code> with use mod_admin_config<\/code> permission.\nThe background scan visits every medium present when it starts, including its\nowned original and its extracted preview image (preview_filename<\/code>). Logical\nfilenames and media URLs remain unchanged. Identical bytes share one CAFS object.<\/p>\n
Local files are hashed even when they have no checksum yet. Remote-only originals\nwith a valid SHA-256 digest can reuse their existing remote location without a\ndownload. Remote-only files without a checksum, including extracted previews,\nare skipped and logged. This operation never downloads files merely to hash them;\nrerun it after making missing bytes available locally if needed.<\/p>\n
The scan runs in bounded batches, and individual files retry independently after\nfailures. Cleanup also retries separately. A redundant local copy is removed only\nafter the CAFS mapping is published and a canonical copy is available. A legacy\nremote copy is retired only after a healthy canonical remote entry exists. When\nreusing an existing remote object, its registry path changes atomically with the\nCAFS attachment; the remote URL and bytes stay in place. Deletion of redundant\nremote objects follows the filestore retention policy (including\nNever<\/strong>). Disabled or read-only backends delay cleanup. It is safe to rerun the\noperation; tasks resume after a restart. This does not delete extracted preview\noriginals or regenerate resized previews.<\/p>\n
Both media maintenance actions only queue work from the admin request. Durable\nbackground tasks perform the scan, hashing and cleanup, and resume after restarts.\nThe Maintenance<\/strong> and System status<\/strong> pages show Running<\/strong>, Queued \/ waiting<\/strong>\nor Idle<\/strong> for each operation and refresh every five seconds. Deduplication remains\npending while its per-medium or legacy-cleanup tasks remain, even after the scan\nfinishes. These are task counts, not file counts. Idle means no maintenance tasks\nremain; remote objects already marked for deletion still follow filestore retention.<\/p>\n
On-demand promotion and legacy previews<\/h3>\n
Requests for owned legacy originals also trigger background promotion. Local files\ncan be hashed immediately; a remote file is promoted after the download already\nstarted by the request completes. The response does not wait for hashing, and no\nadditional download is started for promotion. The worker holds a cache lock while\nusing downloaded bytes and rechecks the medium before attaching them. It updates\nthe original checksum and uses the same remote adoption and legacy cleanup as\nmaintenance migration. Extracted posters are handled as owned source files, too.<\/p>\n
Unowned files and database-free sites are excluded. An overloaded worker pool,\nfailed download, or download that has not completed within five minutes leaves\nthe legacy file usable; promotion can be tried on a later cold lookup. Completed\npromotions queue durable cleanup, including retries for a busy filestore.<\/p>\n
Opening a media overview may request previews of older files. When new rendering\nis necessary, promotion first tries to reuse the original's existing remote\nlocation. Busy uploads and unavailable storage are retried before allowing a new\nCAFS upload. Redundant legacy paths use the same background cleanup as maintenance\nmigration; remote removal respects retention.<\/p>\n
The editrev<\/code> URL addition alone no longer forces regeneration. If the source,\nmediaclass checksum and edit revision are current, lookup also tries the old URL\nwith exactly the same rendering operations and output extension. Existing local or\nremote preview bytes are promoted into CAFS without resizing. This requires a\nknown source hash: local sources are migrated on demand, and remote sources with a\nknown checksum can be adopted. Remote sources without a checksum keep their legacy\npreviews until the original becomes available; promotion does not download those\noriginals. A remote preview itself may be downloaded to calculate its output hash.<\/p>\n
Promotion publishes the shared derivative before background cleanup removes its\nold local path. Cleanup adopts an existing remote object without uploading it again,\nor retires a redundant remote copy according to filestore retention. Busy uploads\nand unavailable backends are retried. Local-only outputs become eligible for normal\nCAFS upload once remote absence is confirmed.<\/p>\n
On subsequent cold lookups, a registered current URL resolves directly to its CAFS\noutput. This uses two metadata reads (the output mapping and current source), plus\nany uncached edit settings and the protected physical file\/filestore lookup.\nRegistering a CAFS reader briefly locks the object row to exclude concurrent\ncollection or local offloading before the file is opened. It avoids legacy\npath probes and usage metadata writes. Source, mediaclass, edit revision and renderer\nversion checks remain enforced; obsolete URLs still follow the retention path.\nChanged operations produce a different legacy path, and invalid signatures cannot\nuse the fallback. Database-free sites continue to use their local preview directory.<\/p>\n
Legacy-file migration safeguards<\/h3>\n
Original mappings and promoted preview outputs commit together with their cleanup\ntasks. A rollback or restart cannot publish a migration while losing its required\ncleanup. Files that disappear or change during staging are left for a later retry.\nUnowned source files are neither promoted nor removed, including when a resizer\nrequests them. Cleanup also preserves an original path shared with an unowned\nmedium reference.<\/p>\n
Legacy local copies stay in place while a file entry still uses them. Cleanup\nretries after those entries go idle, instead of interrupting a response. Completed\nfilestore downloads are resolved by their stable cache key, so promotion and reads\ncontinue to work after the temporary download process exits.<\/p>\n
When mod_filestore.is_local_keep<\/code> is false, adopted remote objects also enter the\nnormal filestore queue for local offloading. This does not upload the object again.\nActive CAFS pins and file entries postpone removal of the local canonical copy.<\/p>","slug":"how-zotonic-stores-and-collects-media-files","is_protected":false,"visible_for":0,"tz":"UTC","language":["en"],"doc_source_hash":"7250d45e8f37db154684087ca36661855545bb9c2e4db51328b71224c3d4aefc","is_featured":false,"content_group_id":{"id":2551,"is_a":["meta","content_group"],"name":"content_group_imported_docs","title":"Imported documentation","uri":"https:\/\/zotonic.com\/id\/content_group_imported_docs"},"category_id":{"id":317,"is_a":["meta","category"],"name":"developerguide","title":"Developer guide","uri":"https:\/\/zotonic.com\/id\/developerguide"},"doc_source_path":"doc\/guides\/file-storage.md","publication_start":"2026-11-05T18:43:22Z","github_url":"https:\/\/github.com\/zotonic\/zotonic\/blob\/master\/doc\/guides\/file-storage.md","pivot_location_lng":null,"doc_source_kind":"guide","name":"doc_guide_file_storage","is_unfindable":false,"is_published":true,"pivot_geocode":null,"created":"2026-11-05T18:43:22Z","uri":null,"doc_status":"current","is_dependent":false,"publication_end":"9999-06-01T00:00:00Z","modifier_id":{"id":1,"is_a":["person"],"name":"administrator","title":"Site Administrator","uri":"https:\/\/zotonic.com\/id\/1"},"privacy":0,"doc_source_commit":"becf161621dcc4de964bbf083a88aff6d3ab21ae\n","creator_id":{"id":1,"is_a":["person"],"name":"administrator","title":"Site Administrator","uri":"https:\/\/zotonic.com\/id\/1"},"modified":"2026-11-05T18:43:22Z","title_slug":"how-zotonic-stores-and-collects-media-files"},"uri":"https:\/\/zotonic.com\/id\/3142","uri_template":"https:\/\/zotonic.com\/id\/:id","websub":{"hub":"https:\/\/zotonic.com\/.zotonic\/websub","topic":"https:\/\/zotonic.com\/.zotonic\/websub\/topic\/3142"}},"status":"ok"}