Support for storing uploaded and generated images and documents on external services.<\/p>\n
This module stores uploaded files and generated preview-images on an external S3-compatible service. It listens for\nmedium and file related notifications for any newly uploaded or generated files.<\/p>\n
If a file is added then the file is queued in an upload queue. After a delay a separate process polls this queue and\nwill upload the file to the external service.<\/p>\n
If a file is needed and not locally available then the mod_filestore module will check its file registry to see if the\nfile is stored on an external service. If so then then a filezcache<\/em> process is added and a download of the file is started.<\/p>\n The file is served from the filezcache whilst it is being downloaded.<\/p>\n The filezcache will stop the entry after a random amount of time—if the entry was not recently used.<\/p>\n Configure the following permissions on your S3 service for mod_filestore to work correctly:<\/p>\n After the mod_filestore is enabled an extra menu entry ‘Cloud File Store’ is added to the ‘System’ menu in the admin.<\/p>\n Selecting the menu will show the configuration panel for the Could File Store.<\/p>\n Here you can define where the files should be stored and give the credentials to access the storage.<\/p>\n If you save the url and credentials then the system will try to upload a small file to the remote storage. If it\nsucceeds then the configuration is saved. If it does not succeed then an error message will be displayed and the\nconfiguration will not be changed.<\/p>\n It is possible to (temporarily) disable uploading new files by unchecking the checkbox Upload new files to the cloud<\/em>.<\/p>\n You can also configure file deletion behaviour, i.e. what should happen when a file is removed from Zotonic. You can\nchoose to immediately remove the file from the filestore, not delete it at all (to make your store immutable) or delete\nthe file after a certain delay (to be able to restore accidentally deleted files).<\/p>\n The system shows statistics:<\/p>\n Media<\/p>\n All medium records and a sum of the sizes. A single medium record can have 0, 1 or 2 files attached.<\/p>\n Local Files<\/p>\n These are all files found in the Cloud Files<\/p>\n All files registered to be on any cloud service. This is extracted from the database and not by scanning the remote\ncloud service.<\/p>\n Queues<\/p>\n These are the queues being processed by mod_filestore. On a quiet (stable) system they are usually empty.<\/p>\n It is possible to move (almost) all files from the local file system to the cloud. And vice versa, from the cloud to the\nlocal file system. This is useful when starting or changing the cloud storage location.<\/p>\n If a file is moved to the cloud then it is first placed in the filezcache. The filezcache will start purging the files\nif the cache is bigger than configurated in the filezcache application (default 10GB for all sites combined).<\/p>\n The system waits 10 minutes before a queued file is uploaded. This period is meant for a cool down<\/em> of the file, as in\nthe first moments after an upload some resize and preview operations will take place. The delay makes it less probable\nthat a freshly uploaded file vanishes (to the cache) whilst a preview-generation is starting.<\/p>\n The mod_filestore hooks into the following notifications, whose definitions can be found in Hooks into the Zotonic file management notifications to upload, delete or lookup files. This will trigger downloads of\nexternal files and interfaces to the filezcache.<\/p>\n Maps a local path and optional resource id to a service, external location and key\/password for that external service.\nThis can be used to store different resources on different external services.<\/p>\n Maps a cloud file service and location to a key, password and request location.<\/p>\n Queues newly inserted medium files into the upload queue.<\/p>\n To add the Cloud File Store menu to the admin.<\/p>\n The filestore uses the s3filez and filezcache Erlang applications.<\/p>\n This application is used for uploading, downloading and deleting files on S3 compatible services. It provides\nasynchronous services and is compatible with the filezcache application. It is also able to stream files to and from the\nexternal S3 service, this makes it possible to have start serving a file before it is downloaded to the filezcache.<\/p>\n This application manages a cache of downloaded files. The cache is shared between all sites. Every cache entry is\nmanaged by its own process, which can stream newly received data directly to any requesting processes.<\/p>\n The filezcache keeps a presistent disk_log<\/em> with a description of all files in the cache. This log is read on startup\nto repopulate the cache with already present files. For each file the size and a hash is stored to check cache consistency.<\/p>\n The filezcache has a garbage collector. It keeps a pool of randomly selected cache entries, from which it will elect\nrandomly processes to be garbage-collected. The processes themselves will decide if they will stop or not.<\/p>\n After a cache process stops it will keep running for a short period to handle late incoming requests.<\/p>\n Filezcache entries are started by the mod_filestore and filled by either moving a local file to the cache or by\ns3filez download processes.\nExternal file storage module for media\/file offloading and synchronization.<\/p>\n The statistics are generated dynamically, which is not a good idea with many files. This will be changed.<\/p>\n This module handles the following notifier callbacks:<\/p>\nConfiguration<\/h2>\n
S3 configuration<\/h3>\n
Resource<\/th> Permissions<\/th><\/tr><\/thead> \/<\/code><\/td>* s3:ListBucket<\/td><\/tr> \/-zotonic-filestore-test-file-<\/code><\/td>* s3:GetObject * s3:PutObject * s3:DeleteObject<\/td><\/tr> \/preview\/*<\/code><\/td>* s3:GetObject * s3:PutObject * s3:DeleteObject (if file deletion is enabled)<\/td><\/tr> \/archive\/*<\/code><\/td>* s3:GetObject * s3:PutObject * s3:DeleteObject (if file deletion is enabled)<\/td><\/tr><\/tbody><\/table>\n mod_filestore configuration<\/h3>\n
File deletion<\/h3>\n
Statistics<\/h3>\n
files<\/code> directory, this includes files that won’t ever be uploaded.<\/p>\nMoving files<\/h3>\n
Notifications<\/h2>\n
zotonic_file.hrl<\/code>:<\/p>\n#filestore{}<\/code><\/p>\n#filestore_credentials_lookup{}<\/code><\/p>\n#filestore_credentials_revlookup{}<\/code><\/p>\n#medium_update_done{}<\/code><\/p>\n#admin_menu{}<\/code><\/p>\nApplications<\/h2>\n
s3filez<\/h3>\n
filezcache<\/h3>\n
Accepted Events<\/h2>\n
observe_admin_menu<\/code>: Add filestore settings and browser links to the admin menu.<\/li>observe_filestore<\/code>: Resolve filestore read\/write\/delete requests for configured remote storage backends.<\/li>observe_filestore_credentials_lookup<\/code>: Map the local path to the URL of the remotely stored file using filestore_config:service<\/code>.<\/li>observe_filestore_credentials_revlookup<\/code>: Given the service, find the credentials to do a lookup of the remote file using filestore_config:service<\/code>.<\/li>observe_filestore_request<\/code>: Handle filestore upload, download and delete requests via the configured backend credentials.<\/li>observe_media_update_done<\/code>: Push newly uploaded local media files to the configured filestore backend.<\/li><\/ul>","slug":"mod_filestore","is_protected":false,"visible_for":0,"tz":"UTC","language":["en"],"doc_source_hash":"fa3b79110e697cc29970e6667294bf833619ac54b4f7adc2ada874292a02d9a9","is_featured":false,"content_group_id":{"id":2551,"is_a":["meta","content_group"],"name":"content_group_imported_docs","title":"Imported documentation","uri":"https:\/\/zotonic.com\/id\/content_group_imported_docs"},"category_id":{"id":320,"is_a":["meta","category"],"name":"module","title":"Modules","uri":"https:\/\/test.zotonic.com\/id\/320"},"doc_source_path":"apps\/zotonic_mod_filestore\/src\/mod_filestore.erl","publication_start":"2023-06-24T08:17:53Z","github_url":"https:\/\/github.com\/zotonic\/zotonic\/blob\/master\/apps\/zotonic_mod_filestore\/src\/mod_filestore.erl","pivot_location_lng":null,"doc_source_kind":"module","name":"doc_module_mod_filestore","is_unfindable":false,"is_published":true,"pivot_geocode":null,"created":"2020-05-30T05:47:18Z","uri":null,"doc_status":"current","is_dependent":false,"erlang_app":"zotonic_mod_filestore","publication_end":"9999-06-01T00:00:00Z","modifier_id":{"id":1,"is_a":["person"],"name":"administrator","title":"Site Administrator","uri":"https:\/\/zotonic.com\/id\/1"},"privacy":0,"doc_source_commit":"66cb145eae7002b184e7d8c4c7bba5e53e8248e5\n","creator_id":{"id":336,"is_a":["person","robot"],"name":"gitbot","title":"Git","uri":"https:\/\/zotonic.com\/id\/336"},"modified":"2026-10-02T12:40:20Z","title_slug":"mod_filestore"},"uri":"https:\/\/zotonic.com\/id\/1329","uri_template":"https:\/\/zotonic.com\/id\/:id","websub":{"hub":"https:\/\/zotonic.com\/.zotonic\/websub","topic":"https:\/\/zotonic.com\/.zotonic\/websub\/topic\/1329"}},"status":"ok"}