The Rationale of Zotonic

In this document we will look into the structure and backgrounds of the current system. First we visit the data model, explain the big ideas behind the simple concepts. This will enable you to think in terms of the data model, see why it is this way and use it for your own models.

Then we look into the code, check the module system and how modules extend Zotonic. We will also show the two message buses in Zotonic, one for the trusted communication between modules and system code, the other for untrusted user facing communication.

The Data Model

Everything is a thing. That is the basis of the data model. Basically there are two tables: one with things, and one with connections between those things. This is modeled after the ideas of the semantic web. The things in the thing table are called resources, and the connections are edges.

Every edge has a label, the predicate, this label gives meaning to the connection. Examples are author, or about. As in: this book has author that person. And yes, edges are directed, the person does not have a book as author. 

Every resource (or thing) has a category, which defines what that resource is. Examples are article, book, or person. Resources also have properties, they are serialized and stored inside the resource as big fat blob. Examples of properties are title, body, name, address, etc. You can define and use properties as you go, there is no limited pre-defined list.

So, if you need something new, you don’t make a new table with some columns. Instead you define a new category and grow the properties it has and how it connects to other resources. You also don’t need to write any code for this, as it can all be defined from the user interface and directly used in the templates. 

The resource and edge models are extremely flexible and easy to use. In code but especially in templates. No need to write controllers or queries to collect information, all data is directly accessible from the templates.

Why resources and edges?

It is all quite simple. Consider keywords. In almost all systems keywords reside in a special keyword table. But then... they need translations, so more columns are added. Then they need descriptions, because you want to write what those keywords mean. You might want to group them, give a longer descriptive name, and a shorter name for when there is no space to display. You might want to define who added the keyword, and when. Then someone wants to write an article about a very special keyword... and now your keyword became a full fledged resource on its own.

Often things–like keywords–start simple, but as you grow more functionality, content and meaning, is added to it. It becomes an entity on its own, with its own page and own representations.

By starting out with resources (and edges) we simplify the whole model. There is no need for extra code, extra tables, extra APIs, extra queries. All whilst ensuring you can grow your data into the future, give it new purpose and meaning. Purposes you might not have thought of when you started.

Don’t forget: the data will live much longer than the web site. It will be copied, molded, reused. The web site will be gone at some time, its data might well be alive and kicking inside a new system.

Which brings us neatly at the following subject.

Decoupling Data and Representation

We mentioned it before: representation. This is an important difference between Zotonic and other content management systems. In many systems the HTML page is the data. You can see this often reflected in that you must give a path name for your data.

But what if you want to use this data on another page? On another web site? In another format than HTML? For example JSON?

In Zotonic the HTML page is a representation of a resource. You can use other formats as well. Modules can add representations and you can request them by specifying your preferred content-types.

It is also typical that a representation changes during the life time of a web site. New templates, new CSS, new ideas. And data can be copied to other systems, having a completely different representation on those systems,

There is a requirement in Zotonic: every resource should have a HTML representation. We call this the page of a resource. That is why in the editorial environment the word pages is used, as resources tends to confuse normal human beings. For the same reason we use the word connection for edges. It is easier for people who are great at content but not so well versed in technical terms.

This doesn’t mean that every HTML page should be a work of art. You can make very simple pages, or even pages that redirect somewhere else, or pages that signal search engines that they don’t want to be indexed.

Make sure that every resource has a page on the web.

URLs and Representations

To stay in semantic web terms: the page (with the HTML representation) has a location: the URL (or URI, depending to whom you are talking). The page URL is an informational URL, as it shows some representation of the resource.

There is also a non-informational URL. Usually the path is something like /id/1234, where 1234 is the id of the resource. This URL doesn’t show anything, if you go there with a browser you will be redirected to the HTML page. 

This redirect happens because the browser sends along a HTTP header which specifies that the browser would like to have HTML, but if not available it is also happy with other representations, like images or whatnot.

If you point your code to the non-informational URL you could specify that you like JSON over HTML. In this case Zotonic will redirect you to the URL that will give the JSON representation of the resource.

In the same fashion you could specify VEVENT in case you know it is a calendar item, or XML, or whatever is available.

So you don’t need to know the special API endpoints to give you some representation of the resources. Specify what you want (with possible fallbacks) and ask the non-informational URL to redirect you to the place you need to be.

In the same way Zotonic will select the language that best matches the languages accepted by the browser. That is, if you don’t force a language by specifying it in the URL. For example /en/id/1234 should give you English.

Languages and Translations

Do you remember those sites that have a completely different menu structure, depending on the language? Or that redirect you back to the home page if you switch to another language?

Zotonic does no such thing. It will keep you where you are and will do its best to show you the content in the languages that best match your preference or selection.

That means that if you select German and something is not available in German, but it is in English, then you will see the English version. That is because most people do speak multiple languages and pretending content is not there is worse than showing another language version. (Especially with the modern translation services.)

In Zotonic you translate page by page. Resource by resource. So you can go live with a language when enough content is translated. Your visitors might see (and read!) some alternative language versions till everything is translated.

All translations are stored in the resource properties. So the title property will have all translations inside the property. There are no separate properties per language, and there are no separate resources per language. Everything is packed inside the properties of the single resource.

In the editorial environment you can select which languages are active for the resource you are editing. 

Administrators add, remove, enable and disable languages for the whole site.

This makes Zotonic ideal for multi-lingual sites. And is another example of content that grows in time.

Time zones

Time is very relative. In Zotonic all dates and times are stored in UTC (aka Universal Time or Greenwich Mean Time). At the moment of display they are mapped to the correct time zone.

The time zone can be fixed for the site, or depending on the visitor. For this we run a small JavaScript on the browser which reports the time zone back to the server. We also use a tiny cookie to store the selected time zone so that the next visit we make a better educated guess which time zone should be used.

With Zotonic you need to worry a lot less about time and time zones.

Files, images and documents

In Zotonic every resource can have file, video or whatever document (or embed code) attached. That is one or zero, to keep things manageable we don’t allow more than one media item “inside” a resource. Of course this single media item can have a preview, like the one that is needed for PDFs or videos (the poster image).

That a media item is a resource like all other resources makes for endless possibilities. Want to add a description? Check!  Want to add dates, keywords, a transcription, references to other versions or related content? No problem.

Want to use an image with some article? Add a connection from the article to the image, using the predicate depiction.

Want to make an album? Make a resource of the category collection and add haspart connections to all the media items that should be in your album.

No fantasy is needed to see that this is a very flexible approach to media handling, much more flexible than a separate directory with files.

Everything is a thing, so a media item is also a thing. Like all other things.

Modules

The core of Zotonic is just that, a core. It has some essential models and support routines. The main functionality of Zotonic is in its modules. Modules are standard (Erlang) applications. One such application bundles templates, controllers, css, images, data models and more. You could call a Zotonic module a micro service.

The Zotonic core keeps track of which modules are present, what files they have, and which URLs map to the module’s controllers.

This indexing builds on an important principle: modules can overrule templates and other files defined in other modules. This works on module priority, if two modules have the same file, then the module with the higher priority wins and the file from that higher priority module will be used. This is especially used for files (think templates,  css an images) and dispatch rules (mappings from URLs to controllers).

As templates can also extend or overrule other templates this priority mechanism allows to add functionality in sites. Examples are extra panels in the editorial interface, or even two factor authentication.

This all without programming or configuration. Enable a module to start using its functionality.

Module Decoupling with Notifications 

Modules can extend or hook into core functionality, or functionality provided by other modules. We don’t want to change code or configurations whenever we enable a module. It should seamlessly hook into existing functions and extend their functionality.

For this we have a notification system. This is an internal PubSub system for trusted communication. There are simple topics a module can subscribe to and other modules can publish on. The subscription is done by adding observer functions, this can be done by having specially named functions in the module. Zotonic core will subscribe those observers to their topics when the module is started.

For publishing there are notification functions: asynchronous, synchronous, first, map, and fold. These combined give a powerful extension mechanism where modules can hook into core functionality or provide hooks on their own.

If you need stronger coupling then a module can define what it provides and what it depends on. This is used by the module manager to build a dependency tree and start the modules in the correct order.

Sites

A web site is a module with a special site configuration file. Zotonic supports virtual hosting to run many sites at once. It will even start sites depending on the incoming requests, delaying requests till their site is started.

Every site runs in its own supervision tree, isolating it from the other sites.

If a site crashes then it is restarted automatically, with incremental back off in case of severe problems.

We added virtual hosting so that you don’t need to install extra machines or proxies to be able to run more than one site. Start the site along other sites and point your DNS to the server. With the Let’s Encrypt module we automatically add a free and valid SSL certificate, no extra configuration needed.

MQTT — Glueing Worlds Together

The internal message bus for communication between modules is the notification system. The external message bus is MQTT. This is used for communication with user facing systems. Examples are browsers, APIs, and Internet of Things devices.

In Zotonic MQTT is an integrated solution. The MQTT topic tree includes all models and is accessible via the HTTP API. Every topic is available as a URL, every model is reachable via topics, and every open browser page is reachable via a topic.

The browser integration is taken further by having a local topic tree in the browser, one for each tab. These topic trees are bound to the server’s topic tree via bridge topics. The effect is a fully unified name space where every browser and client can communicate with any other part of the system.

Having the models directly mapped to topic trees, and the topics directly mapped from the HTTP APIs makes the models extremely powerful. Add a model and access it directly from the templates, APIs, MQTT, and JavaScript. No special coding needed.

With the bridge topics it is even possible for an IoT system to publish directly to a browser. And vice versa, with JavaScript talking to an IoT device.

Another example is access to the browser’s LocalStorage. Not only from JavaScript in the browser, but from server side code. The messages are routed via the topic trees and a response is routed back. With Erlang being multithreaded we can wait for the answer, as if we called a process on the server itself.

Truly an unified address space, with the Zotonic server binding everything together.

Conclusion

With Zotonic we created not only a content management system, but also a communication framework and hub. Its module system makes it extensible. Its template and multi-lingual foundation makes for easy creation of HTML pages. And the MQTT message bus ties everything together into a modern real-time system.

Zotonic is a powerful tool to create real time web systems with ease.