Everything Must Go

The ownCloud Infinite Scale (oCIS)enterprise file-sharing and syncing software stack is a fast and scalable yet complete rewrite of the classic ownCloud. The current version of oCIS is the technology preview 1.8.0 (as of July 2021). Go, Vue.js, and a modular approach replace the outdated PHP, database, and POSIX filesystem the older versions needed. In internal documents, ownCloud promises up to 10 times better performance than ownCloud 10.6.0 published in December 2020 and, so far, the last PHP-based ownCloud variant.

Some History

A shooting star in 2010, ownCloud hasn't make it into the news that much recently, other than with its fork Nextcloud [1]. Whoever thinks that might be attributed to Nextcloud's media success and big community misses the point, though. Over the years, the two companies that once were one have developed in completely different directions. Nextcloud focused on its community and web front end, a web app store with many community apps targeting the growing market of web desktops like Microsoft 365, whereas ownCloud quietly improved the back end and dealt with enterprise needs of (not just) their customers by refactoring the software and making it faster and more robust.

In hindsight, it almost looks like shared work, but it never was. After all, some of these different expectations inside management were why the company broke up in 2016 – but that is another story. Today both companies claim to be very successful with their approaches, and jokes have it that both are convinced they are light years ahead of their competitor.

PHP vs. Performance

Now in mid 2021, a big change is about to come. ownCloud is diverging from the common path and presents a complete rework of the back end. Over the years, the ownCloud developers had to realize the limits of the PHP architecture, especially in large setups with the need for massive scalability (e.g., CERN). Version 1 of oCIS [2] is the result of years of work: a complete rewrite of the software, a new back end in Go, a front end in Vue.js, and other changes (e.g., removing the need for a database).

When ownCloud was founded, PHP seemed to be the natural choice because Ruby, Go, Vue.js, and reactive design were not around then. PHP dates back to 1995, and even the fifth major release, PHP 5, was eight years old when ownCloud finally left stealth mode. Developers have invested a lot of work to adapt the dinosaur to modern software principles and to improve its performance, as have admins. The basic design and architecture of PHP, however, has remained the same throughout the years, which effectively means that a number of almost mandatory modern day software principles are not honored.

Scalability Without PHP and POSIX

A further ownCloud requirement was the need for a separate database in the background, a web server hosting PHP content, and a filesystem for storage. To be more precise, ownCloud has been centered around the idea of having a POSIX-compatible filesystem to store data uploaded by users, different versions of these files and the trash files, as well as configuration files and logs. By default, all files belonging to users can be found in a path like /var/www or /srv/www on the ownCloud instance – a web server's document root.

Admins who have ever called an ownCloud or Nextcloud their own know that ownCloud setups grow massively in size. Today, most of them even begin at much bigger sizes than ownCloud was originally planned to serve. One of the largest ownCloud users today, an Australian university is storing the data of more than 100,000 users.

oCIS is the company's attempt to break free of all the barriers and limitations of the PHP past. The paradigm shift goes deep: The last years have been spent fixing and adapting the existing codebase to squeeze a bit more speed out of the system. All suffering admins, users, and developers may now see a light at the end of the tunnel: oCIS does not recycle any code from ownCloud; it's a complete rewrite.

Let's Go

The programming language Go brings in a large number of advantages (e.g., concurrency). Most computer programs, even if they appear to be one monolithic piece of code from the outside, are split into different components internally. Web servers (e.g., the Apache web server, which is usually deployed with ownCloud) are excellent examples. Internally, they usually have a function handling TCP/IP connections; another function might be responsible for handling SSL, and yet another piece of code might execute the requested PHP files and deliver the results to the end user.

All of those events happen, however, in a certain order. PHP and Apache simply can't handle concurrency: The individual steps of handling requests are done one after another, not at the same time, even if it was logically possible. For a client request to be served, multiple steps must be performed. Any environment with concurrency will allow all those events to happen at the same time instead of serially. Servers that are capable of handling requests in parallel will have to wait less for processes to finish their work, thus offering faster results to the user. By the way, this feature is one of the reasons why Go has become one of the de facto standards for containerized microarchitecture applications.

A More Modular ownCloud

ownCloud is adapting to an architecture centered around the principle of microservices. oCIS splits into three tiers: storage, core, and front end. In this article, we look at each of these tiers separately. Even though for clients who upload and download files all that counts is the overall performance (including bandwidth), choosing the right underlying storage is crucial. In its three-tier model, ownCloud considers the storage available to the system to be the lowest tier.

Along with performance goes scalability: Large ownCloud instances must be able to cope with the load of thousands of clients and must allow additional disk space if the existing space fills up. Like so many concepts, object stores and scalable storage wasn't as available in the past when ownCloud was initially designed as it is today, which is yet another good argument to review oCIS.

Administrators today are used to having more choices, and ownCloud permits outsourcing the handling of physical storage devices to an external solution. Although S3-based object storage, storage based on the Samba service, or POSIX-compatible filesystem options continue to be supported, the preferred way to deploy oCIS is the way of Earth Observing System (EOS) Storage [3].

EOS to the Rescue

Originally provided by CERN in Geneva for its Large Hadron Collider experiment, EOS is optimized to feature very low latency when accessing files. It provides disk-based storage to clients by the XRootD [4] framework but also permits other protocols to access files. ownCloud uses the EOS HTTP protocol extension to talk to the storage solution (over HTTPS, of course). Far more interesting, though, is that it allows almost infinite scalability. For instance, at the time of writing this article, the CERN setup featured more than 200PB of disk storage and continues to grow. By turning to EOS, ownCloud got rid of a number of shortcomings: EOS does not have a typical single point of failure, and all relevant services are run redundantly, including its ability to scale out and add additional instances of all existing services. EOS promises never to run out of disk space and comes with built-in redundancy for the data stored therein.

Other Options

Consequently, at least for large environments, ownCloud will expect the administrator to deploy an instance of EOS along with oCIS. The trade-off is clear: In exchange for the burden of having to maintain a separate storage system, you get the benefit of not having to worry about scalability and performance of the oCIS instance. This assumption also provides a hint about the assumed use case for oCIS and ownCloud: ownCloud's strategy with oCIS targets big data centers. For SoHo and end-user setups, EOS will, however, likely be over the top and much more complex than what the average end user is willing to maintain, let alone a lack of appropriate hardware.

Smaller setups are served by oCIS through the framework by enabling the aforementioned support for Amazon S3, Samba, and POSIX-compatible filesystems, which is possible because EOS is not hard-coded into oCIS. Reva [5] on the one hand can't provide the same feature set as EOS but will accomplish most of the needs of end users and non-EOS-sized setups. Its strategic importance if EOS, on the other hand, shows when considering that only the combination of EOS and oCIS permits the full feature set of ownCloud.

Microservice Core

The second tier of oCIS is (thanks to Go) more of a collection of microservices than a core. Each and every microservice is responsible for a single task in the background (e.g., scanning for viruses). Basically all functionality provided by oCIS is the direct result of the work of a specific microservice, like authenticating requests with OpenID Connect against an identity provider. In the end, that makes connecting existing user directories (e.g., ADFS, Azure AD, or LDAP) to ownCloud a simple task. For those who do not have an existing identity provider, ownCloud ships its own instance, effectively making ownCloud maintain its own user database.

Brand New User Interface

The third tier of oCIS is what the vendor calls Phoenix or ownCloud Web [6]. The completely rewritten user interface is based on the Vue.js JavaScript framework. Just like the oCIS core itself, Phoenix is written according to microservice principles and hence allows for better performance and scalability. The developers also used the opportunity to give the web interface a makeover; compared with the previous ownCloud versions, the oCIS web interface looks reduced and slick.

The oCIS developers did an impressive job regarding compliance with modern software design principles. The fundamental problem in building applications while following the microservice approach is usually how to make the individual components of the environment communicate with each other. APIs come to the rescue, but then every microservice component must have its own, well-defined API interface.

Luckily, a number of tools are available in the wild taking that burden off the shoulders of developers, most notably Google remote procedure calls (gRPC). The basic idea behind gRPC is to have a set of predefined APIs that allow actions in one component to be triggered from within another.

Traefik

The application design brings about some challenges to the underlying network, and the oCIS developers have chosen the Traefik framework [7] to tackle those. Traefik automatically load balances between the different instances of microservices, takes care of automated SSL encryption, and allows for the additional deployment of firewall rules.

The split between the back end and the front end add additional advantages to oCIS. In fact, actions triggered by the user by means of ownCloud Web are completely decoupled from the ownCloud engine performing the respective task in the back end. If a user manually starts a virus check on their file(s) stored in ownCloud, they do not have to wait for the check to be finished. Instead, the check itself happens in the background, and the user only ever sees the result once the check is completed – the principle of concurrency at work.

Extensions as Microservices

Like other web services, ownCloud also supports extending its capabilities with extensions. oCIS doesn't change that, but it is promising to tackle a well-known problem, especially with community apps. Apps of unknown origin can cause trouble in the server and hamper updates and often have negative effects on the server's overall performance. The new oCIS gRPC-based architecture makes it much easier to create additional extensions spawned alongside the already existing microservices. Because the API is predefined by gRPC, all developers really have to do is create a microservice featuring the desired functionality that can be controlled by gRPC. Traefik will, on a per-case basis, ensure that newly deployed add-ons are automatically added to the existing communication mesh. For users and third parties interested in ownCloud, this means that a flood of additional ownCloud functionality will likely start to pour in for oCIS.

Arbitrary Extensions

The oCIS developers make it very clear that they want to support such extensions, and they are even making it very easy for developers to integrate custom modules. Their documentation [8], for instance, contains a detailed explanation of how to write oCIS extensions, along with code for a real-world example.

Said documentation suggests that writing an oCIS extension is not super-complex for experienced Go jockeys. Three major components are required: the server providing the extension's functionality through a gRPC API, a so-called "greeter API" to serve requests to the extension from the outside, and, if desired, a separate component integrating the extension into ownCloud Web.

The greeter API integrates with the oCIS server itself to allow commands to the server to be caught on the API of the oCIS instance. It communicates with the server through gRPC. The Hello World example [9] in the oCIS documentation illustrates this clearly: For instance, the command received through the oCIS API could be for Hello World to be returned as the result of an oCIS API request. The greeter API would catch that request, trigger the corresponding action over gRPC, and deliver the return value. The web interface can be used to send the Hello World request to the greeter API.

Although a simple Hello World is an oversimplified example, the flow of commands brought up by oCIS developers also allows for highly complex operations to be performed by oCIS extensions. Of course, all oCIS extensions will still benefit from Go's concurrency functionality and its default optimization for performance purposes.

Goodbye MySQL!

ownCloud's switch to gRPC and microservices eliminates the need for a relational database. Instead, components in oCIS required to store metadata will do that on their own. With Reva and no MySQL dependencies, the complexity of running ownCloud in small environments is reduced considerably – a point interesting not just to large-scale data center admins.

Up and Running

ownCloud published a technical preview of oCIS 1.0 at the end of 2020, shipping it as a Docker container and binaries. More examples on getting it running are linked in the deployment section of the GitHub repository.

The vendor's claim that getting the oCIS Docker container up and running is easy, is true, although things get complicated with EOS (see below). Docker images for oCIS are available on Docker Hub [10]; their latest tag always reflects the current master branch.

Any standard virtual machine with one of the big cloud providers or any entry server in the data center with a standard Linux distribution should be sufficient, provided the system has a container runtime installed. On openSUSE, for example,

zypper in docker

provides the needed container. One or both commands

service docker enable
service docker start

create the standard Docker environment.

Pulling the oCIS container and starting it is all that is required to get the server and ownCloud Web up and running with the default usernames (Figure 1):

# zypper in docker
# service docker start
# docker pull owncloud/ocis
# docker run --rm -ti -p 9200:9200 owncloud/ocis

That's it. oCIS is now at your service on localhost, port 9200 (http://localhost:9200). The demo accounts and passwords are einstein:relativity, marie:radioactivity, and richard:superfluidity. Admin accounts are moss:vista and admin:admin. If oCIS is run on a server with a resolvable hostname, it can request an SSL certificate from Let's Encrypt via Traefik.

oCIS Binary

An alternative to Docker is a pre-compiled binary that users – thanks to Go – can simply download from GitHub. The latest binaries from the master branch are in the testing section. Listing 1 downloads oCIS from the ownCloud download repository.

# for Mac
curl https://download.owncloud.com/ocis/ocis/1.0.0/ocis-1.0.0-darwin-amd64 --output ocis
# for Linux
curl https://download.owncloud.com/ocis/ocis/1.0.0/ocis-1.0.0-linux-amd64 --output ocis
chmod +x ocis

The binary edition of oCIS expects /var/tmp/ocis as the default storage location, but that can be changed in its configuration (Figure 2). The server can be started with ./ocis server. Basic management and extension commands are shown in Table 1. If the exit code of the health check equals zero, the service should be up and running. If the exit code is greater than zero, the service is not in a healthy state. The oCIS binary already contains some extensions that can be managed by oCIS commands.

EOS Is Complicated

If you want to follow ownCloud's recommendations to deploy oCIS along with EOS for large environments, you have to take some more steps. EOS not only adds to the amount of required hardware and increases the complexity of the whole environment, it's also a slightly bigger task to set up. To help, you will find separate and concise EOS documentation from CERN [3] and a step-by-step guide from ownCloud [11].

In a nutshell, users have to get and start EOS and oCIS containers; configure LDAP support; and kill home, user, and metadata storage before starting the EOS configuration. The Accounts service needs to be set up to work with EOS, too. All of these steps are docker compose commands, which are covered in the documentation.

The Storage backends | EOS page in the oCIS documentation also provides information on verification and troubleshooting and has a command reference for the built-in EOS shell. Said documentation also gives an impression into why dealing with oCIS is likely interesting only for large oCIS environments.

Data Paths

Although it is relatively easy to imagine the flow of data through ownCloud 10 with Apache and PHP, the great number of components in oCIS makes imagining the path of data more difficult. ownCloud therefore provides a diagram that explains the flow of data in detail [12]. Before working with oCIS for the first time, you should have a look at that document for a better understanding of data flow.

Initially, a file coming from a client will hit the oCIS proxy component, which is part of Traefik and ensures smooth load balancing across the different instances of oCIS components. The incoming request – in this example, a file upload – is then forwarded to one of the available Reva storage front ends.

From the administrator's point of view, it will look as if data disappears at this point. Reva comprises a number of microservices itself: a gateway for incoming requests, a DAV-speaking component that initially deals with incoming requests, and its own registry. The DAV component supplies the oCIS proxy with all relevant information for placing a file in the environment. Once Reva has successfully determined where the file needs to go, it reports that information back to the oCIS proxy, which finally puts the file into the desired location.

What looks like a side note here is in fact something very important to note: Just like ownCloud 10, oCIS continues to use the DAV interface for external clients. Therefore, any existing clients for ownCloud, including the project's very own iOS and Android applications, will continue to work with oCIS. It also allows ownCloud to create a bridge mode between ownCloud 10 and oCIS, making migrations from one system to the other much easier.

oCIS Configuration

Administrators familiar with ownCloud 10 or earlier are used to the tedious configuration process, because it is an application running inside a web server with PHP. Parts of the settings required affect PHP directly and must go into the PHP configuration file. Other settings are Apache-related and must be in the web server's configuration file (even there, you have different options, e.g., the VirtualHost or httpd configuration). Moreover, some ownCloud-specific configuration files define parameters such as the database connection to use. Tuning ownCloud 10 to the maximum accordingly requires a number of configuration file changes. Because these settings can be different on different Linux distributions, automating ownCloud has been a tedious task in the past.

oCIS changes the scene. Because ownCloud no longer requires Apache or PHP, their configuration files are absent. Instead, you are expected to use the ocis binary as the central command to control behavior. Understanding the power of that tool is tricky at first glance, though, so a closer look is necessary.

One Binary to Rule Them All

In oCIS, ocis is the universal binary. It starts the microcomponents that make up the oCIS core and the ownCloud Web interface. Additionally, it runs the components that allow oCIS to connect to EOS, if EOS is part of the setup. The use of microcomponents would be violated if oCIS were a huge binary incorporating all that functionality. Therefore, ocis is a skeleton that loads most of the functionality by starting the respective extensions. For instance, the ownCloud Web interface for the Vue.js-based GUI is such an extension. Consequently, for every extension oCIS can handle, you control the component's behavior either directly from the command line or by means of a configuration file.

If you decide to deploy configuration files, oCIS lets you choose between the YAML and JSON formats. A single file in /etc/ocis controls the behavior of the oCIS core and the ocis command itself. The individual extensions typically use their own file for determining configuration settings, but the details are left for the authors of the extensions to define. What is elegant though is that oCIS supports sharing configuration files between the host and the individual oCIS containers. Files in /etc/ocis will be present in any oCIS-related container, allowing for a seamless automation of oCIS setups.

A Massive Change in Handling

Overall, the introduction of the ocis CLI binary to control any and all aspects of ownCloud's behavior is by far the most invasive change for experienced ownCloud administrators and will make adapting to oCIS a bit more challenging – particularly for those who have to maintain ownCloud 10 and oCIS setups in parallel (for compatibility reasons or different use cases in the same company). If you are in this camp, you will have to cope with two very different configuration approaches until ownCloud 10 is retired completely.

Monitoring

Massively scalable environments like ownCloud bring new challenges in terms of monitoring. In conventional systems, monitoring usually refers to event monitoring only. Accordingly, the question a typical event monitoring environment will try to answer is whether a certain service at a certain point in time performs according to the desired state.

For scale-out solutions such as ownCloud, that is simply not enough. In fact, the status of an individual component of the setup doesn't matter that much. Microservice approaches are inherently redundant – if one service of a certain type fails, the other services automatically take over that component's workload (gRPC as well as Traefik allow oCIS to follow this principle easily). What is much more relevant to the administrator is knowing ahead of time when additional storage space must be added and when additional servers for oCIS core components need to be commissioned.

This challenge is not exactly new, environments such as Kubernetes or OpenStack have paved the way already and forced developers to come up with mighty solutions. Hence, monitoring usually is just one aspect of what administrators call monitoring, alerting, and trending (MAT). Solutions for MAT in scalable environments exist; Prometheus is one of the most prominent alongside InfluxDB. In oCIS, ownCloud developers add an interface for Prometheus to read out the most important metric data during normal operations.

Prometheus to the Rescue

Prometheus itself is a well-established standard by now. By following the pull principle, other components must make their metric data available through a pre-defined API interface to be fetched by Prometheus. If you enable the Prometheus interface in oCIS, every oCIS instance will allow the Prometheus instances of a setup to fetch metric data from port 8001 through the oCIS debug endpoint.

Available metrics include a number of different values related to Go, different memory statistics, threads in use, and more of the like. Additionally, ownCloud recommends deploying Telegraf on oCIS servers for general-purpose system metrics, such as disk space in use, CPU usage, or other vital parameters. Combined with the Jaeger client, it even becomes possible to do in-depth performance debugging of individual oCIS components in case some parts of the environment do not live up to expectations.

Long story short, MAT functionality is now at the core of oCIS and much more easily available than it was in previous ownCloud versions. Admins should absolutely leverage that new functionality.

Tough Call and a Risk to Take

The technical aspects of oCIS that are visible are already nothing short of spectacular. The software is very easily installed, and its completely redesigned components are faster than its predecessor and offer much better scalability. The modular design with microservices and APIs, even for extensions, looks promising. The biggest risk for ownCloud appears to be that, while taking big technological steps, they might loose a significant part of their PHP-based community. Community management in terms of Go and Vue.js (and EOS) is surely a big task for the company now and in the years to come.