/* Title: MARKDOWN-BASED FLAT-FILE CMS PERFORMANCE COMPARISON ON LOW-POWER COMPUTERS Description: My bachelor’s thesis, Tallinn University Of Technology, 2016 Date: 2016/07/20 Category: Web hosting Keywords: Linux, RaspberryPi, TinyCoreLinux, CMS, Markdown Template: post_eng */
I am posting the abstract, introduction and summary parts as they appear in the paper, the full paper can be downloaded here.
Happy reading!
Abstract
The purpose of this work is to analyse and compare the performance of flat-file content management systems on low-power computers with various hardware specifications.
The objective is to determine whether such hardware is a viable option for hosting websites that generate web content from specifically formatted text files and find out the bottlenecks of such solutions as well as an approximate workload that these systems can handle, so that unnecessarily complex server setups can be replaced with lightweight alternatives if workload requirements are not demanding.
This work is aimed at finding out if it is reasonable to dynamically generate web pages or statically generated systems should be preferred, whether the web server software is better optimized for certain hardware architectures, as well as how the CPU speed, the number of CPU cores, RAM size or the network interface speed influence the performance of different CMS-s.
In the course of this work it was determined that both Raspberry Pi systems in question were capable of running an instance of Apache web server with a flat-file CMS on top of it. They have managed to provide response times of up to 3 seconds within the local area network for a given set of files. They could handle the load of up to 64 simultaneous requests. This proves these setups can be utilized for small working environments where complex functionality is not required, e.g. a corporate intranet web server, a personal blog or similar.
This thesis is written in English and is 73 pages long, including 7 chapters, 24 figures and 4 tables.
Introduction
According to a research [1], over 25% of all websites available on the Internet are powered by Wordpress, Joomla or other similar platforms. These traditional CMS-s usually make use of a DBMS for backend storage, all of which require additional management for security and performance reasons. In order to get the new features as they are added in new versions of these CMS-s, the system needs to be updated as well. This additional management includes, but is not limited to, performance monitoring, checking for data corruption and backing up the data itself.
Flat-file CMS-s, on the other hand, generate website content from regular text files, as long as they conform to the required syntax a particular CMS supports. Markdown [2] language allows the files to be written in a manner that is easily readable while still supporting basic text formatting. Markdown implementations [3] are also available for numerous commonly used programming languages.
Low-power systems are generally considered a niche product when it comes to website hosting. Hardware products like the Raspberry Pi [4] have made ARM-based devices available to the wider audience due to their low cost and low entry barrier. They are available in most of the countries around the world and there are a number of various operating systems that can run on them. These ARM-based products also require as little as 180mA to operate [5]. This makes them a viable computing platform for people who care about the natural environment.
Summary
The goal of this work was to analyse the performance of flat-file CMS-s that have Markdown support on low-power hardware to determine whether they are a viable option for certain use cases.
Performance metrics, such as web page load times, CPU load and RAM usage were analysed and compared on Raspberry Pi 1 Model B, Raspberry Pi 2 Model B and x86 systems. Overall system responsiveness under higher load was also examined. Raspberry Pi 1 Model B could deliver reasonable performance only when serving static files and pages. The smallest sample file loading times exceeded 1s for most CMS-s, except Singularity. CPU load was over 90% throughout the tests, average load exceeded 1.00, meaning there were operations constantly waiting for CPU time. RAM usage never exceeded 50%. Multiple concurrent requests often brought the system to a halt, sometimes crashing the Apache web server.
Raspberry Pi 2 Model B clearly takes advantage of the 4 CPU cores. In the case of Bludit, Mecha, Phile, Pico and Yellow the loading times are almost the same for 1, 2 and 4 concurrent connections to the same resource. In the case of HTMLy, PuppyCMS and Singularity the smaller sample data surprisingly meant faster loading times for 4 concurrent connections compared to 1.
Raspberry Pi 2 Model B was slower than the test x86 system in most of the cases, however, Raspberry Pi 2 managed to outperform x86 when using PuppyCMS with larger sample files. The advantages of multiple cores could also be seen during ApacheBench tests where Raspberry Pi 2 outperformed the x86 system in a lot of cases as the number of concurrent requests grew.
Results by ApacheBench or JMeter did not match the loading times reported by the Chrome web browser. In the case of PuppyCMS and Singularity, results varied by orders of magnitude, so the author made his assumptions based on real-world results from the browser. ApacheBench results were only considered in the context of measuring relative performance with various numbers of concurrent connections. The best-performing CMS turned out to be Singularity. It had the fastest loading times for both ARM-based test systems. Those never exceeded 3.1s even for the largest sample file. It managed to withstand all concurrency tests without crashing the systems and not going above 10s for page loading times according to ApacheBench. Although the results by ApacheBench cannot be considered trustworthy, they still suggest that Singularity can serve up to 64 users simultaneously.
The difference between the fastest CMS and the canonical performance of the same static files in the case of the Raspberry Pi 1 was 2-5 times, in the case of the Raspberry Pi 2, the difference was 2-10 times, depending on the sample files.
Results observed clearly show that most of the tested flat-file CMS-s are not suitable in conjunction with both Raspberry Pi models. Dynamic page loading and rendering from text files requires a lot of CPU resources which the current generation of these products can merely provide. CPU is the main system component impacting performance. With the given hardware, static web pages or static site generators should be preferred.
Performance of at least 1 flat-file CMS (Singularity) can be considered adequate. The author believes that this CMS-s can be used for small organisations where up to 64 people are accessing a given web resource simultaneously.
Results show that software used for web hosting is not optimised for ARM architecture. In nearly all cases, a single-core x86 system outperformed a quad-core ARM system with almost the same clock speed. The only cases where the Raspberry Pi 2 managed to outperform the test x86 system were situations with multiple concurrent connections, where 4 CPU cores of the Raspberry Pi 2 managed to split the workload to achieve better results.
A topic for further study could be the performance impact of using the network interface as a limiting factor. The same USB wireless adapter could be used to artificially limit the network performance to mimic the real world scenario where latency is an issue to see if overall system performance differences are distinguishable.
Another topic worth investigating would be to measure performance on more devices with a wider choice of system architectures. ARM is a fast-evolving system architecture and new devices are being manufactured constantly, to the point where it is impossible to get one’s hands on all of them.
Lastly, results achieved in this work could be compared to similar setups where source data is stored on a traditional storage device, e.g. spinning hard drive, solid-state drive, USB flash drive, external USB hard drive.