A Brief History of Containers: From the 1970s Till Now
When we first published this blog post in 2017, the technology landscape for containers was quite different than it is today. Over the past two years, we have seen significant changes take place that affected, and continue to affect how Containers are adopted. As we enter the new decade, we want to recap the changes and developments that we saw and offer our view of where we believe Containers are heading to in 2020.
For this trip, step into my DeLorean time machine, and let’s journey to 1979, when the concept of containers first emerged.
1979: Unix V7
Note to reader, yes, I was less than 10 years old at the time. During the development of Unix V7 in 1979, the chroot system call was introduced, changing the root directory of a process and its children to a new location in the filesystem. This advance was the beginning process isolation: segregating file access for each process. Chroot was added to BSD in 1982.
2000: FreeBSD Jails
Flash-forward nearly two decades later to 2000, when a small shared-environment hosting provider came up with FreeBSD jails to achieve clear-cut separation between its services and those of its customers for security and ease of administration. FreeBSD Jails allows administrators to partition a FreeBSD computer system into several independent, smaller systems – called “jails” – with the ability to assign an IP address for each system and configuration.
2001: Linux VServer
Like FreeBSD Jails, Linux VServer is a jail mechanism that can partition resources (file systems, network addresses, memory) on a computer system. Introduced in 2001, this operating system virtualization that is implemented by patching the Linux kernel. Experimental patches are still available, but the last stable patch was released in 2006.
2004: Solaris Containers
In 2004, the first public beta of Solaris Containers was released that combines system resource controls and boundary separation provided by zones, which were able to leverage features like snapshots and cloning from ZFS.
2005: Open VZ (Open Virtuzzo)
This is an operating system-level virtualization technology for Linux which uses a patched Linux kernel for virtualization, isolation, resource management and checkpointing. The code was not released as part of the official Linux kernel.
2006: Process Containers
Process Containers (launched by Google in 2006) was designed for limiting, accounting and isolating resource usage (CPU, memory, disk I/O, network) of a collection of processes. It was renamed “Control Groups (cgroups)” a year later and eventually merged to Linux kernel 2.6.24.
LXC (LinuX Containers) was the first, most complete implementation of Linux container manager. It was implemented in 2008 using cgroups and Linux namespaces, and it works on a single Linux kernel without requiring any patches.
CloudFoundry started Warden in 2011, using LXC in the early stages and later replacing it with its own implementation. Warden can isolate environments on any operating system, running as a daemon and providing an API for container management. It developed a client-server model to manage a collection of containers across multiple hosts, and Warden includes a service to manage cgroups, namespaces and the process life cycle.
Let Me Contain That For You (LMCTFY) kicked off in 2013 as an open-source version of Google's container stack, providing Linux application containers. Applications can be made “container aware,” creating and managing their own subcontainers. Active deployment in LMCTFY stopped in 2015 after Google started contributing core LMCTFY concepts to libcontainer, which is now part of the Open Container Foundation.
When Docker emerged in 2013, containers exploded in popularity. It’s no coincidence the growth of Docker and container use goes hand-in-hand.
Just as Warden did, Docker also used LXC in its initial stages and later replaced that container manager with its own library, libcontainer. But there’s no doubt that Docker separated itself from the pack by offering an entire ecosystem for container management.
2016: The Importance of Container Security Is Revealed
With the wide adoption of container-based applications, systems became more complex and risk increased, laying the groundwork for container security. Vulnerabilities like dirty COW only furthered this thinking. This led to a shift left in security along the software development lifecycle, making it a key part of each stage in container app development, also known as DevSecOps. The goal is to build secure containers from the ground up without reducing time to market.
2017: Container Tools Become Mature
Hundreds of tools have been developed to make container management easier. While these types of tools have been around for years, 2017 is the year that many of them earned their stripes. Just look at Kubernetes; since its adoption into the Cloud Native Computing Foundation (CNCF) in 2016, VMWare, Azure, AWS, and even Docker have announced their support, on top of their infrastructures.
While the marketplace is still growing, some tools have come to define certain functions in the container community. Ceph and REX-Ray set standards for container storage, while Flannel connects millions of containers across datacenters. And in CI/CD, Jenkins is completely changing the way we build and deploy apps.
Adoption of rkt and Containerd by CNCF
The container ecosystem is unique as it is powered by a community-wide effort and commitment to open source projects. Docker’s donation of the Containerd project to the CNCF in 2017 is emblematic of this concept, as well as the CNCF's adoption of the rkt (pronounced "rocket") container runtime around the same time. This has led to greater collaboration between projects, more choices for users, and a community centered around improving the container ecosystem.
Kubernetes Grows Up
In 2017 the open-source project demonstrated great strides towards becoming a more mature technology. Kubernetes supports increasingly complex classes of applications - enabling enterprise transition to both hybrid cloud and microservices. At DockerCon in Copenhagen, Docker announced they will support the Kubernetes container orchestrator, and Azure and AWS fell in line, with AKS (Azure Kubernetes Service) and EKS, a Kubernetes service to rival proprietary ECS. It was also the first project adopted by the CNCF and commands a growing list of third-party system integration service providers.
2018: The Gold Standard
2018 saw containerization become the foundation for modern software infrastructure and Kubernetes being used for most enterprise container projects. In 2018, the Kubernetes project on GitHub had over 1500 contributors, with one of the most significant open source communities, with over 27000 stars. The massive adoption of Kubernetes pushed cloud vendors such as AWS, Google with GKE (Google Kubernetes Engine), Azure, and Oracle with Container Engine for Kubernetes, to offer managed Kubernetes services. Furthermore, leading software vendors such as VMWare, RedHat, and Rancher started offering Kubernetes-based management platforms.
Infrastructure provider VMware moved toward adopting Kubernetes when in late 2018 it announced that it was acquiring Heptio, a consultant firm that helps enterprises deploy and manage Kubernetes.
We also witnessed emerging hybrid technologies that combine VM-like isolation with container speed. Open source projects such as Kata containers, gVisor, and Nabla attempt to provide secured container runtimes with lightweight virtual machines that perform the same way container do, but provide stronger workload isolation.
2019: A Shifting Landscape
The year ushered in significant changes in the containers landscape. New runtime engines now started replacing the Docker runtime engine, most notably containerd, an open source container runtime engine, and CRI-O, a lightweight runtime engine for Kubernetes.
In 2019 we saw a tectonic shift take place in the containers landscape, when Docker Enterprise was acquired and split off, resulting in Docker Swarm being put on a 2-year end-of-life horizon. At the same time, we witnessed the decline in popularity of the rkt container engine, while officially still part of the CNCF stable.
VMware doubled down on its commitment to Kubernetes by first acquiring Heptio and then Pivotal Software (with both PAS and PKS). The move is intended to offer enterprises the ability to take advantage of cloud-like capabilities of cloud native deployments in their on-premise environments.
Last year we also saw advancements in adoption of serverless technology with platforms such as Knative, a Kubernetes-based serverless workloads management platform, gaining traction with organizations.
2019 saw the launch of Kubernetes-based hybrid-cloud solutions such as IBM Cloud Paks , Google Anthos , Aws Outposts, and Azure Arc. These cloud platforms blur the traditional lines between cloud and on-prem environments , as you can now manage on-prem and single-vendor cloud clusters.
We believe that the emergence of that these new capabilities represent the next step in the evolution of Kubernetes. New cloud capabilities such as Anthos, Arc and Outposts point toward the future of hyper abstraction, where the compute resources are decoupled from the management layer, similar to how Kubernetes works, which decouples the workloads from where they are managed.
In today's Kubernetes, the master node sits in the same physical cluster as the worker node. With hyper abstraction the workload management plane will manage workloads in nodes that can be distributed among several compute infrastructures, and the user will not know or care where they physically run.
Judging by the record high 12,000+ attendees at KubeCon San Diego, going forward we believe that Kubernetes will become the standard management platform for containers, virtual machines and other cloud native workloads.