Apache Mesos

Apache Mesos
Developer(s) Apache Software Foundation
Stable release 0.22.0 / March 25, 2015
Development status Active
Written in C++
Operating system Cross-platform
Type Cluster management software
License Apache License 2.0
Website mesos.apache.org

Apache Mesos is an open-source cluster manager that was developed at the University of California, Berkeley. It "provides efficient resource isolation and sharing across distributed applications, or frameworks".[1] The software enables resource sharing in a fine-grained manner, improving cluster utilization.

Since being developed at UC Berkeley, it has been adopted by several large software companies, including Twitter, Airbnb and Apple. At least 50 organizations currently use Mesos.[2]

Mesos is a top level Apache project, having graduated from incubation on the 24th of July, 2013.[3]

History

Mesos began as a research project in the UC Berkeley RAD Lab by then PhD students Benjamin Hindman, Andy Konwinski, and Matei Zaharia, as well as professor Ion Stoica. The students started working together on the project as part of a course on Advanced Topics in Computer Systems taught by David Culler. It was originally named Nexus but due to a conflict with another university's project, was renamed to Mesos.[4]

Mesos was first presented in 2009 (while still named Nexus) by Andy Konwinski at HotCloud '09 in a talk accompanying the first paper published about the project, A Common Substrate for Cluster Computing. Later in 2011 it was presented in a more mature state in a talk by Matei Zaharia at the USENIX Symposium on Networked Systems Design and Implementation conference about the paper "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center" by Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, Ion Stoica.[5]

Architecture

Mesos uses a two-level scheduling mechanism where resource offers are made to frameworks (applications that run on top of Mesos). The Mesos master node decides how many resources to offer each framework, while each framework determines the resources it accepts and what application to execute on those resources. This method of resource allocation allows near-optimal data locality when sharing a cluster of nodes amongst diverse frameworks.

Competition

Mesos is often compared to the Google Omega scheduler, a highly secretive platform used internally to manage and distribute Google's services.[6]

Frameworks

Mesos is the foundation for a number of distributed systems. These distributed systems are called frameworks.

Marathon

Marathon is a popular PaaS or container orchestration system scaling to thousands of physical servers. It is fully REST based and allows canary style deploys and deployment topologies. It's written in Scala and is highly available, elastic, and distributed.[7]

Chronos

Chronos is a distributed cron like system, that's elastic and can also express dependencies between jobs. It can scale to thousands of physical servers and is, like most Mesos frameworks, fully elastic and fault tolerant.[8]

Prominent users

Social networking site Twitter began using Mesos in 2010, after (then PhD student) Benjamin Hindman gave a presentation to a group of Twitter engineers.[6]

Airbnb uses Mesos to run a number of data processing systems like Apache Hadoop and Apache Spark.[9]

Popular internet auction website eBay uses Mesos to run continuous integration on a per-developer basis. They accomplish this by using a custom Mesos plugin that allows developers to launch their own private Jenkins instance.[10]

In April 2015, it was announced that Apple service Siri is using their own Mesos framework called Jarvis.[11]

Commercial support

Software startup Mesosphere sells "the datacenter operating system", a distributed operating system, based on Apache Mesos.[12]

References

  1. ↑ "Mesos". twitter.com. Retrieved 13 January 2015.
  2. ↑ "Organizations using Mesos". mesos.apache.org. Retrieved 12 January 2015.
  3. ↑ Lester, Dave. "Mesos Graduates from Apache Incubation". twitter.com. Twitter Engineering Blog. Retrieved 12 January 2015.
  4. ↑ Zaharia, Matei. "HUG Meetup August 2010: Mesos: A Flexible Cluster Resource manager - Part 1". youtube.com. Retrieved 13 January 2015.
  5. ↑ Hindman, Benjamin; Konwinski, Andy; Zaharia, Matei; Ghodsi, Ali; Joseph, Anthony; Katz, Randy; Shenker, Scott; Stoica, Ion (2011). "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center" (PDF). NSDI 11: 22-22. Retrieved 12 January 2015.
  6. ↑ 6.0 6.1 Metz, Cade. "Return of the Borg: How Twitter Rebuilt Google’s Secret Weapon". wired.com. Wired. Retrieved 12 January 2015.
  7. ↑ "Marathon". Mesosphere.GitHub.io. Mesosphere. 2014. Retrieved 30 March 2015.
  8. ↑ "Chronos". GitHub.com. GitHub. Retrieved 30 March 2015.
  9. ↑ Harris, Derrick. "Airbnb is engineering itself into a data-driven company". gigaom.com. Retrieved 12 January 2015.
  10. ↑ The eBay PAAS Team. "Delivering eBay’s CI Solution with Apache Mesos - Part I". EbayTechBlog.com. eBay. Retrieved 12 January 2015.
  11. ↑ Harris, Derrick (2015-04-23). "Apple Details How It Rebuilt Siri on Mesos". Mesosphere.com. Mesosphere. Retrieved 2015-04-27.
  12. ↑ "The Mesosphere DCOS". mesosphere.com. Retrieved 13 January 2015.

External links