Monday, November 06, 2017

Run Big Data Apps on Containers with Mesosphere


Apache Mesos was created in 2009 at UC Berkeley. Designed to run large scale webapps like Twitter, Uber, etc. It can scale upto 10,000s of nodes and supports Docker Containers.

Mesos is a distributed OS kernel:

  • Two level resource scheduling
  • Launch tasks across the cluster
  • Communication between tasks (like IPC)
  • APIs for building “native” applications (aka frameworks): program against the datacenter
  • APIs in C++, Python, JVM-languages, Go and counting
  • Pluggable CPU, memory, IO isolation
  • Multi-tenant workloads
  • Failure detection & Easy failover and HA

Mesos is a multi-framework platform solution: weighted fair sharing, roles, etc. Runs Docker containers alongside other popular frameworks e.g. Spark, Rails, Hadoop, Allows users to run regular services and batch apps in the same cluster. Mesos has advanced scheduling: resources, constraints, global view of resources, which is designed for HA and self-healing.

Mesos is now a proven at scale, battle-tested in production running the biggest of the web apps. 

1 comment:

Sita said...
This comment has been removed by a blog administrator.