Skip to content

Latest commit

 

History

History
297 lines (210 loc) · 5.76 KB

mesos.md

File metadata and controls

297 lines (210 loc) · 5.76 KB

Mesos

Obsolete. Use Kubernetes instead.

Written in C++ with Java and Python bindings

Key Points

  • Marathon - task launcher, uses Docker
  • Chronos - cron (highly available, fault-tolerant)
  • Mesos Consul - Docker image by Cisco that registers all services with Consul
  • can use Docker containers via Marathon

Ports

Port Description
5050 Master UI
5051 Slave
8080 Cronos UI / REST API

HA

Each HA Master UI redirects to active master, so each master uses both a leader and detector to ZooKeeper

Industrial Light and Magic used Consul because Mesos DNS couldn't easily register docker containers outside Mesos cluster

  • resource isolation using Cgroups and Docker
  • Slave => resource offer (CPU, RAM, disk) => Master
  • Master => resource offer => Scheduled App
  • Allocation Module (pluggable) decides which App Framework to offer resources to

Security

  • Framework auth 0.15+ (basic shared secrets)
  • Slave auth 0.19+
  • SSL only protects Scheduler <-> Master data
  • must encrypt Task data yourself
  • future - HTTP Web UI auth

configure --with-sasl=/path/to/sasl2

master:

--authenticators  # default: crammd5
--credentials  # text/json file
--authenticate  # for frameworks
--authenticate_slaves

slave:

--authenticatee  # default: crammd5
--credential  # text/json file

Setup

brew install autoconf automake libtool subversion maven

Not needed in newer versions of Mesos, mesos 0.21 failed to compile anyway

./configure --with-svn=/usr/local/Cellar/subversion/1.8.13
./configure

Compile with SSL:

./configure --enable-libevent --enable-ssl

Must also use:

SSL_ENABLED=1 SSL_KEY_FILE=key SSL_CERT_FILE=cert opt-flags binary

SSL_ENABLED=1 SSL_KEY_FILE=key SSL_CERT_FILE=cert SSL_REQUIRE_CERT=1 SSL_ENABLE_TLS_V1_1=true master.sh
make
make check

(optional)

make install

Mesos CLI

pip install mesos.cli
mesos

From Mesos build:

mesos.sh # wrapper for Mesos commands
mesos-local.sh # run local mode pseudo cluster
mesos-tests.sh # run test suite
mesos-daemon.sh
mesos-start-cluster.sh
mesos-start-masters.sh
mesos-start-slaves.sh

list of hosts, uses SSH like Hadoop for mesos-start-cluster.sh etc scripts

$MESOS_HOME/var/mesos/deploy/masters
$MESOS_HOME/var/mesos/deploy/slaves

Can set any MESOS_<OPTION_NAME> env var:

export MESOS_MASTER # must specify port to prevent parsing localhost error

in .bashrc:

export MESOS_MASTER=localhost:5050

Use full path otherwise uses /usr/local/bin/mesos => /usr/local/Cellar/mesos/0.23.0/bin/mesos

Brew version (0.23)

mesos master --work_dir=/var/lib/mesos --log_dir=/tmp/mesos-master-logs --cluster=myCluster # --ip=127.0.0.1 # localhost fails to parse
mesos slave --master=127.0.0.1:5050 --log_dir=/tmp/mesos-slave-logs

Build version

--log_dir=$MESOS_HOME/logs # but couldn't find this in /usr/local/Cellar/mesos/0.23 nor /usr/local/mesos-0.24

$MESOS_HOME/src/mesos-master --work_dir=/var/lib/mesos --log_dir=/tmp/mesos-master-logs --cluster=myCluster
--authenticate        # for frameworks
--authenticate-slave
--ip --advertise_ip # if different to allow frameworks to connect

$MESOS_HOME/src/mesos-slave --log_dir=/tmp/mesos-slave-logs
--master=localhost:5050 # causes duplicate --master switch error when MESOS_MASTER defined
--attributes=rack:1,dc:2 --credentials=file:///path/to/file # where file contains "<username> <password>" or json '{ "principal"="hari", "secret"="blah" }'
--work-dir=/tmp/mesos # default

For HA Masters:

export ZKLIST=host1:2181,host2:2181,host3:2181
export ZKLIST=localhost:2181
$MESOS_HOME/src/mesos-master --work_dir=/var/lib/mesos --log_dir=/tmp/mesos-master-logs --cluster=myCluster --zk=zk://$ZKLIST/mesos
$MESOS_HOME/src/mesos-slave  --master=zk://$ZKLIST/mesos --log_dir=/tmp/mesos-master-logs

Test frameworks to check Mesos is working

C++ - failed with TASK_LOST

$MESOS_HOME/src/test-framework --master=$MESOS_MASTER

Java (had to unset MESOS_NATIVE_LIBRARY):

$MESOS_HOME/src/examples/java/test-framework $MESOS_MASTER

Python:

$MESOS_HOME/src/examples/python/test-framework $MESOS_MASTER

Chronos

  • distributed Cron
  • needs ZooKeeper
  • UI port 8080 + REST API
zkServer.sh start
mesos-master --work_dir=/var/lib/mesos --log_dir=/tmp/mesos-master-logs --cluster=myCluster --ip=127.0.0.1 --zk=zk://localhost:2181/mesos --quorum=1
mesos-slave --log_dir=/tmp/mesos-slave-logs
git clone https://github.com/mesos/chronos.git
export MESOS_NATIVE_LIBRARY # (set in .bashrc based on Mac/Linux)

on Mac:

export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/mesos/src/.libs/libmesos.dylib

on Linux (check this path):

export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/mesos/lib/libmesos.so
mvn package
java -cp target/chronos*.jar org.apache.mesos.chronos.scheduler.Main --master zk://localhost:2181/mesos

Used by BlackRock (talk 24/9/2015)

Mesos + Kubernetes on Mesos works but partial features

Ported from private Knowledge Base page 2015+