Kubernetes on Mesos

Kubernetes is a decentralized architecture solution based on container technology and is the result of more than decade’s worth of Google’s extensive use of container technology. In July of this year, mesosphere announced a partnership with Google to combine Kubernetes with mesos to enable users to use Kubernetes with other leading datacenter services such as Hadoop, Spark and Chronos. This allows the Kubernetes application to execute in conjunction with other types of applications on the same set of servers, while Mesos ensure that resources are fairly distributed and isolates each application.

Kubernetes -Mesos is currently in alpha and still under development, so it is not recommended for production  environments.

Kubernetes-Mesos Architecture

An Apache Mesos cluster consists of one or more Master and one or more Slaves, while Kubernetes-Mesos (K8sm) is a Mesos Framework that executes on top of Mesos. K8sm provides two components and connects Mesos with Kubernetes:

1. Scheduler: Integrates the Kubernetes scheduling API with the Mesos scheduler runtime.
2. Executor: Integrates the Kubernetes kubelet service with the Mesos executor runtime.

When a pod is created through Kubernetes, the K8sm scheduler will create a related Mesos task and queue it for scheduling. It will be allocated to the appropriate node depending on the resources required by the pod/task. The pod/task will then be launched and passed to the executor. When the executor launches the pod/tasl, it registers the pod with the kubelet and starts managing the lifecycle of the pod.

Installing Kubernetes on Mesos

This article will guide you through installing Kubernetes on a Mesos cluster. It provides step-by-step instructions to add Kubernetes to the Mesos cluster and execute the pod of the first nginx web server.

Preparations

Prepare a Mesos cluster environment

Select one of the clusters as the Kubernetes master node. The following packages are required:

Go (For installing the Go language version please refer to Kubernetes Development Guide)

make (eg build-example)

Docker needs to be installed on every node

You can deploy Kubernetes-Mesos to the same node as Mesos master or on different nodes.

Deploying Kubernetes-Mesos

First, select the node you want to install Kubernetes-Mesos on and  build Kubernetes-Mesos

$ git clone https://github.com/kubernetes-incubator/kube-mesos-framework
$ cd kube-mesos-framework
$ make

Set some environment variables

$ export KUBERNETES_MASTER_IP=172.22.132.22
$ export KUBERNETES_MASTER=http://${KUBERNETES_MASTER_IP}:8888

Deploying etcd

Start etcd and verify that it has executed

$ docker run -d --hostname $(uname -n) --name etcd \
-p 4001:4001 -p 7001:7001 quay.io/coreos/etcd:v2.2.1 \
--listen-client-urls http://0.0.0.0:4001 \
--advertise-client-urls http://${KUBERNETES_MASTER_IP}:4001
$ docker ps
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                                                           NAMES
e2afe13e2319        quay.io/coreos/etcd:v2.2.1   "/etcd --listen-cl..."   3 days ago          Up 3 days           0.0.0.0:4001->4001/tcp, 2379-2380/tcp, 0.0.0.0:7001->7001/tcp   etcd

The following method is also a good way to test if your etcd is working properly

$ curl -L http://${KUBERNETES_MASTER_IP}:4001/v2/keys/

If the connection is normal, you will see the output of etcd on the console

Starting the Kubernetes-Mesos service

Updating your Path will make it easier to execute the Kubernetes-Mesos binaries:

$ export PATH="$(pwd)/_output/local/go/bin:$PATH"

Check your Mesos Master, depending on how Mesos was installed, the host:port could be mesos-master:5050 or the ZooKeeper URL could be zk://zookeeper:2181/mesos:

$ epxort MESOS_MASTER=<host:port or zk://url>

In order for Kubernetes to work properly when Mesos-Master changes, it is recommended to use the ZooKeeper URL in a  production environment.

Create a cloud config named mesos-cloud.conf in the current directory and enter the following:

$ cat <<EOF >mesos-cloud.conf
[mesos-cloud]
mesos-master        = ${MESOS_MASTER}
EOF

Now start the kubernetes-mesos API server, controller manager, and scheduler on the master node:

$ km apiserver \
--address=${KUBERNETES_MASTER_IP} \
--etcd-servers=http://${KUBERNETES_MASTER_IP}:4001 \
--service-cluster-ip-range=10.10.10.0/24 \
--port=8888 \
--cloud-provider=mesos \
--cloud-config=mesos-cloud.conf \
--secure-port=0 \
--v=1 >apiserver.log 2>&1 &
$ km controller-manager \
--master=${KUBERNETES_MASTER} \
--cloud-provider=mesos \
--cloud-config=./mesos-cloud.conf  \
--v=1 >controller.log 2>&1 &
$ km scheduler \
--address=${KUBERNETES_MASTER_IP} \
--mesos-master=${MESOS_MASTER} \
--etcd-servers=http://${KUBERNETES_MASTER_IP}:4001 \
--mesos-user=root \
--api-servers=${KUBERNETES_MASTER} \
--cluster-dns=10.10.10.10 \
--cluster-domain=cluster.local \
--v=2 >scheduler.log 2>&1 &

These services will keep running in the background, if you want to terminate when you log out, input the following:

$ disown -a

Verifying the KM Service

Interacting with the kubernetes-mesos framework via kubectl:

$ kubectl get pods
NAME      READY     STATUS    RESTARTS   AGEs
$ kubectl get services
NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
k8sm-scheduler   ClusterIP   10.10.10.158   <none>        10251/TCP   2d
kubernetes       ClusterIP   10.10.10.1     <none>        443/TCP     3d

Executing a POD

Create and edit a pod’s yaml file:

$ cat <<EOPOD >nginx.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
EOPOD

Use kubectl to build a nginx pod:

$ kubectl create -f ./nginx.yaml
pod "nginx" created

Use kubectl to observe the status of the pod:

$ kubectl get pods
NAME      READY     STATUS    RESTARTS   AGE
nginx     1/1       Running   0          8m

Reference Material:

1. Kubernetes Docs – Kubernetes on Mesos
2. Kubernetes incubator Github
3. Mesosphere Github

撰文： 白凱仁 迎棧科技軟體工程師