Intro
I’ve been playing with Project Atomic as a platform to run Docker containers for some time now. The reason that I like Project Atomic is something for another blogpost. But one of the reasons is that while it’s a minimal OS, it comes with Python so I can use Ansible to do orchestration and configuration management.
Now, running Docker containers on a single host is nice, but the real fun starts when you can run containers spread over a number of hosts. This is easier said than done and requires some extra services like a scheduler, service discovery, overlay networking,… There are several solutions, but one of the most popular these days is Kubernetes.
ProjectAtomic happens to ship with all necessary pieces needed to deploy a Kubernetes cluster using Flannel for the overlay networking. The only thing left is the configuration. Now this happens to be something Ansible is particularry good at.
The following wil describe how you can deploy a 4 node cluster on top of Atomic hosts using Ansible. Let’s start with the Ansible inventory.
Inventory
We will keep things simple here by using a single file-based inventory file where we explicitly specify the ip adresses of the hosts for testing purposes. The important part here are the 2 groups k8s-nodes and k8s-master. The k8s-master group should contain only one host which will become the cluster manager. All nodes under k8s-nodes will become nodes to run containers on.
[k8s-nodes]
atomic02 ansible_ssh_host=10.0.0.2
atomic03 ansible_ssh_host=10.0.0.3
atomic04 ansible_ssh_host=10.0.0.4
[k8s-master]
atomic01 ansible_ssh_host=10.0.0.1
Variables
Currently these roles don’t have many variables that can be configured but we
need to provide the variables for the k8s-nodes group. Create a folder
group_vars with a file that has the same name of the group. If you checked out
the repository you already have it.
$ tree group_vars/
group_vars/
└── k8s-nodes
The file should have following variables defined.
skydns_enable: true
# IP address of the DNS server.
# Kubernetes will create a pod with several containers, serving as the DNS
# server and expose it under this IP address. The IP address must be from
# the range specified as kube_service_addresses.
# And this is the IP address you should use as address of the DNS server
# in your containers.
dns_server: 10.254.0.10
dns_domain: kubernetes.local
Playbook
Now that we have our inventory we can create our playbook. First we configure the k8s master node. Once this is configured we can configure the k8s nodes.
deploy_k8s.yml
---
- name: Deploy k8s Master
hosts: k8s-master
remote_user: centos
become: true
roles:
- k8s-master
- name: Deploy k8s Nodes
hosts: k8s-nodes
remote_user: centos
become: true
roles:
- k8s-nodes
Run the playbook.
$ ansible-playbook -i hosts deploy_k8s.yml
If all ran without errors you should have your kubernetes cluster running. Lets see if we can connect to it.
$ kubectl --server=192.168.124.40:8080 get nodes
NAME STATUS AGE
192.168.124.166 Ready 20s
192.168.124.55 Ready 20s
192.168.124.62 Ready 19s
That looks good. Lets see if we can run a container on this cluster.
$ kubectl --server=192.168.124.40:8080 run nginx --image=nginx
replicationcontroller "nginx" created
Check the status:
$ kubectl --server=192.168.124.40:8080 get pods
NAME READY STATUS RESTARTS AGE
nginx-ri1dq 0/1 Pending 0 55s
If you see the pod status in state pending, just wait a few moments. If this is the first time you run the nginx container image, it needs to be downloaded first which can take some time. Once your pod is is running you can try to enter the container.
$ kubectl --server=192.168.124.40:8080 exec -ti nginx-ri1dq -- bash
root@nginx-ri1dq:/#
This a rather basic setup (no HA masters, no auth, etc..). The idea is to improve these Ansible roles and add more advanced configuration.
If you are interested and want to try it out yourself you can find the source here: