In the following tutorial we will show how to quickly boot an OpenShift Origin multinode deployment on Amazon AWS using CloudFormation and Ansible. We found the reference architecture had too many additional dependencies like Lambda, Route53, etc and we wanted to build a simple deployment procedure in a similar fashion to Heptio’s Kubernetes AWS quick start.
Introduction
Kubernetes has had an spectacular market penetration in the container space during 2017. And OpenShift is just the icing on the cake. There are a ton of interesting features that both the OSS oriented (Origin) and commercial versions of OpenShift add on top of vanilla Kubernetes like CI/CD workflows, Docker images registry, etc.. you can read more about How Does OpenShift Extend Kubernetes in its own homepage. Here at Sysdig we regularly spin up and destroy test scenarios for demo environments, testing and development with different orchestration tools like Kubernetes, Docker Swarm, DC/OS Mesos and of course, OpenShift too. For us, the perfect environment requires to hit a complexity sweet spot: we need a few nodes for high-availability scenarios but at the same time we want something that we can frequently spawn in our cloud without worrying too much about complex deployment, resource use / cost, where anyone from our team can access. We looked at minishift, wonderful for developers, but would be too simple (single node and intended to be run locally in your laptop) for the case in point. Red Hat AWS Quick Start is really well designed, but too complex and with many additional dependencies for our purposes. So we needed something in the middle, that’s why we created our own OpenShift deployment on AWS scripts, and we hope you can reuse some of it! How to deploy #OpenShift cluster on #AWS using CloudFormation and #Ansible Click to tweetUsing CloudFormation to create the OpenShift infrastructure on AWS
Manually creating your AWS resources doesn’t work for disposable environments, seriously, don’t do it, it is:- Prone to human error
- Tedious, repetitive task
- It’s extremely easy to forget to delete some of the cloud entities, littering your cloud account
- 1 CentOS-based master node
- 2 CentOS-based worker or minion nodes
- A separate VPC and subnets to isolate our environment from any other entities (and also making the scenario clean-up easy and safe)
- Security groups that will open the following public ports:
- 22 SSH for all host
- 8443 for the OpenShift Web console, master node
- 10250 master proxy to node hosts, master node
- Instead of defining two identical nodes / volumes / etc use LaunchConfiguration and Auto Scaling group, so you just define the entities once and can spawn any number of master / worker nodes using parameters. We didn’t want our clusters to grow automatically, so we didn’t proceed with this.
- Deploying a bastion host to tunnel SSH connections, like the Heptio’s Kubernetes QuickStart.
aws cloudformation create-stack
--region <deployment_region>
--stack-name <stack_name>
--template-url "https://<cloudformation_file_url>"
--parameters
ParameterKey=AvailabilityZone,ParameterValue=<AvailabilityZone>
ParameterKey=KeyName,ParameterValue=<infrastructure_aws_key_name>
--capabilities=CAPABILITY_IAM
Installing OpenShift Origin on CentOS
Now we are ready to install OpenShift. We decided to install the community version, known as “Origin” as these clusters are intended for non production usage. The project provides a set of Ansible playbooks to automate installation: openshift-ansible and we will be using the 3.6 branch in this example. First, you need to create an Ansible hosts inventory file, the version we used looks like this:
File: AnsibleOpenshiftSysdighost
--------------------------------
[OSEv3:children]
masters
etcd
nodes
[OSEv3:vars]
ansible_ssh_user=centos
ansible_sudo=true
ansible_become=true
deployment_type=origin
os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'
openshift_install_examples=true
openshift_docker_options='--selinux-enabled --insecure-registry 172.30.0.0/16'
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/openshift/openshift-passwd'}]
openshift_disable_check=disk_availability,docker_storage,memory_availability
[masters]
master-host-name
[etcd]
master-host-name
[nodes]
master-host-name openshift_node_labels="{'region':'infra','zone':'east'}" openshift_schedulable=true"
worker1-host-name openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
worker2-host-name openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
- The master node is marked as region infra and also as schedulable, this way we can run some special pods (like the router and Docker registry) without needing a dedicated infrastructure host.
- Httpasswd auth is enabled using the file /etc/openshift/openshift-passwd in order to require authentication to access the OpenShift Web Console.
- In this playbook we disable several pre-flight checks with the openshift_disable_check config key, this way we can use modest AWS instaces (default requisites are actually pretty high for non-production purposes).
File: prepare.yml
-----------------
---
- hosts: nodes
gather_facts: no
pre_tasks:
- name: 'install python2'
raw: sudo yum install -y python
tasks:
- name: upgrade packages
yum: state=latest name={{ item }}
with_items:
- docker
- NetworkManager
- name: enable network-manager
shell: systemctl enable NetworkManager && systemctl start NetworkManager
- name: docker storage conf file
copy:
content: "DEVS=/dev/xvdbnVG=docker-vgn"
dest: /etc/sysconfig/docker-storage-setup
- name: docker-storage-setup
shell: docker-storage-setup
- name: enable docker
shell: systemctl enable docker && systemctl start docker
ansible-playbook prepare.yml -i <your_hostfile> --key-file <your_infra_ssh_key>
git clone https://github.com/openshift/openshift-ansible.git
cd openshift-ansible
git checkout origin/release-3.6
ansible-playbook -c paramiko -i <your_hostfile> openshift-ansible/playbooks/byo/config.yml --key-file <your_infrastructure_ssh_key>
OpenShift cluster post installation configuration
Once the installation playbook is finally completed, you can directly SSH in the host you designated as master and check your cluster’s status.[centos@ip-10-0-0-7 ~]$ oc get nodes
NAME STATUS AGE VERSION
ip-10-0-0-4.ec2.internal Ready 11d v1.6.1+5115d708d7
ip-10-0-0-7.ec2.internal Ready 11d v1.6.1+5115d708d7
ip-10-0-0-9.ec2.internal Ready 11d v1.6.1+5115d708d7
sudo htpasswd -b /etc/openshift/openshift-passwd admin <your_pass>
Monitoring and secure OpenShift cluster with Sysdig Container Intelligence Platform
As a bonus point, like in all the cluster deployments we do, we will install the Sysdig agent following the OpenShift specific instructions. This agent connect the OpenShift cluster to the Sysdig platform (Sysdig Monitor and Sysdig Secure). First create a project, an account and give it the required access level:oc new-project sysdigcloud
oc create serviceaccount sysdigcloud
oc adm policy add-cluster-role-to-user cluster-reader system:serviceaccount:sysdigcloud:sysdigcloud
oc adm policy add-scc-to-user privileged system:serviceaccount:sysdigcloud:sysdigcloud
oc create -f sysdigcloud_daemonset.yaml
$ oc get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
sysdig-agent-pdxrw 1/1 Running 0 1d 10.0.0.9 ip-10-0-0-9.ec2.internal
sysdig-agent-pzwdf 1/1 Running 0 1d 10.0.0.7 ip-10-0-0-7.ec2.internal
sysdig-agent-rzpvj 1/1 Running 0 1d 10.0.0.4 ip-10-0-0-4.ec2.internal
…and implementing security policies for your new OpenShift environment!
Hopefully you find this deployment method useful and you can reuse and customize some of the code snippets we used to get started with this OpenShift cluster.