chainkindle.com

Deploying Consul and Vault as HA in Kubernetes (Part 1)

Linux Oct 08, 2019

Consul
what is consul?

Consul is an Harshicorp tool, which has the capability of providing service discovery, monitoring,segmentation with a configuration that is also distributed and highly available. It supports proxy and a quite native integration model. It as an in built proxy but also third party proxy integrations can be used with it.

As briefly mentioned Consul supports some key features. A few are listed below

  • Service Discovery: Clients been registered as a service e.g. api . It then tries discovering specific service provider with the help of DNS or HTTP.

  • Health Checking: It also supports cluster Health check monitoring

  • KV Store: It also supports key/value stores.

  • Secure Service Communication: with the help of TLS connections and certificates for services.

  • Multi Datacenter: It supports multiple datacenters.

more information can be found in the official documentation of Consul
https://www.consul.io/docs/index.html

Installation Guideline

I am running kubernetes in an openstack cloud environment, in this post. you can check my post on how to deploy kubernetes in the link below.

https://chainkindle.com/installingkubernetes/

I will be using a centos machine as my control host in this installation in kubernetes. we will be using Traefik as an ingress controller, because of its ease of deployment. Please see the link below on how to deploy traefik in your container engine

https://github.com/johnbayo/traefik.git

So as not to get confused, with the different necessary files. consider making a directory structure as shown below, in my test environment i structured mine this way, feel free to structure yours as desired

mkdir config && cd config && mkdir kubernetes-manifest certificate-config && cd certificate-config && mkdir certs
.
├── ~/config
├── kubernetes-manifest
├── certificate-config
│   ├── ca-config.json
│   ├── ca-csr.json
│   ├── consul-csr.json
│   └── vault-csr.json
│   └── certs
│       ├── ca-key.pem
│       ├── ca.csr
│       └──ca.pem

The kubernetes manifest for consul is shown below

cd ~ kubernetes-manifest
vim 01-namespace.yml
########################################
#Consul namespace template
########################################
apiVersion: v1
kind: Namespace
metadata:
  name: consul
  labels:
    app.kubernetes.io/name: consul
    app.kubernetes.io/part-of: consul

vim 02-statefulset.yml
########################################
#Consul statefulset template
########################################
apiVersion: v1
kind: ConfigMap
metadata:
  name: consul
  namespace: consul
  labels:
    app: consul-app
data:
  server.json: |-
    {
      "ca_file": "/etc/tls/ca.pem",
      "cert_file": "/etc/tls/consul.pem",
      "key_file": "/etc/tls/consul-key.pem",
      "verify_incoming": false,
      "verify_incoming_rpc": false,
      "verify_incoming_https": false,
      "verify_outgoing": true,
      "verify_server_hostname": true,
      "primary_datacenter": "dc1",
      "ports": {
        "https": 8443
      },
      "dns_config": {
        "allow_stale": false
      }
    }
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: consul-statefulset
  namespace: consul
spec:
  serviceName: consul-cluster-service
  replicas: 3
  template:
    metadata:
      labels:
        app: consul-app
    spec:
      terminationGracePeriodSeconds: 10
      securityContext:
        fsGroup: 1000
      containers:
        - name: consul
          image: consul:latest
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: GOSSIP_ENCRYPTION_KEY
              valueFrom:
                secretKeyRef:
                  name: consul
                  key: gossip-encryption-key
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          args:
            - "agent"
            - "-advertise=$(POD_IP)"
            - "-bind=0.0.0.0"
            - "-bootstrap-expect=3"
            - "-client=0.0.0.0"
            - "-config-file=/etc/consul"
            - "-datacenter=dc1"
            - "-data-dir=/var/lib/consul"
            - "-domain=cluster.local"
            - "-server"
            - "-ui"
            - "-disable-host-node-id"
            - "-join=consul-cluster-service"
            - "-retry-join=consul-cluster-service"
          volumeMounts:
            - name: data
              mountPath: /var/lib/consul
            - name: config-consul
              mountPath: /etc/consul/server.json
              subPath: server.json
            - name: consul-cert
              mountPath: /etc/tls/consul.pem
              subPath: consul.pem
            - name: consul-key
              mountPath: /etc/tls/consul-key.pem
              subPath: consul-key.pem
            - name: gossip-encryption-key
              mountPath: /etc/tls/gossip-encryption-key
              subPath: gossip-encryption-key
            - name: consul-ca
              mountPath: /etc/tls/ca.pem
              subPath: ca.pem
          lifecycle:
            preStop:
              exec:
                command:
                - /bin/sh
                - -c
                - consul leave
          ports:
            - containerPort: 8500
              name: ui-port
            - containerPort: 8400
              name: alt-port
            - containerPort: 53
              name: udp-port
            - containerPort: 8443
              name: https-port
            - containerPort: 8080
              name: http-port
            - containerPort: 8301
              name: serflan
            - containerPort: 8302
              name: serfwan
            - containerPort: 8600
              name: consuldns
            - containerPort: 8300
              name: server
      volumes:
        - name: config-consul
          configMap:
            name: consul
            items:
              - key: server.json
                path: server.json
        - name: consul-ca
          secret:
            secretName: consul-ca
            items:
              - key: ca.pem
                path: ca.pem
        - name: consul-cert
          secret:
            secretName: consul
            items:
              - key: consul.pem
                path: consul.pem
        - name: consul-key
          secret:
            secretName: consul
            items:
              - key: consul-key.pem
                path: consul-key.pem
        - name: gossip-encryption-key
          secret:
            secretName: consul
            items:
              - key: gossip-encryption-key
                path: gossip-encryption-key
  volumeClaimTemplates:
  - metadata:
      name: data 
      namespace: consul
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: nfs-rw
      resources:
        requests:
          storage: 10Gi
    
vim 03-service.yml
########################################
#Consul service template
########################################
apiVersion: v1
kind: Service
metadata:
  name: consul-cluster-service
  namespace: consul
spec:
  sessionAffinity: ClientIP
  ports:
    - name: http
      port: 8500
      targetPort: 8500
    - name: https
      port: 8443
      targetPort: 8443
    - name: rpc
      port: 8400
      targetPort: 8400
    - name: serflan-tcp
      protocol: "TCP"
      port: 8301
      targetPort: 8301
    - name: serflan-udp
      protocol: "UDP"
      port: 8301
      targetPort: 8301
    - name: serfwan-tcp
      protocol: "TCP"
      port: 8302
      targetPort: 8302
    - name: serfwan-udp
      protocol: "UDP"
      port: 8302
      targetPort: 8302
    - name: server
      port: 8300
      targetPort: 8300
    - name: consuldns
      port: 8600
      targetPort: 8600
  selector:
    app: consul-app

vim 04-ingress.yml
########################################
#Consul Ingress template
########################################
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
  name: consul-ingress
  namespace: consul
spec:
  rules:
  - host: consul.example.com
    http:
      paths:
      - backend:
          serviceName: consul-cluster-service
          servicePort: http
        path: /

go installation

sudo yum install gcc
sudo wget https://dl.google.com/go/go1.12.5.linux-amd64.tar.gz -O go.tar.gz; sudo tar -xzf go.tar.gz -d /usr/local
vim ~/.bashrc
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
exec bash

go get -u github.com/cloudflare/cfssl/cmd/cfssl
go get -u github.com/cloudflare/cfssl/cmd/cfssljson
sudo wget https://releases.hashicorp.com/consul/1.5.1/consul_1.5.1_linux_amd64.zip -O consul.zip; sudo unzip consul.zip -d /usr/local/bin;

consul deployment

cd certificate-config
cfssl gencert -initca ca-csr.json | cfssljson -bare certs/ca
2019/05/11 12:29:25 [INFO] generating a new CA key and certificate from CSR
2019/05/11 12:29:25 [INFO] generate received request
2019/05/11 12:29:25 [INFO] received CSR
2019/05/11 12:29:25 [INFO] generating key: rsa-2048
2019/05/11 12:29:25 [INFO] encoded CSR
2019/05/11 12:29:25 [INFO] signed certificate with serial number xxxxxxxxxxxxxxxxxxxxx
cfssl gencert -ca=certs/ca.pem -ca-key=certs/ca-key.pem -config=ca-config.json -profile=default consul-csr.json | cfssljson -bare  ~/kubernetes-manifest/consul
2019/05/11 13:24:08 [INFO] generate received request
2019/05/11 13:24:08 [INFO] received CSR
2019/05/11 13:24:08 [INFO] generating key: rsa-2048
2019/05/11 13:24:09 [INFO] encoded CSR
2019/05/11 13:24:09 [INFO] signed certificate with serial number xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
cd ~/kubernetes-manifest
kubectl -n consul create secret generic consul --from-file=gossip-encryption-key  --from-file=consul.pem --from-file=consul-key.pem
secret/consul created
kubectl -n consul create secret generic consul-ca --from-file=--from-file=~/certificate-config/certs/ca.pem
secret/consul-ca created
kubectl apply -f ./
service/consul-cluster-service created
configmap/consul created
statefulset.apps/consul-statefulset created
ingress.extensions/consul-ingress created

John Adebayo

Recommended for you

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
#