Deploying Consul and Vault as HA in Kubernetes (Part 2)
Vault
what is vault?
Vault is an Harshicorp tool that is used for secrets management. By secrets management we mean it secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and any secrets of your choice. you can read more on
https://learn.hashicorp.com/vault
From the previous post in the link below,
https://chainkindle.com/deploying-consul-and-vault-as-ha-in-kubernetes
we deployed Consul in a kubernetes cluster. in this post we will be deploying vault in a Production environment using Consul as our backend.
Perequisites
- Already deployed consul in the Kubernetes cluster or container engine
- Traefik as well for ssl termination
- https://chainkindle.com/deploying-consul-and-vault-as-ha-in-kubernetes/
Directory Structure
.
├── ~/config
├── kubernetes-manifest
├── certificate-config
│ ├── ca-config.json
│ ├── ca-csr.json
│ ├── consul-csr.json
│ └── vault-csr.json
│ └── certs
│ ├── ca-key.pem
│ ├── ca.csr
│ └── ca.pem
ldap Integration to vault
url="ldaps://xxxxxxxxxxxxxxxxxx:636"
userattr="displayName"
userdn="OU=User,DC=chainkindle,DC=ng"
groupdn="OU=IT_OU,DC=chainkindle,DC=ng"
groupattr="member"
binddn="CN=function-ad,OU=Project,DC=chainkindle,DC=ng"
bindpass=''
certificate=@ldap_ca_cert.pem
insecure_tls=false
starttls=true
The kubernetes manifest for Vault is shown below
---
apiVersion: v1
kind: Service
metadata:
name: vault-cluster-service
namespace: consul
spec:
sessionAffinity: ClientIP
selector:
app: vault
ports:
- name: http
port: 8200
protocol: TCP
targetPort: 8200
- name: cluster-port
port: 8201
protocol: TCP
targetPort: 8201
---
apiVersion: v1
kind: ConfigMap
metadata:
name: vault
namespace: consul
labels:
app: vault
data:
server.json: |-
{
"listener": {
"tcp":{
"address": "0.0.0.0:8200",
"cluster_address": "0.0.0.0:8201",
"tls_disable": 1,
"tls_cert_file": "/etc/tls/vault.pem",
"tls_key_file": "/etc/tls/vault-key.pem"
}
},
"storage": {
"consul": {
"address": "consul-cluster-service:8500",
"advertise_address": "vault-cluster-service:8200",
"check_timeout": "30s",
"path": "vault/",
"ha_enabled": "true"
}
},
"cache": {
"use_auto_auth_token": "false"
},
"api_addr": "http://0.0.0.0:8200",
"cluster_addr": "http://0.0.0.0:8201",
"disable_mlock": true,
"bootstrap": true,
"ui": true,
"log_level": "debug",
"default_lease_ttl": "168h",
"max_lease_ttl": "720h"
}
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: vault
namespace: consul
spec:
serviceName: vault-cluster-service
replicas: 1
template:
metadata:
labels:
app: vault
spec:
terminationGracePeriodSeconds: 10
securityContext:
fsGroup: 1000
containers:
- name: vault
command: ["vault", "server", "-config", "/vault/config/server.json"]
securityContext:
capabilities:
add:
- IPC_LOCK
image: "vault:latest"
imagePullPolicy: IfNotPresent
ports:
- name: vault-http
containerPort: 8200
- name: cluster-port
containerPort: 8201
env:
- name: VAULT_ADDR
value: "http://127.0.0.1:8200"
- name: VAULT_SKIP_VERIFY
value: "true"
volumeMounts:
- name: config-vault
mountPath: /vault/config/server.json
subPath: server.json
- name: vault-cert
mountPath: /etc/tls/vault.pem
subPath: vault.pem
- name: vault-key
mountPath: /etc/tls/vault-key.pem
subPath: vault-key.pem
- name: consul-ca
mountPath: /etc/tls/ca.pem
subPath: ca.pem
- name: ldap-cert
mountPath: /etc/tls/LDAP-Proxy-CA.pem
subPath: LDAP-Proxy-CA.pem
- name: consul-agent
image: "consul:latest"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: GOSSIP_ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: consul
key: gossip-encryption-key
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
args:
- "agent"
- "-advertise=$(POD_IP)"
- "-server=false"
- "-datacenter=dc1"
- "-domain=cluster.local"
- "-disable-host-node-id"
- "-bind=0.0.0.0"
- "-client=127.0.0.1"
- "-join=consul-cluster-service"
- "-retry-join=consul-cluster-service"
- "-ui"
volumeMounts:
- name: config-consul
mountPath: /etc/consul
- name: consul-cert
mountPath: /etc/tls/consul.pem
subPath: consul.pem
- name: consul-key
mountPath: /etc/tls/consul-key.pem
subPath: consul-key.pem
- name: gossip-encryption-key
mountPath: /etc/tls/gossip-encryption-key
subPath: gossip-encryption-key
- name: consul-ca
mountPath: /etc/tls/ca.pem
subPath: ca.pem
volumes:
- name: config-vault
configMap:
name: vault
items:
- key: server.json
path: server.json
- name: config-consul
configMap:
name: consul
items:
- key: server.json
path: server.json
- name: vault-cert
secret:
secretName: vault
items:
- key: vault.pem
path: vault.pem
- name: vault-key
secret:
secretName: vault
items:
- key: vault-key.pem
path: vault-key.pem
- name: ldap-cert
secret:
secretName: ldap-cert
items:
- key: LDAP-Proxy-CA.pem
path: LDAP-Proxy-CA.pem
- name: consul-ca
secret:
secretName: consul-ca
items:
- key: ca.pem
path: ca.pem
- name: consul-cert
secret:
secretName: consul
items:
- key: consul.pem
path: consul.pem
- name: consul-key
secret:
secretName: consul
items:
- key: consul-key.pem
path: consul-key.pem
- name: gossip-encryption-key
secret:
secretName: consul
items:
- key: gossip-encryption-key
path: gossip-encryption-key
volumeClaimTemplates:
- metadata:
name: data
namespace: consul
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "put-custom-storage-class"
resources:
requests:
storage: 10Gi
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: traefik
name: vault-ingress
namespace: consul
spec:
rules:
- host: vault.example.ng
http:
paths:
- backend:
serviceName: vault-cluster-service
servicePort: http
path: /
vault deployment
cd certificate-config
cfssl gencert -ca=certs/ca.pem -ca-key=certs/ca-key.pem -config=ca-config.json -profile=default vault-csr.json | cfssljson -bare ~/kubernetes-manifest/vault
2019/05/11 13:11:33 [INFO] generate received request
2019/05/11 13:11:33 [INFO] received CSR
2019/05/11 13:11:33 [INFO] generating key: rsa-2048
2019/05/11 13:11:34 [INFO] encoded CSR
2019/05/11 13:11:34 [INFO] signed certificate with serial number xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
cd ~/kubernetes-manifest
kubectl -n consul create secret generic vault --from-file=vault.pem --from-file=vault-key.pem
secret/vault created
kubectl apply -f .
service/vault-cluster-service created
configmap/vault created
statefulset.apps/vault created
Initializing Vault
kubectl -n consul exec -it "vault-0" /bin/sh
Defaulting container name to vault.
Use 'kubectl describe pod/vault-0 -n consul' to see all of the containers in this pod.
/ # vault operator init
Unseal Key 1: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Unseal Key 2: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Unseal Key 3: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Unseal Key 4: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Unseal Key 5: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Initial Root Token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
/ # vault operator unseal
Unseal Key (will be hidden):xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Version 1.1.2
HA Enabled true
/ # vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Version 1.1.2
HA Enabled true
/ # vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.1.2
Cluster Name vault-cluster-xxxxxxxxxx
Cluster ID xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>
/ # vault login s.XXXXXXXXXXXXXXXXXXXXXXXXXX
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token XXXXXXXXXXXXXXXXXXXXXXXX
token_accessor XXXXXXXXXXXXXXXXXXXXXXXX
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
/ # vault audit list
No audit devices are enabled.
/ # vault audit enable file file_path=/var/log/audit.log
Success! Enabled the file audit device at: file/
/ # vault auth enable ldap
Success! Enabled ldap auth method at: ldap/