Boundery on Kubernetes with Keycloak
We have 3 clusters running 2 on AWS and 1 on-prem. And to sort out connections for developers and admin the goal is to implement boundary as an access point. To verify the user we use Keycloak and 2FA, Then based on roles we give the different users access to different services inside the cluster.
Service
The user should be able to connect to an ssh server inside the network but also to service running inside Kubernetes like elasticsearch ore MySQL,
2 stages
We are to set up boundary in two stages the first is to deploy the boundary service into the cluster. And the second is to config boundery using terraform.
Requirement
To get started we have a Kubernetes cluster and we also need an external that we can use to get access. I will expose the boundary service using metal-lb but you can change to use node ports on example EKS or GCP.
Postgress SQL boundary uses a Postgres SQL to store its configs. We need a Postgres SQL with username and password ready.
We also need some keys to be used for bounder and the easy way to generate new keys are to run boundary dev
boundary dev
[Controller] AEAD Key Bytes: kj9LoQqyZs2a3cfwmDy/u3tDwWdGEyPYhY3rXDoc5+A=
[Recovery] AEAD Key Bytes: 2kgQXuYYuc5TyTyNi+DOg+DqiJVqZFuWlohUPfhz1Tc=
[Worker-Auth] AEAD Key Bytes: okXvaWDI2FuRZO6ZnNJm1vBXL32jZsnrMNuZ7wQ8MHE=- Lets deploy
Let’s get a certificate with lets encrypt
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: boundery-tls
namespace: boundery
spec:
secretName: boundery-tls
issuerRef:
name: letsencrypt
kind: ClusterIssuer
commonName: boundery.example.com
dnsNames:
- boundery.example.com
PSP
I have a cluster that enforces PSP hard and for that, I use the following PSP. You may not need this for example on EKS and PSP is to be terminated.
apiVersion: v1
kind: ServiceAccount
metadata:
name: controller
namespace: boundery
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
name: 10-boundery-controller
spec:
allowedCapabilities:
- IPC_LOCK
- SETFCAP
fsGroup:
rule: RunAsAny
privileged: true
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- secret
- configMap
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/cluster-service: "true"
name: psp-boundery-controller # named for psp-<namespace>-<serviceaccount>
rules:
- apiGroups:
- policy
resourceNames:
- 10-boundery-controller
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
annotations:
kubernetes.io/description: 'tailored PSP for vault'
labels:
kubernetes.io/cluster-service: "true"
name: psp-boundery-controller
namespace: boundery
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp-boundery-controller
subjects:
- kind: ServiceAccount
namespace: boundery
name: controllerNow we can deploy the controller
You need to add the keys from above and also add the posgress password IN 2 PLACES
apiVersion: v1
kind: ConfigMap
metadata:
name: boundery
namespace: boundery
data:
boundary.hcl: |-
# Disable memory lock: https://www.man7.org/linux/man-pages/man2/mlock.2.html
disable_mlock = true
# Controller configuration block
controller {
# This name attr must be unique across all controller instances if running in HA mode
name = "controller"
description = "Controller"
public_cluster_addr = "boundery.example.com"
# Database URL for postgres. This can be a direct "postgres://"
# URL, or it can be "file://" to read the contents of a file to
# supply the url, or "env://" to name an environment variable
# that contains the URL.
database {
url = "env://BOUNDARY_PG_URL"
}
}
# API listener configuration block
listener "tcp" {
# Should be the address of the NIC that the controller server will be reached on
address = "0.0.0.0"
# The purpose of this listener block
purpose = "api"
tls_disable = false
tls_cert_file = "/tls/tls.crt"
tls_key_file = "/tls/tls.key"
tls_min_version = "tls13"
# Uncomment to enable CORS for the Admin UI. Be sure to set the allowed origin(s)
# to appropriate values.
cors_enabled = false
#cors_allowed_origins = ["https://boundery.examples.com", "serve://boundary"]
}
# Data-plane listener configuration block (used for worker coordination)
listener "tcp" {
# Should be the IP of the NIC that the worker will connect on
address = "0.0.0.0"
# The purpose of this listener
purpose = "cluster"
tls_disable = false
tls_cert_file = "/tls/tls.crt"
tls_key_file = "/tls/tls.key"
tls_min_version = "tls13"
}
# Root KMS configuration block: this is the root key for Boundary
# Use a production KMS such as AWS KMS in production installs
kms "aead" {
purpose = "root"
aead_type = "aes-gcm"
key = "ADD YOUR HERE"
key_id = "global_root"
}
# Worker authorization KMS
# Use a production KMS such as AWS KMS for production installs
# This key is the same key used in the worker configuration
kms "aead" {
purpose = "worker-auth"
aead_type = "aes-gcm"
key = "ADD YOUR HERE"
key_id = "global_worker-auth"
}
# Recovery KMS block: configures the recovery key for Boundary
# Use a production KMS such as AWS KMS for production installs
kms "aead" {
purpose = "recovery"
aead_type = "aes-gcm"
key = "ADD YOUR HERE"
key_id = "global_recovery"
}
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: controller
namespace: boundery
spec:
replicas: 1
selector:
matchLabels:
app: controller
serviceName: controller
podManagementPolicy: Parallel
template:
metadata:
labels:
app: controller
name: controller
spec:
initContainers:
- name: init-db
image: hashicorp/boundary:0.7.1
args: ["database", "init","-skip-auth-method-creation","-skip-host-resources-creation","-skip-scopes-creation","-skip-target-creation","-config", "/boundary/boundary.hcl"]
env:
- name: BOUNDARY_PG_URL
value: postgresql://postgres:PASSWORD@boundery-postgresql:5432/boundery?sslmode=disable
- name: HOSTNAME
value: boundary
securityContext:
capabilities:
add:
- IPC_LOCK
volumeMounts:
- name: config
mountPath: /boundary/boundary.hcl
subPath: boundary.hcl
readOnly: true
- name: tls
mountPath: /tls
readOnly: true
containers:
- name: controller
image: hashicorp/boundary:0.7.1
args: ["server", "-config", "/boundary/boundary.hcl"]
env:
- name: BOUNDARY_PG_URL
value: postgresql://postgres:PASSWORD@boundery-postgresql:5432/boundery?sslmode=disable
- name: HOSTNAME
value: boundary
resources:
requests:
cpu: 200m
memory: 1024Mi
limits:
cpu: 500m
memory: 2048Mi
ports:
- containerPort: 9200
name: api
- containerPort: 9201
name: connections
- containerPort: 9202
name: access
securityContext:
capabilities:
add:
- IPC_LOCK
volumeMounts:
- name: config
mountPath: /boundary/boundary.hcl
subPath: boundary.hcl
readOnly: true
- name: tls
mountPath: /tls
readOnly: true
serviceAccountName: controller
volumes:
- name: config
configMap:
name: boundery
- name: tls
secret:
secretName: boundery-tlsApply the yaml to the cluster and verify the controller starts up. When it running we can go on deploying the worker. Adding Worker
apiVersion: v1
kind: ConfigMap
metadata:
name: boundery-worker
namespace: boundery
data:
boundary.hcl: |-
# Disable memory lock: https://www.man7.org/linux/man-pages/man2/mlock.2.html
disable_mlock = true
listener "tcp" {
purpose = "proxy"
address = "0.0.0.0"
tls_disable = false
tls_cert_file = "/tls/tls.crt"
tls_key_file = "/tls/tls.key"
tls_min_version = "tls13"
}
worker {
# Name attr must be unique across workers
name = "worker"
description = "Worker in the cluster"
# Workers must be able to reach controllers on :9201
controllers = [
"boundery.example.com"
]
public_addr = "boundery.example.com"
tags {
type = ["onprem"]
}
}
# must be same key as used on controller config
kms "aead" {
purpose = "worker-auth"
aead_type = "aes-gcm"
key = ""
key_id = "global_worker-auth"
}
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: worker
namespace: boundery
spec:
replicas: 1
selector:
matchLabels:
app: worker
serviceName: worker
podManagementPolicy: Parallel
template:
metadata:
labels:
app: worker
name: worker
spec:
containers:
- name: worker
image: hashicorp/boundary:0.7.1
args: ["server", "-config", "/boundary/boundary.hcl"]
env:
- name: HOSTNAME
value: boundary
volumeMounts:
- name: config
mountPath: /boundary/boundary.hcl
subPath: boundary.hcl
readOnly: true
- name: tls
mountPath: /tls
readOnly: true
resources:
requests:
cpu: 200m
memory: 1024Mi
limits:
cpu: 500m
memory: 2048Mi
ports:
- containerPort: 9201
name: connections
securityContext:
capabilities:
add:
- IPC_LOCK
volumeMounts:
- name: config
mountPath: /boundary/boundary.hcl
subPath: boundary.hcl
readOnly: true
serviceAccountName: worker
volumes:
- name: config
configMap:
name: boundery-worker
- name: tls
secret:
secretName: boundery-tlsNote the Following
tags {
type = ["onprem"]
}
}This is how we can make different works handle different endpoints. Say we have an ssh server that is only are accessible from the on-prem cluster. Then we need to tag the worker with on-prem. And also tag the connection we will create later with on-prem. Then we can control so the right worker handle the right endpoints
Now let’s deploy our service so we can access the boundary server. Here I use metal-lb and external-dns to setup IP and DNS. You need to config this to match your setup.
---
apiVersion: v1
kind: Service
metadata:
name: controller
namespace: boundery
annotations:
metallb.universe.tf/address-pool: boundery
metallb.universe.tf/allow-shared-ip: "boundery"
external-dns.alpha.kubernetes.io/hostname: boundery.example.com
spec:
ports:
- port: 9200
name: controller
targetPort: 9200
- port: 9201
name: workers
targetPort: 9201
selector:
app: controller
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: worker
namespace: boundery
annotations:
metallb.universe.tf/address-pool: boundery
metallb.universe.tf/allow-shared-ip: "boundery"
spec:
ports:
- port: 9202
name: api
targetPort: 9202
selector:
app: worker
type: LoadBalancerTest now so you have access to boundary on port 9200 you should see the login page of boundery.
If not go back and check your settings, TLS certs are created and postgress is running.
If you have the login let’s move on to set up boundary with terraform
First, we create the settings for boundary
terraform.tf
provider "boundary" {
addr = "https://boundery.example.com:9200"
recovery_kms_hcl = <<EOT
kms "aead" {
purpose = "recovery"
aead_type = "aes-gcm"
key = "ADD YOUR HERE"
key_id = "global_recovery"
}
EOT
}
This will then be used to connect to boundary and config it
some scopes
resource "boundary_scope" "bundery" {
scope_id = "global"
name = "Boundey"
description = "globa Tech"
auto_create_admin_role = false
auto_create_default_role = true
}
resource "boundary_scope" "onprem" {
name = "Onprem"
description = "Onprem"
scope_id = boundary_scope.bundery.id
auto_create_admin_role = false
auto_create_default_role = true
}
resource "boundary_scope" "aws" {
name = "aws"
description = "AWS Prod"
scope_id = boundary_scope.bundery.id
auto_create_admin_role = false
auto_create_default_role = true
}Setting up auth against keycloak
resource "boundary_auth_method_oidc" "keycloak" {
name = "SSO"
description = "Keycloak SSO"
type = "oidc"
api_url_prefix = "https://boundery.example.com:9200/"
client_id = "boundery"
is_primary_for_scope = "true"
#claims_scopes = ["aud","sub","iss","auth_time","name","given_name","family_name","preferred_username","email","acr"]
client_secret = "92e9509d-8215-4e53-8ffc-4f742fbab720"
scope_id = "global"
signing_algorithms = ["PS384","ES384","RS384","ES256","RS256","ES512","PS256","PS512","RS512"]
issuer = "https://auth.example.com/auth/realms/master"
}
some auth
#Logged in users
resource "boundary_role" "user" {
name = "user_org"
description = "User"
principal_ids = ["u_auth"]
grant_strings = [
"id=*;type=*;actions=read,list",
]
scope_id = boundary_scope.boundery.id
}
resource "boundary_role" "user_onprem" {
name = "access_onprem"
description = "Access for onprem"
principal_ids = ["u_auth"]
grant_strings = [
"id=*;type=*;actions=read,list,authorize-session",
"id=*;type=host-catalog;actions=read,list",
"id=*;type=target;actions=list,read,authorize-session",
"id=*;type=session;actions=cancel:self"
]
scope_id = boundary_scope.onprem.id
}
resource "boundary_role" "user_aws" {
name = "access_aws"
description = "Access for AWS"
principal_ids = ["u_auth"]
grant_strings = [
"id=*;type=*;actions=read,list,authorize-session",
"id=*;type=host-catalog;actions=read,list",
"id=*;type=target;actions=list,read,authorize-session",
"id=*;type=session;actions=cancel:self"
]
scope_id = boundary_scope.aws.id
}
## Anon Users
resource "boundary_role" "global_anon_listing" {
scope_id = "global"
grant_strings = [
"id=*;type=auth-method;actions=list,authenticate",
"id=*;type=scope;actions=read,list",
"id={{account.id}};actions=read,change-password"
]
principal_ids = ["u_anon"]
}
resource "boundary_role" "org_anon_listing_aws" {
scope_id = boundary_scope.aws.id
grant_strings = [
"id=*;type=auth-method;actions=list,authenticate",
"id=*;type=scope;actions=read,list",
"id={{account.id}};actions=read,change-password"
]
principal_ids = ["u_anon"]
}
resource "boundary_role" "org_anon_listing_onprem" {
scope_id = boundary_scope.onprem.id
grant_strings = [
"id=*;type=auth-method;actions=list,authenticate",
"id=*;type=scope;actions=read,list",
"id={{account.id}};actions=read,change-password"
]
principal_ids = ["u_anon"]
}
And now finally let’s create some targets to connect to
#Make a hostgroup
resource "boundary_host_catalog" "k8s_access" {
name = "k8s"
description = "K8s Endpoints"
type = "static"
scope_id = boundary_scope.onprem.id
}
#Make a host
resource "boundary_host" "search_onprem" {
type = "static"
name = "search"
description = "search"
address = "search.elasticsearch.svc" <--- LOK AT THIS THIS WORKER IS RUNNING INSIDE K8S AND WE CAN ACCESS INTERNAL SERVICE !!!!
host_catalog_id = boundary_host_catalog.k8s_access.id
}
# Make a Hostset
resource "boundary_host_set" "search_onprem" {
type = "static"
name = "k8s_master"
description = "Host set for k8s master"
host_catalog_id = boundary_host_catalog.k8s_access.id
host_ids = [
boundary_host.search_onprem.id
]
}
# The target we use the host and add the port
resource "boundary_target" "search_onprem" {
type = "tcp"
name = "search_9200"
description = "9200 to search"
scope_id = boundary_scope.onprem.id
default_port = "9200"
worker_filter = "\"onprem\" in \"/tags/type\"" <--- REMEBER THIS TAG WE USE IN THE WORKER
host_source_ids = [
boundary_host_set.search_onprem.id
]
}so you can add more hosts and add them with IP ore DNS, Then you can assign targets to the hosts.
If you have workers running in some other location like AWS then you can simply deploy the worker and change the tag so it matches.