Using Secretless in Kubernetes

Steps for Security Admin

You are now the Security Admin

The Security Admin sets up PostgreSQL, configures Secretless, and has sole access to the credentials.

Create PostgreSQL Service in Kubernetes

PostgreSQL is stateful, so we’ll use a StatefulSet to manage it.

Deploy PostgreSQL StatefulSet

To deploy a PostgreSQL StatefulSet:

1. Create a dedicated namespace for the storage backend:

   kubectl create namespace quick-start-backend-ns
   

   namespace "quick-start-backend-ns" created
   

2. Create a self-signed certificate (see PostgreSQL documentation for more info):

   openssl req -new -x509 -days 365 -nodes -text -out server.crt \
     -keyout server.key -subj "/CN=pg"
   chmod og-rwx server.key
   

   Generating a 2048 bit RSA private key
   ....................................................................................+++++
   .......+++++
   writing new private key to 'server.key'
   -----

3. Store the certificate files as Kubernetes secrets in the quick-start-backend-ns namespace:

   kubectl --namespace quick-start-backend-ns \
     create secret generic \
     quick-start-backend-certs \
     --from-file=server.crt \
     --from-file=server.key
   

   secret "quick-start-backend-certs" created

   While Kubernetes Secrets are more secure than hard-coded ones, in a real deployment you should secure secrets in a fully-featured vault, like Conjur.

4. Create and save the PostgreSQL StatefulSet manifest in a file named pg.yml in your current working directory:

   cat << EOF > pg.yml
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: pg
     labels:
       app: quick-start-backend
   spec:
     serviceName: quick-start-backend
     selector:
       matchLabels:
         app: quick-start-backend
     template:
       metadata:
         labels:
           app: quick-start-backend
       spec:
         securityContext:
           fsGroup: 999
         containers:
         - name: quick-start-backend
           image: postgres:9.6
           imagePullPolicy: IfNotPresent
           ports:
             - containerPort: 5432
           env:
             - name: POSTGRES_DB
               value: postgres
             - name: POSTGRES_USER
               value: security_admin_user
             - name: POSTGRES_PASSWORD
               value: security_admin_password
           volumeMounts:
           - name: backend-certs
             mountPath: "/etc/certs/"
             readOnly: true
           args: ["-c", "ssl=on", "-c", "ssl_cert_file=/etc/certs/server.crt", "-c", "ssl_key_file=/etc/certs/server.key"]
         volumes:
         - name: backend-certs
           secret:
             secretName: quick-start-backend-certs
             defaultMode: 384
   EOF
   

   In the manifest above, the certificate files for your database server are mounted in a volume with defaultMode: 384 giving it permissions 0600 (Why? Because 600 in base 8 = 384 in base 10).
   The pod is deployed with 999 as the group associated with any mounted volumes, as indicated by fsGroup: 999. 999 is a the static postgres gid, defined in the postgres Docker image

5. Deploy the PostgreSQL StatefulSet:

   kubectl --namespace quick-start-backend-ns apply -f pg.yml
   

   statefulset "pg" created
   

   This StatefulSet uses the DockerHub postgres:9.6 container.

   On startup, the container creates a superuser from the environment variables POSTGRES_USER and POSTGRES_PASSWORD, which we set to the values security_admin_user and security_admin_password, respectively.

   Going forward, we’ll call these values the admin-credentials, to distinguish them from the application-credentials our application will use.

   In the scripts below, we’ll refer to the admin-credentials by the environment variables SECURITY_ADMIN_USER and SECURITY_ADMIN_PASSWORD.

6. To ensure the PostgreSQL StatefulSet pod has started and is healthy (this may take a minute or so), run:

   kubectl --namespace quick-start-backend-ns get pods
   

   NAME      READY     STATUS    RESTARTS   AGE
   pg-0      1/1       Running   0          6s
   

Expose PostgreSQL Service

Our PostgreSQL StatefulSet is running, but we still need to expose it publicly as a Kubernetes service.

To expose the database, run:

cat << EOF > pg-service.yml
kind: Service
apiVersion: v1
metadata:
  name: quick-start-backend
spec:
  selector:
    app: quick-start-backend
  ports:
    - port: 5432
      targetPort: 5432
      nodePort: 30001
  type: NodePort

EOF
kubectl --namespace quick-start-backend-ns  apply -f pg-service.yml

service "quick-start-backend" created

The database is now available at $(minikube ip):30001, which we’ll compose with REMOTE_DB_HOST and REMOTE_DB_PORT variables.

The database has no data yet, but we can verify it works by logging in as the security admin and listing the users:

export SECURITY_ADMIN_USER=security_admin_user
export SECURITY_ADMIN_PASSWORD=security_admin_password
export REMOTE_DB_HOST=$(minikube ip)
export REMOTE_DB_PORT=30001

docker run --rm -it -e PGPASSWORD=${SECURITY_ADMIN_PASSWORD} postgres:9.6 \
  psql -U ${SECURITY_ADMIN_USER} "postgres://${REMOTE_DB_HOST}:${REMOTE_DB_PORT}/postgres" -c "\du"

                                          List of roles
       Role name        |                         Attributes                    
     | Member of
------------------------+-------------------------------------------------------
-----+-----------
 security_admin_user    | Superuser, Create role, Create DB, Replication, Bypass
 RLS | {}

Create Application Database

In this section, we assume the following:

• You already have a PostgreSQL database exposed as a Kubernetes service.
• It’s publicly available at $REMOTE_DB_HOST:$REMOTE_DB_PORT
• You have admin-level database credentials
• The SECURITY_ADMIN_USER and SECURITY_ADMIN_PASSWORD environment variables hold those credentials

If you followed along in the last section and are using minikube, you can run:

export SECURITY_ADMIN_USER=security_admin_user
export SECURITY_ADMIN_PASSWORD=security_admin_password
export REMOTE_DB_HOST="$(minikube ip)"
export REMOTE_DB_PORT="30001"

Next, we’ll create the application database and user, and securely store the user’s credentials:

1. Create the application database
2. Create the pets table in that database
3. Create an application user with limited privileges: SELECT and INSERT on the pets table
4. Store these database application-credentials in Kubernetes secrets.

So we can refer to them later, export the database name and application-credentials as environment variables:

export APPLICATION_DB_NAME=quick_start_db

export APPLICATION_DB_USER=app_user
export APPLICATION_DB_INITIAL_PASSWORD=app_user_password

Finally, to perform the 4 steps listed above, run:

docker run --rm -i -e PGPASSWORD=${SECURITY_ADMIN_PASSWORD} postgres:9.6 \
    psql -U ${SECURITY_ADMIN_USER} "postgres://${REMOTE_DB_HOST}:${REMOTE_DB_PORT}/postgres" \
    << EOSQL

CREATE DATABASE ${APPLICATION_DB_NAME};

/* connect to it */

\c ${APPLICATION_DB_NAME};

CREATE TABLE pets (
  id serial primary key,
  name varchar(256)
);

/* Create Application User */

CREATE USER ${APPLICATION_DB_USER} PASSWORD '${APPLICATION_DB_INITIAL_PASSWORD}';

/* Grant Permissions */

GRANT SELECT, INSERT ON public.pets TO ${APPLICATION_DB_USER};
GRANT USAGE, SELECT ON SEQUENCE public.pets_id_seq TO ${APPLICATION_DB_USER};
EOSQL

CREATE DATABASE
You are now connected to database "quick_start_db" as user "security_admin_user".
CREATE TABLE
CREATE ROLE
GRANT
GRANT

Create Application Namespace and Store Credentials

The application will be scoped to the quick-start-application-ns namespace.

To create the namespace run:

kubectl create namespace quick-start-application-ns

namespace "quick-start-application-ns" created

Next we’ll store the application-credentials in Kubernetes Secrets:

kubectl --namespace quick-start-application-ns \
  create secret generic quick-start-backend-credentials \
  --from-literal=host="${REMOTE_DB_HOST}" \
  --from-literal=port="${REMOTE_DB_PORT}" \
  --from-literal=username="${APPLICATION_DB_USER}" \
  --from-literal=password="${APPLICATION_DB_INITIAL_PASSWORD}"

secret "quick-start-backend-credentials" created

Create Secretless Broker Configuration ConfigMap

With our database ready and our credentials safely stored, we can now configure the Secretless Broker. We’ll tell it where to listen for connections and how to proxy them.

After that, the developer’s application can access the database without ever knowing the application-credentials.

A Secretless Broker configuration file defines the services that Secretless with authenticate to on behalf of your application.

To create secretless.yml in your current directory, run:

cat << EOF > secretless.yml
version: "2"
services:
  pets-pg:
    connector: pg
    listenOn: tcp://localhost:5432
    credentials:
      host:
        from: kubernetes
        get: quick-start-backend-credentials#host
      port:
        from: kubernetes
        get: quick-start-backend-credentials#port
      username:
        from: kubernetes
        get: quick-start-backend-credentials#username
      password:
        from: kubernetes
        get: quick-start-backend-credentials#password
EOF

Here’s what this does:

• Defines a service called pets-pg that listens for PostgreSQL connections on localhost:5432
• Says that the database host, port, username and password are stored in Kubernetes Secrets
• Lists the ids of those credentials within Kubernetes Secrets

Next we create a Kubernetes ConfigMap from this secretless.yml:

kubectl --namespace quick-start-application-ns \
  create configmap \
  quick-start-application-secretless-config \
  --from-file=secretless.yml

configmap "quick-start-application-secretless-config" created

Create Application Service Account and Grant Entitlements

To grant our application access to the credentials in Kubernetes Secrets, we’ll need a ServiceAccount:

kubectl --namespace quick-start-application-ns \
  create serviceaccount \
  quick-start-application

serviceaccount "quick-start-application" created

Next we grant this ServiceAccount “get” access to the quick-start-backend-credentials. This is a 2 step process:

1. Create a Role with permissions to get the quick-start-backend-credentials secret
2. Create a RoleBinding so our ServiceAccount has this Role

Run this command to perform both steps:

cat << EOF > quick-start-application-entitlements.yml
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: quick-start-backend-credentials-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["quick-start-backend-credentials"]
  verbs: ["get"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-quick-start-backend-credentials
subjects:
- kind: ServiceAccount
  name: quick-start-application
roleRef:
  kind: Role
  name: quick-start-backend-credentials-reader
  apiGroup: rbac.authorization.k8s.io
EOF

kubectl --namespace quick-start-application-ns \
  apply -f quick-start-application-entitlements.yml

role "quick-start-backend-credentials-reader" created
rolebinding "read-quick-start-backend-credentials" created

Up next...

As an Application Developer, you no longer need to worry about all the passwords and database connections! You will deploy an application and leave it up to the Secretless Broker to make the desired connection to the database.