Using Secretless in Kubernetes

Estimated time to complete: 5 min

Products used: Kubernetes Secrets, PostgreSQL Service Connector

Steps for Security Admin

Security Admin
You are now the Security Admin
The Security Admin sets up PostgreSQL, configures Secretless, and has sole access to the credentials.

Create PostgreSQL Service in Kubernetes

PostgreSQL is stateful, so we’ll use a StatefulSet to manage it.

Deploy PostgreSQL StatefulSet

To deploy a PostgreSQL StatefulSet:

  1. Create a dedicated namespace for the storage backend:

    kubectl create namespace quick-start-backend-ns
    
    namespace "quick-start-backend-ns" created
    
  2. Create a self-signed certificate (see PostgreSQL documentation for more info):

    openssl req -new -x509 -days 365 -nodes -text -out server.crt \
      -keyout server.key -subj "/CN=pg"
    chmod og-rwx server.key
    
    Generating a 2048 bit RSA private key
    ....................................................................................+++++
    .......+++++
    writing new private key to 'server.key'
    -----
  3. Store the certificate files as Kubernetes secrets in the quick-start-backend-ns namespace:

    kubectl --namespace quick-start-backend-ns \
      create secret generic \
      quick-start-backend-certs \
      --from-file=server.crt \
      --from-file=server.key
    
    secret "quick-start-backend-certs" created
    While Kubernetes Secrets are more secure than hard-coded ones, in a real deployment you should secure secrets in a fully-featured vault, like Conjur.
  4. Create and save the PostgreSQL StatefulSet manifest in a file named pg.yml in your current working directory:

    cat << EOF > pg.yml
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: pg
      labels:
        app: quick-start-backend
    spec:
      serviceName: quick-start-backend
      selector:
        matchLabels:
          app: quick-start-backend
      template:
        metadata:
          labels:
            app: quick-start-backend
        spec:
          securityContext:
            fsGroup: 999
          containers:
          - name: quick-start-backend
            image: postgres:9.6
            imagePullPolicy: IfNotPresent
            ports:
              - containerPort: 5432
            env:
              - name: POSTGRES_DB
                value: postgres
              - name: POSTGRES_USER
                value: security_admin_user
              - name: POSTGRES_PASSWORD
                value: security_admin_password
            volumeMounts:
            - name: backend-certs
              mountPath: "/etc/certs/"
              readOnly: true
            args: ["-c", "ssl=on", "-c", "ssl_cert_file=/etc/certs/server.crt", "-c", "ssl_key_file=/etc/certs/server.key"]
          volumes:
          - name: backend-certs
            secret:
              secretName: quick-start-backend-certs
              defaultMode: 384
    EOF
    
    In the manifest above, the certificate files for your database server are mounted in a volume with defaultMode: 384 giving it permissions 0600 (Why? Because 600 in base 8 = 384 in base 10).
    The pod is deployed with 999 as the group associated with any mounted volumes, as indicated by fsGroup: 999. 999 is a the static postgres gid, defined in the postgres Docker image
  5. Deploy the PostgreSQL StatefulSet:

    kubectl --namespace quick-start-backend-ns apply -f pg.yml
    
    statefulset "pg" created
    

    This StatefulSet uses the DockerHub postgres:9.6 container.

    On startup, the container creates a superuser from the environment variables POSTGRES_USER and POSTGRES_PASSWORD, which we set to the values security_admin_user and security_admin_password, respectively.

    Going forward, we’ll call these values the admin-credentials, to distinguish them from the application-credentials our application will use.

    In the scripts below, we’ll refer to the admin-credentials by the environment variables SECURITY_ADMIN_USER and SECURITY_ADMIN_PASSWORD.

  6. To ensure the PostgreSQL StatefulSet pod has started and is healthy (this may take a minute or so), run:

    kubectl --namespace quick-start-backend-ns get pods
    
    NAME      READY     STATUS    RESTARTS   AGE
    pg-0      1/1       Running   0          6s
    

Expose PostgreSQL Service

Our PostgreSQL StatefulSet is running, but we still need to expose it publicly as a Kubernetes service.

To expose the database, run:

cat << EOF > pg-service.yml
kind: Service
apiVersion: v1
metadata:
  name: quick-start-backend
spec:
  selector:
    app: quick-start-backend
  ports:
    - port: 5432
      targetPort: 5432
      nodePort: 30001
  type: NodePort

EOF
kubectl --namespace quick-start-backend-ns  apply -f pg-service.yml
service "quick-start-backend" created
The service manifest above assumes you're using minikube, where NodePort is the correct service type; for a GKE cluser, you may prefer a different service type, such as a LoadBalancer.

The database is now available at $(minikube ip):30001, which we’ll compose with REMOTE_DB_HOST and REMOTE_DB_PORT variables.

The database has no data yet, but we can verify it works by logging in as the security admin and listing the users:

export SECURITY_ADMIN_USER=security_admin_user
export SECURITY_ADMIN_PASSWORD=security_admin_password
export REMOTE_DB_HOST=$(minikube ip)
export REMOTE_DB_PORT=30001

docker run --rm -it -e PGPASSWORD=${SECURITY_ADMIN_PASSWORD} postgres:9.6 \
  psql -U ${SECURITY_ADMIN_USER} "postgres://${REMOTE_DB_HOST}:${REMOTE_DB_PORT}/postgres" -c "\du"
                                          List of roles
       Role name        |                         Attributes                    
     | Member of
------------------------+-------------------------------------------------------
-----+-----------
 security_admin_user    | Superuser, Create role, Create DB, Replication, Bypass
 RLS | {}

Create Application Database

In this section, we assume the following:

  • You already have a PostgreSQL database exposed as a Kubernetes service.
  • It’s publicly available at $REMOTE_DB_HOST:$REMOTE_DB_PORT
  • You have admin-level database credentials
  • The SECURITY_ADMIN_USER and SECURITY_ADMIN_PASSWORD environment variables hold those credentials
If you're using your own database server and it's not SSL-enabled, please see the service connector documentation for how to disable SSL in your Secretless configuration.

If you followed along in the last section and are using minikube, you can run:

export SECURITY_ADMIN_USER=security_admin_user
export SECURITY_ADMIN_PASSWORD=security_admin_password
export REMOTE_DB_HOST="$(minikube ip)"
export REMOTE_DB_PORT="30001"

Next, we’ll create the application database and user, and securely store the user’s credentials:

  1. Create the application database
  2. Create the pets table in that database
  3. Create an application user with limited privileges: SELECT and INSERT on the pets table
  4. Store these database application-credentials in Kubernetes secrets.

So we can refer to them later, export the database name and application-credentials as environment variables:

export APPLICATION_DB_NAME=quick_start_db

export APPLICATION_DB_USER=app_user
export APPLICATION_DB_INITIAL_PASSWORD=app_user_password

Finally, to perform the 4 steps listed above, run:

docker run --rm -i -e PGPASSWORD=${SECURITY_ADMIN_PASSWORD} postgres:9.6 \
    psql -U ${SECURITY_ADMIN_USER} "postgres://${REMOTE_DB_HOST}:${REMOTE_DB_PORT}/postgres" \
    << EOSQL

CREATE DATABASE ${APPLICATION_DB_NAME};

/* connect to it */

\c ${APPLICATION_DB_NAME};

CREATE TABLE pets (
  id serial primary key,
  name varchar(256)
);

/* Create Application User */

CREATE USER ${APPLICATION_DB_USER} PASSWORD '${APPLICATION_DB_INITIAL_PASSWORD}';

/* Grant Permissions */

GRANT SELECT, INSERT ON public.pets TO ${APPLICATION_DB_USER};
GRANT USAGE, SELECT ON SEQUENCE public.pets_id_seq TO ${APPLICATION_DB_USER};
EOSQL
CREATE DATABASE
You are now connected to database "quick_start_db" as user "security_admin_user".
CREATE TABLE
CREATE ROLE
GRANT
GRANT

Create Application Namespace and Store Credentials

The application will be scoped to the quick-start-application-ns namespace.

To create the namespace run:

kubectl create namespace quick-start-application-ns
namespace "quick-start-application-ns" created

Next we’ll store the application-credentials in Kubernetes Secrets:

kubectl --namespace quick-start-application-ns \
  create secret generic quick-start-backend-credentials \
  --from-literal=host="${REMOTE_DB_HOST}" \
  --from-literal=port="${REMOTE_DB_PORT}" \
  --from-literal=username="${APPLICATION_DB_USER}" \
  --from-literal=password="${APPLICATION_DB_INITIAL_PASSWORD}"
secret "quick-start-backend-credentials" created
While Kubernetes Secrets are more secure than hard-coded ones, in a real deployment you should secure secrets in a fully-featured vault, like Conjur.

Create Secretless Broker Configuration ConfigMap

With our database ready and our credentials safely stored, we can now configure the Secretless Broker. We’ll tell it where to listen for connections and how to proxy them.

After that, the developer’s application can access the database without ever knowing the application-credentials.

A Secretless Broker configuration file defines the services that Secretless with authenticate to on behalf of your application.

To create secretless.yml in your current directory, run:

cat << EOF > secretless.yml
version: "2"
services:
  pets-pg:
    protocol: pg
    listenOn: tcp://localhost:5432
    credentials:
      host:
        from: kubernetes
        get: quick-start-backend-credentials#host
      port:
        from: kubernetes
        get: quick-start-backend-credentials#port
      username:
        from: kubernetes
        get: quick-start-backend-credentials#username
      password:
        from: kubernetes
        get: quick-start-backend-credentials#password
EOF

Here’s what this does:

  • Defines a service called pets-pg that listens for PostgreSQL connections on localhost:5432
  • Says that the database host, port, username and password are stored in Kubernetes Secrets
  • Lists the ids of those credentials within Kubernetes Secrets
This configuration is shared by all Secretless Broker sidecar containers. There is one Secretless sidecar in every application Pod replica.
Since we don't specify an sslmode in the Secretless Broker config, it will use the default require value.

Next we create a Kubernetes ConfigMap from this secretless.yml:

kubectl --namespace quick-start-application-ns \
  create configmap \
  quick-start-application-secretless-config \
  --from-file=secretless.yml
configmap "quick-start-application-secretless-config" created

Create Application Service Account and Grant Entitlements

To grant our application access to the credentials in Kubernetes Secrets, we’ll need a ServiceAccount:

kubectl --namespace quick-start-application-ns \
  create serviceaccount \
  quick-start-application
serviceaccount "quick-start-application" created

Next we grant this ServiceAccount “get” access to the quick-start-backend-credentials. This is a 2 step process:

  1. Create a Role with permissions to get the quick-start-backend-credentials secret
  2. Create a RoleBinding so our ServiceAccount has this Role

Run this command to perform both steps:

cat << EOF > quick-start-application-entitlements.yml
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: quick-start-backend-credentials-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["quick-start-backend-credentials"]
  verbs: ["get"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-quick-start-backend-credentials
subjects:
- kind: ServiceAccount
  name: quick-start-application
roleRef:
  kind: Role
  name: quick-start-backend-credentials-reader
  apiGroup: rbac.authorization.k8s.io
EOF

kubectl --namespace quick-start-application-ns \
  apply -f quick-start-application-entitlements.yml
role "quick-start-backend-credentials-reader" created
rolebinding "read-quick-start-backend-credentials" created

Up next...

As an Application Developer, you no longer need to worry about all the passwords and database connections! You will deploy an application and leave it up to the Secretless Broker to make the desired connection to the database.

Want to learn more? Check out our documentation for more information, like how to use Secretless Broker in your Kubernetes environment!