Completing the PKI: pgAdmin mTLS and the Vault Cleanup

The Third: Getting Easier

#

In the two previous posts I covered the mTLS migration of Grafana and mnemosyne. The first required five destroy+create cycles and half a HelmRelease repair; the second needed a Go refactor, code review, and a couple of rotation tests.

pgAdmin was supposed to be the simplest. And it was — but not without surprises.

What Was Already There

#

Unlike mnemosyne, where I had to modify Go code and add env vars and volume mounts from scratch, pgAdmin’s deployment already had everything it needed from earlier PKI project commits: the volume mount for the db-client-pgadmin-tls secret at /etc/pgadmin/certs and the environment variables PGSSLMODE=verify-full, PGSSLCERT, PGSSLKEY, PGSSLROOTCERT. The VaultPKISecret was already deployed with a 24h TTL.

Past me (during the PKI project) had already thought about pgAdmin. Only the finishing touches were missing.

What Was Missing

#

Four small things:

1. rolloutRestartTargets. The VaultPKISecret didn’t reference the pgAdmin deployment. When VSO rotates the certificate (every 24h), the secret changes but the pod doesn’t restart — the new certificate isn’t loaded until someone manually deletes the pod. Added — same pattern as mnemosyne.

2. defaultMode: 384. The secret volume mount didn’t specify permissions. The libpq library (which pgAdmin uses under the hood) requires the private key tls.key to have 0600 permissions — readable only by the owner. Without defaultMode: 384 (which is 0600 in octal), pgAdmin can’t use the key. A detail I’ve already encountered twice and worth keeping in mind.

3. PGSSLROOTCERT pointed to a non-existent file. The deployment said /etc/pgadmin/certs/ca_chain, but the file generated by VSO is called ca.crt. With excludeRaw: true in the template, VSO only generates tls.crt, tls.key, and ca.crt — not ca_chain. A silent error: pgAdmin starts, but SSL connections with CA verification fail. Fixed to /etc/pgadmin/certs/ca.crt.

4. The pg_hba rules. The only rules for pgadmin in PostgreSQL were md5 (password). I added hostssl postgres pgadmin ... cert to enable certificate authentication, followed by host postgres pgadmin ... reject to prevent password fallback. A pattern I learned from the mnemosyne review: if you don’t explicitly block password after cert, the generic host all all ... md5 rule still matches.

servers.json

#

The only new addition: a ConfigMap with a pre-configured servers.json. pgAdmin is a GUI — users must manually connect to the database each time. With this file, the “TazLab Database” server already appears in the list, with certificate paths pre-filled. No path hunting. No manual setup.

The Surprise: the Kustomization

#

The only real problem was trivial: I created the servers-configmap.yaml file but forgot to add it to kustomization.yaml. The new pod started, the init container stayed in PodInitializing for minutes, and the event log said:

MountVolume.SetUp failed for volume "pgadmin-config" : configmap "pgadmin-servers" not found

The ConfigMap didn’t exist because Flux wasn’t creating it. Adding one line to kustomization.yaml fixed it. A stupid oversight that blocked the rollout for a good ten minutes.

What This Means for the Infrastructure

#

With pgAdmin migrated, the PostgreSQL application landscape is:

And the Vault Database Engine? Removed.

#

The last item in the table — the Vault database engine — deserves a deeper look. When I wrote the first draft of this article, it was still an open question: keep it for future automation, or remove it?

I did an in-depth enterprise research, sourcing HashiCorp documentation, security best practices, and risk analyses. The answer was clear: remove it.

The database engine (database/) was enabled to generate dynamic PostgreSQL users — temporary passwords with leases, created on the fly and automatically rotated. It was the mechanism Grafana used before switching to client certificates. Now that all applications use mTLS, there are no consumers left.

Keeping it active would mean storing a PostgreSQL administrative credential (CREATEROLE) inside Vault for no reason. A potential privilege escalation vector with zero operational benefit. The research also highlighted a performance issue: the database engine runs continuous CREATE ROLE and DROP ROLE statements, fragmenting PostgreSQL’s system catalogs.

So yes: vault secrets disable database, lease revocation, removal of the vault-admin user from PostgreSQL. The Terraform configuration stays in the repository — recreating it in the future, if needed, takes seconds.

Three apps out of three on client certificates. Zero passwords. Zero unused engines.

This article is part of a series on TazLab infrastructure management. Previous posts: After the PKI: Migrating mnemosyne to mTLS, Vault PKI Follow-Up. Code on github.com/tazzo.