This guide is for an already deployed service. Backup and restore are separate operations: if you only want a backup, do not run the restore command. Rehearse in isolation before planning a production switch.
Backup scope
Keep three things: MySQL configuration, Vault and credential state; the original Master Key and startup settings; and ClickHouse access/audit logs. The database backup cannot decrypt values without the key, so back up both but protect them separately. Transient NATS messages are not a configuration recovery source.
Protect backups with encryption, access controls, retention and off-site storage. SHA-256 manifests detect corruption, not independent authenticity. Keep the Master Key outside the backup archive and its access-credential scope.
1. Prepare tools and credentials
deploy/backup provides MySQL 8.0.22 tools. Install the matching 8.0.22 client tools and mount /run/secrets/mysql.cnf, for example:
[client]
host=mysql.example.internal
port=3306
protocol=tcp
user=configra_backup
password=replace-through-secret-volumeReplace the hostname and account, supply the password through a controlled Secret, and do not commit the real file. The backup account needs to read the selected database; the restore account needs to create, populate and remove the specified new target on failure. /backup must be a writable, persistent backup mount.
2. Back up MySQL
Start from the repository root and enter the tools directory. configra is the database being backed up. /backup must already be a persistent backup mount. Use a new empty directory for each backup:
cd deploy/backup
mkdir -m 700 /backup/new-point
sh mysql-backup.sh /run/secrets/mysql.cnf configra /backup/new-pointThe dump uses a consistent transaction, emits compressed SQL and a hash manifest, and refuses existing output. Restore validates the manifest and creates a new database rather than overwriting one. Do not run schema upgrades concurrently with the dump.
After success, the directory contains mysql.sql.gz and manifest.sha256. Use different backup/restore names for another run rather than deleting an existing recovery point to make it pass. These scripts are not a cross-version 8.0.22-to-8.4 migration tool.
3. Rehearse a restore into a new database
Still in deploy/backup, first verify the connection file points at the rehearsal server and uses an account with restore permissions. configra_restore_trial must be a new, unused database name, not the production database:
sh mysql-restore.sh /run/secrets/mysql.cnf /backup/new-point configra_restore_trialThen start an isolated Configra instance using that database and the original Master Key. Check /health/ready, real configuration reads and client certificates. Plan a switch only after those checks pass; copying backup files is not a verified recovery.
Readiness is not a full database decryption check. In v1.0.1, doctor --verify-vault checks historical Vault revisions, CA private keys and notification credentials. Follow deploy/backup/README.md in the repository to supply a read-only restore account and the matching Master Key. Check the results, then perform the real reads and certificate checks above. Do not initialize the database again to work around a decryption failure.
Changing the Master Key
Do not directly replace the configra-master-key Secret. Stored data still requires the old key. Replacing the Secret without rotating the data breaks decryption for newly started Pods; restarting existing Pods does not repair it. The Master Key is not an application password or client certificate, and replacing it cannot invalidate a stolen old backup.
In v1.0.1, rotate-master-key requires a maintenance window; it does not stop services or update Kubernetes Secrets for you. Use this order:
- Rehearse on an isolated restore, back up the database, and protect both keys separately. Record Management/API replica counts and suspend any HPA, GitOps or jobs that could restart them.
- Scale both services to zero and confirm every old Pod has exited before running the rotation command from the repository runbook. A maintenance Pod uses the same image version and needs only database access and separately mounted old/new keys; it does not serve traffic.
- Run
doctor --verify-vaultwith the new key and compare record counts. Keep services stopped on failure. If the command is interrupted or the commit outcome is unknown, check with each key instead of guessing or immediately retrying rotation. - After verification, update the externally managed Master Key Secret and restore the recorded replicas. Check real Config/File reads, the SDK, CA issuance and rotation Audit. Kubernetes consumers must receive a new configuration update; an old mounted file alone does not prove recovery.
- Back up the rotated database and verify the recovery point before resuming autoscaling and rollouts. Old backups still need the old key; do not destroy it early.
Use restartPolicy: Never for maintenance Pods, omit HTTP probes and do not give them the service selector labels. For a Job, also set backoffLimit: 0; Never alone still permits replacement Pods after failure. Do not retry automatically when the rotation outcome is unknown. See Kubernetes Job retries.
After a committed rotation, restoring the old Secret is not a rollback. Rotate in reverse during another offline window, or restore both the old database and its matching key; the latter may discard changes made after that backup.
Logs and upgrades
ClickHouse uses native backups restored to a new target. Verify row counts, retention and access before switching queries. Configuration reads can recover before historical logs are restored; Management still needs a reachable ClickHouse service.
Before upgrades, record the previous image/tag, database recovery point and Master Key recovery location. Upgrade Management, then API, then verify SDK / Kubernetes consumers. Do not assume an older binary accepts a newer schema.
Troubleshooting
| Symptom | Check first |
|---|---|
| Readiness fails | MySQL connectivity/state and whether the Master Key matches the database |
| OIDC login lacks permissions | Issuer, callback, trusted role Claim and Admin / Viewer mappings |
| TLS handshake fails | API hostname, server CA, client identity and TLS passthrough |
| A newly issued CA is not accepted immediately | Allow the roughly five-second public-trust refresh, then recheck credentials |
| Machine read is denied | Token Environment grants/expiry/revocation and client/issuer status |
| Vault reference fails | Four-part syntax, field type and values for the target Environment |
| Binding refuses a same-named target | Owner reference; the controller deliberately protects unrelated objects |
| Secret changed but application did not | File reload, subPath usage, or whether an envFrom Pod was replaced |
| CA appears but issuance fails | Master Key, CA status and validity; do not regenerate the Master Key as a repair |
Issue reports should include versions, Request IDs, status codes and sanitized structure, not live Tokens, private keys, Vault values or full raw responses.
Before production
- Run sustained load tests on the actual deployment; historical QPS is not a capacity guarantee.
- Rehearse recovery from backup plus Master Key and verify both configuration and certificates.
- Monitor expiry, database failures, synchronization failures and unavailability.
- Confirm that granting a whole environment does not let applications read data they should not share.
- Review limitations and security before choosing rollout and rollback procedures.