Skip to main content

Changing an account secret

The Deepdesk backend services deployed for a single account use a unique secret for hashing, generating and decoding JWTs, and encrypting sensitive data. It is possible to change the secret, but involves careful configuration, and cannot be done without some downtime.

Prerequisites​

  • You have gcloud, kubectl, kubectx, kubens installed.
  • You have access to the relevant Kubernetes cluster in Google Cloud, e.g. deepdesk-nl-production:
gcloud container clusters get-credentials deepdesk-services-v1 --region europe-west4-b --project deepdesk-nl-production
kubectx gke_deepdesk-nl-production_europe-west4-b_deepdesk-services-v1

Steps​

  • Start with noting down the current account secret for reuse later. It can be found in Secret Manager, in the format <account>_account_secret.
  • Create a new version for this secret, while disabling the previous version. Use a password manager like 1Password to create a strong, random password. Minimum length 32 characters.
info

We use External Secrets to sync secrets from Secret Manager to Kubernetes secrets, which in turn are read into environment variables for the containers we deploy. This syncing is currently configured to run every 10 minutes, so it will take some time before changes are reflected. Once the secrets are synced, we then need to restart all the services to load the new environment variables.

  • Now create a new secret <account>_account_secret_back_compat with the value that you just stored.
info

This backwards compatible secret will allow the backend to continue to read encrypted data from the database. Only when all encrypted fields have been saved again (which will use the new secret), can the backwards compatible secret be removed.

  • To check the current secret value, run the following commands:
kubens <account>
kubectl get secret admin-secrets -ojsonpath='{.data.SECRET_KEY}'  | base64 -d
  • If you don't want to wait for the regular sync, you can force the secrets to be updated:
kubectl annotate externalsecret admin-secrets force-sync=$(date +%s) --overwrite
kubectl annotate externalsecret backend-secrets force-sync=$(date +%s) --overwrite
  • Once the secrets have been updated, restart the services (this will break the JWTs currently in use):
kubectl rollout restart deployments/admin
kubectl rollout restart deployments/backend