Skip to main content

Developer Guide

This guide provides technical details about the onboarding architecture, services, and deployment process.

Architecture​

We manage the provisioning and deployment of all accounts using two components:

  • Onboarding, with global account config
  • Admin, with account-specific config

The following diagram illustrates the role of Onboarding and Admin within the overall architecture. A detailed draw.io version of this architecture is available; see the Outline export or ask the team for the link.

Onboarding Service​

We maintain a global registry of accounts in a service called Onboarding (repo), which is only accessible for internal staff. In this registry we maintain global account config, like in which project an account should be deployed, and whether it is active.

The Onboarding service runs outside of the account production clusters (in Cloud Run), since it contains global config needed for all accounts in all regional clusters.

Admin Service​

Admin is deployed for a single account, in one regional cluster, within an isolated account-specific namespace. Admin contains all the config and credentials needed to operate the internal services Backend and Engine, and integrate with the CX Platform.

Deployment Sequence​

The following diagram shows the steps taken to provision and deploy an account. Manual steps (performed by Deepdesk staff) are indicated with note annotations.

Sequence diagram source (sequencediagram.org)​

To generate or edit the sequence diagram at sequencediagram.org, paste the source below. It includes Kubeflow Pipelines; manual steps are shown as Deepdesk-:4>.

title Onboarding

actor Deepdesk
participant Onboarding
participant "deepdesk-config" as deepdesk-config
participant Cloud Storage
participant Cloud Function
participant Cloud Build
participant GKE
participant FluxCD
participant "Kubeflow Pipelines" as Kubeflow
participant Admin

Deepdesk->>Onboarding: Create account
Onboarding->>deepdesk-config: Commit config
Onboarding->>Cloud Storage: Export accounts.json
Onboarding->>Cloud Function: Trigger
Cloud Function->>Cloud Build: Provision Account
Cloud Build->>GKE: Create namespace, secrets, etc
Cloud Build->>GKE: Install FluxCD resources
GKE->>FluxCD: Run controllers
FluxCD->>GKE: Helm install
GKE->>Admin: Run
Cloud Function->>Cloud Build: Run Kubeflow Deployment
Cloud Build->>Kubeflow: Deploy pipelines
Deepdesk->>Admin: Provision
Admin->>Cloud Storage: Export account.json
Admin->>Cloud Function: Trigger
Cloud Function->>Cloud Build: Provision Account
Cloud Build->>GKE: Update
Cloud Function->>Cloud Build: Run Kubeflow Deployment
Cloud Build->>Kubeflow: Deploy pipelines
Deepdesk->>Admin: Deploy
Admin->>deepdesk-config: Commit config
deepdesk-config->>FluxCD: Trigger deployment
CI/CD and FluxCD

The workflow below and the Key Components section describe the role of FluxCD and Cloud Build. For a step-by-step operator guide, see the User Guide.

Key Components​

The workflow below and the Key Components section describe the role of FluxCD and Cloud Build. For a step-by-step operator guide, see the User Guide. :::

Key Components​

Cloud Storage​

The Onboarding service exports account configuration to Cloud Storage as accounts.json, which is then consumed by Cloud Functions to trigger provisioning workflows.

Cloud Functions​

Cloud Functions orchestrate the provisioning and deployment process by invoking Cloud Build jobs. They are triggered via API calls from the Admin service during deployment.

Cloud Build​

Cloud Build runs Terraform to perform the following tasks:

  • Creating Kubernetes namespaces and secrets
  • Installing FluxCD resources

GKE (Google Kubernetes Engine)​

GKE hosts the account-specific namespaces where Admin and other services are deployed.

FluxCD​

FluxCD is used for GitOps-based continuous deployment. It monitors the deepdesk-config repository and automatically deploys changes to the cluster. The Helm install process includes machine learning pipelines as part of the deployment.

Workflow Steps​

Manual Step 1: Create Account​

  1. Account Creation: Deepdesk staff creates an account in the Onboarding service
  2. Config Commit: Onboarding commits configuration to deepdesk-config repository
  3. Export: Account configuration is exported to Cloud Storage
  4. Trigger: Cloud Function is triggered by the export
  5. Provision: Cloud Build provisions the account infrastructure (namespace, secrets, FluxCD resources)
  6. Install Services: FluxCD installs services via Helm charts (including machine learning pipelines)
  7. Admin Ready: Admin service is running and ready for configuration

Manual Step 2: Deploy​

  1. Deploy Trigger: Deepdesk staff triggers the Deploy action in Admin
  2. Infrastructure Update:
    • Admin exports account configuration to Cloud Storage
    • Admin makes an API call to Cloud Function to trigger deployment
    • Cloud Function triggers Cloud Build to provision and update account infrastructure
    • Cloud Build updates GKE infrastructure
  3. Config Deployment:
    • Admin commits configuration changes to deepdesk-config repository
    • FluxCD detects changes and triggers deployment
    • Services are deployed with new configuration