Skip to main content

Training Pipeline

This document describes the training pipeline for the Next Best Message, Search & Autocomplete feature. It includes the steps and processes involved in preparing and training models for production use.

Overview​

The training pipeline is designed to ensure that models are updated and optimized regularly. The pipeline includes data preprocessing, model training, evaluation, and deployment steps.

Steps in the Training Pipeline​

  1. Data Preprocessing

    • Raw data is collected from various sources.
    • Data is cleaned, normalized, and transformed into a format suitable for training.
    • Features are extracted and stored in a feature store.

  2. Model Training

    • Training jobs are scheduled and executed as Kubernetes Cron Jobs.
    • Models are trained using the preprocessed data and optimized for accuracy and performance.

  3. Model Evaluation

    • Trained models are evaluated against a validation dataset.
    • Metrics such as accuracy, precision, recall, and F1-score are calculated.
    • Models that meet the performance criteria are selected for deployment.

  4. Model Deployment

    • Selected models are deployed to the production environment.
    • Deployment involves copying the model to the servables bucket and updating the active model in the system.

Automation and Scheduling​

  • The training pipeline is automated using Kubernetes Cron Jobs.
  • Training jobs are scheduled to run bi-weekly to ensure models are up-to-date.
  • Notifications are sent to the admin team upon successful completion or failure of training jobs.

Diagram​

Additional Notes​

  • The training pipeline is monitored continuously to identify bottlenecks and improve efficiency.
  • Logs and metrics are collected for debugging and performance analysis.