Skip to content

End-to-End MLOps Workflow

This guide outlines the end-to-end machine learning lifecycle on PowerMind ComputeAI: preparing datasets, authoring training scripts, executing training workloads, tracking live metrics, inspecting checkpoints, and deploying models to high-performance inference serving.

1. MLOps Lifecycle Flow

The diagram below illustrates the standard progression from data preparation to real-time inference:

sequenceDiagram
    autonumber
    actor User as ML Engineer / Data Scientist
    participant UI as Web UI / CLI Client
    participant Controller as Central Controller
    participant Work as Worker Node Pool
    participant Tracker as Experiment Tracker
    participant Serve as Inference Serving Engine

    User->>UI: 1. Ingest Datasets into Workspace
    UI->>Controller: Upload & Extract Datasets into $DATASET_DIR

    User->>UI: 2. Submit Training Script
    Note over UI,Controller: Direct transmission — no separate code pre-upload required
    UI->>Controller: Dispatch Job (Script, Framework, GPU/CPU Target)
    Controller->>Work: Allocate compute slot & inject runtime environment

    activate Work
    Work->>Work: Run training loop with autowrapper
    Work->>Tracker: Stream training metrics & loss curves
    Work->>Controller: Persist epoch checkpoints to $CHECKPOINT_DIR
    deactivate Work

    User->>UI: 3. Monitor Live Progress & Inspect Checkpoints
    UI->>Tracker: Display real-time loss, accuracy, and console output

    User->>UI: 4. Register & Deploy Model
    UI->>Controller: Deploy trained model (or BYOB model)
    Controller->>Serve: Provision Triton inference container
    Serve-->>Controller: Return active endpoint URL

    User->>UI: 5. Generate Deployment API Key
    UI->>Controller: Create scoped API key
    Controller-->>User: Return secure token

    User->>Serve: 6. Execute Real-Time Inference
    Note over User,Serve: Send structured JSON tensor payload with Bearer token
    Serve-->>User: Return output tensor predictions

2. Detailed Lifecycle Stages

Phase 1: Workspace & Dataset Ingestion

Before running training jobs, upload your training corpora (CSVs, image directories, TFRecords, or compressed .tar.gz/.zip archives) to your workspace. - Compressed archives can be automatically decompressed upon upload directly into $DATASET_DIR. - Datasets are stored in your tenant-isolated directory, accessible to all worker nodes scheduled for your job. - You can manage datasets using either the interactive Web UI Workspace or the CLI File Commands.

Phase 2: Authoring the Training Script

When authoring scripts for ComputeAI, use the standard environment variables injected by the platform at runtime:

  • Datasets: Read training batches from $DATASET_DIR.
  • Outputs: Save final evaluation plots, metric summaries, and exported models to $JOB_OUTPUT_DIR.
  • Resume Epoch: Access resume_epoch (or COMPUTEAI_RESUME_EPOCH) to determine the starting loop index if resuming a run.
import os
import torch
import torch.nn as nn
import torch.optim as optim

# 1. Resolve Runtime Paths
workspace = os.getenv("USER_WORKSPACE", "./workspace")
dataset_dir = os.getenv("DATASET_DIR", os.path.join(workspace, "datasets"))
output_dir = os.getenv("JOB_OUTPUT_DIR", os.path.join(workspace, "outputs"))
os.makedirs(output_dir, exist_ok=True)

# 2. Extract Starting Epoch (0 if fresh run, >0 if resumed)
start_epoch = int(globals().get("resume_epoch", os.getenv("COMPUTEAI_RESUME_EPOCH", 0)))
total_epochs = start_epoch + 20

# 3. Model & Optimizer
model = nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10))
optimizer = optim.Adam(model.parameters(), lr=0.001)

# 4. Training Loop (Begins cleanly from start_epoch + 1)
# Note: ComputeAI autowrapper AUTOMATICALLY persists epoch checkpoints
print(f"Training from epoch {start_epoch + 1} to {total_epochs}...")
for epoch in range(start_epoch + 1, total_epochs + 1):
    # [Insert model training forward/backward steps here]
    loss = 0.25 / (epoch ** 0.5)
    acc = 80.0 + (1.2 * epoch)

    # stdout is automatically parsed for live UI dashboard graphs
    print(f"Epoch {epoch}/{total_epochs} | Loss: {loss:.4f} | Accuracy: {acc:.2f}%")

[!TIP] No Separate Code Upload Needed: You do not need to pre-upload your Python script to the workspace before submitting. The platform transmits your local .py file content directly upon job dispatch.

Phase 3: Workload Dispatch & Execution

Submit the training job specifying:

  • Framework: PyTorch or TensorFlow.
  • Hardware Target: CPU or GPU (with automatic CUDA device binding).
  • Node Topology: Single-node or multi-node distributed data parallelism.
  • Checkpoint Interval: Frequency (in epochs) for saving state.

Workloads can be dispatched via the Web UI Training Studio or using the CLI Train Command.

Phase 4: Observability, Metrics & Checkpoints

While your job executes:

  1. Live Loss & Accuracy Tracking: The platform autowrapper parses standard stdout formatting and plots live loss and accuracy curves in real time.
  2. Log Streaming: View complete stdout/stderr execution streams directly in the Web UI log modal or via terminal log streaming.
  3. Checkpoint Verification: Inspect saved epoch checkpoints, validation scores, and file sizes in the checkpoints browser.

Phase 5: Model Registration & Inference Deployment

Once training completes (or when importing an external pre-trained model via Bring Your Own Model):

  1. Register Model: Models exported to $JOB_OUTPUT_DIR or registered via BYOB are cataloged in the centralized Model Registry.
  2. Deploy Serving Endpoint: Provision an isolated Triton inference service on dedicated compute worker nodes.
  3. Generate Scoped API Keys: Create cryptographically secure, deployment-scoped tokens to authenticate client applications.

Phase 6: Real-Time Prediction Requests

Client applications send HTTP POST inference payloads containing input tensor structures:

curl -X POST "http://<COMPUTEAI_HOST>:3000/api/tools/computeai/project/<PROJECT_ID>/infer/<DEPLOYMENT_NAME>" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer <API_KEY>" \
     -d '{
       "inputs": [
         {
           "name": "INPUT__0",
           "shape": [1, 10],
           "datatype": "FP32",
           "data": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
         }
       ]
     }'

Next: CLI Command Reference | Back: Core Features