Skip to content

Runtime Variables & True Resume Platform Contract


This document serves as the technical reference for runtime environment variables, workspace file system paths, and the automated checkpoint & true resume protocol on PowerMind ComputeAI.


1. Platform Directory Variables

When a training job executes on a worker node, ComputeAI automatically mounts your workspace directories and injects standard environment variables into your process environment.

┌───────────────────────────────────────────────────────────────────────────┐
│                      CONTAINER RUNTIME ENVIRONMENT                        │
│                                                                           │
│   $USER_WORKSPACE  ────────► /workspace (Root user folder)                │
│   $DATASET_DIR     ────────► /workspace/datasets (Read training data)     │
│   $CHECKPOINT_DIR  ────────► /workspace/checkpoints (Save epoch states)   │
│   $JOB_OUTPUT_DIR  ────────► /workspace/output (Save final metrics/logs)  │
│   $CODE_DIR        ────────► /workspace/code (Supporting user modules)    │
│                                                                           │
│   $COMPUTEAI_RESUME_EPOCH  ──► Starting epoch integer (0 if fresh run)    │
│   resume_epoch             ──► Injected global variable for quick access  │
└───────────────────────────────────────────────────────────────────────────┘

Core Directory Reference:

Variable Name Description Default Fallback Pattern
USER_WORKSPACE Absolute root directory of the user's workspace on the worker node. os.getenv("USER_WORKSPACE", "./workspace")
DATASET_DIR Absolute path containing uploaded datasets (<workspace>/datasets). os.getenv("DATASET_DIR", os.path.join(workspace, "datasets"))
CHECKPOINT_DIR Dedicated directory for saving and loading epoch weights (<workspace>/checkpoints). os.getenv("CHECKPOINT_DIR", os.path.join(workspace, "checkpoints"))
JOB_OUTPUT_DIR Directory reserved for saving final model artifacts, metrics summaries, plots, and logs. os.getenv("JOB_OUTPUT_DIR", "./outputs")
CODE_DIR Path to supporting modules and libraries in the workspace (<workspace>/code). os.getenv("CODE_DIR", os.path.join(workspace, "code"))

2. Checkpoint & True Resume Contract

PowerMind ComputeAI provides resilience against hardware faults, worker node drops, preemption, or deliberate multi-phase training runs through an automated checkpoint/resume protocol.

graph TD
    JobStart["Training Job Initial Run"] --> EpochLoop["Epoch Loop (1 to N)"]
    EpochLoop --> CheckpointSave["Save Periodic Weights to CHECKPOINT_DIR"]
    CheckpointSave -->|Naming Rule| FileRule["checkpoint_epoch_X.pth (PyTorch)<br/>checkpoint_epoch_X.h5 (TensorFlow)"]

    Fault["Job Paused / Resumed"] -.-> ResumeCmd["User triggers: pm-compute resume <JOB_ID> or UI Resume"]
    ResumeCmd --> ScanCheckpoints["ComputeAI Scans CHECKPOINT_DIR for latest checkpoint_epoch_X"]
    ScanCheckpoints --> InjectVars["Injects COMPUTEAI_RESUME_EPOCH=X & resume_epoch=X"]
    InjectVars --> ContinuedRun["Script restores model + optimizer and resumes from epoch X + 1"]

2.1 Checkpoint Naming Standard

To enable automatic checkpoint discovery and seamless resume indexing, your training script must save epoch weights using the standard naming format:

  • PyTorch: checkpoint_epoch_<N>.pth or checkpoint_epoch_<N>.pt
  • TensorFlow / Keras: checkpoint_epoch_<N>.h5 or checkpoint_epoch_<N>.keras

(Where <N> is the 1-indexed epoch integer, e.g. checkpoint_epoch_5.pth, checkpoint_epoch_10.pth).

2.2 Resume Variables Injected by Platform:

Variable Name Type Description
COMPUTEAI_RESUME_EPOCH int The detected starting epoch integer when resuming a job (0 if starting fresh).
resume_epoch int Python global namespace variable injected by the runner for quick access.

2.3 Resume Value Resolution Hierarchy

When resuming a job, your script retrieves the starting epoch using the following precedence:

# Check global namespace injected by platform, then OS environment, then fallback to 0
start_epoch = int(globals().get("resume_epoch", os.getenv("COMPUTEAI_RESUME_EPOCH", 0)))

3. Production Helper Templates

3.1 Training Script Template (PyTorch)

Save this template locally as train_pytorch.py and submit with pm-compute train ./train_pytorch.py --framework pytorch --resource gpu --name pytorch_run:

import os
import torch
import torch.nn as nn
import torch.optim as optim

def main():
    # 1. Workspace and Directory Resolution
    workspace = os.getenv("USER_WORKSPACE", "./workspace")
    dataset_dir = os.getenv("DATASET_DIR", os.path.join(workspace, "datasets"))
    checkpoint_dir = os.getenv("CHECKPOINT_DIR", os.path.join(workspace, "checkpoints"))
    output_dir = os.getenv("JOB_OUTPUT_DIR", os.path.join(workspace, "outputs"))

    os.makedirs(checkpoint_dir, exist_ok=True)
    os.makedirs(output_dir, exist_ok=True)

    # 2. Select Compute Device (GPU if available)
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print(f"Using compute device: {device}")

    # 3. Define Model, Loss, Optimizer
    model = nn.Sequential(
        nn.Linear(784, 256),
        nn.ReLU(),
        nn.Dropout(0.2),
        nn.Linear(256, 10)
    ).to(device)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001)

    # 4. Resume Contract: Extract starting epoch (injected automatically on resume)
    start_epoch = int(globals().get("resume_epoch", os.getenv("COMPUTEAI_RESUME_EPOCH", 0)))
    total_epochs = start_epoch + 20

    if start_epoch > 0:
        print(f"✓ Resuming training run starting from epoch {start_epoch + 1}")
    else:
        print("✓ Starting fresh training run from epoch 1")

    # 5. Training Loop: Begins from start_epoch + 1
    # Note: ComputeAI autowrapper AUTOMATICALLY persists epoch checkpoints to CHECKPOINT_DIR
    print(f"Executing training loop from epoch {start_epoch + 1} to {total_epochs}...")
    for epoch in range(start_epoch + 1, total_epochs + 1):
        model.train()

        # [Insert data loader batch training step here]
        mock_loss = 0.5 / (epoch ** 0.5)
        mock_acc = 0.70 + (0.014 * epoch)

        # Standard stdout log formatting captured by ComputeAI monitoring:
        print(f"Epoch {epoch}/{total_epochs} | Loss: {mock_loss:.4f} | Accuracy: {mock_acc:.4f}")

    # 6. Final Model Export
    final_model_path = os.path.join(output_dir, "model_final.pt")
    torch.save(model.state_dict(), final_model_path)
    print(f"✓ Training complete. Final model saved to: {final_model_path}")

if __name__ == "__main__":
    main()

3.2 Training Script Template (TensorFlow / Keras)

Save this template locally as train_tf.py and submit with pm-compute train ./train_tf.py --framework tensorflow --resource gpu --name tf_run:

import os
import tensorflow as tf

def main():
    # 1. Workspace Path Resolution
    workspace = os.getenv("USER_WORKSPACE", "./workspace")
    dataset_dir = os.getenv("DATASET_DIR", os.path.join(workspace, "datasets"))
    output_dir = os.getenv("JOB_OUTPUT_DIR", os.path.join(workspace, "outputs"))
    os.makedirs(output_dir, exist_ok=True)

    # 2. Build Model
    model = tf.keras.Sequential([
        tf.keras.layers.Input(shape=(784,)),
        tf.keras.layers.Dense(256, activation='relu'),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10, activation='softmax')
    ])

    model.compile(
        optimizer='adam',
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy']
    )

    # 3. Resume Contract: Extract starting epoch (injected automatically on resume)
    start_epoch = int(globals().get("resume_epoch", os.getenv("COMPUTEAI_RESUME_EPOCH", 0)))
    total_epochs = start_epoch + 20

    if start_epoch > 0:
        print(f"✓ Resuming training from epoch {start_epoch + 1}")
    else:
        print("✓ Starting fresh training run from epoch 1")

    # 4. Execute Training (initial_epoch handles resume index cleanly)
    # model.fit(dataset, epochs=total_epochs, initial_epoch=start_epoch)
    print(f"✓ Ready to train TensorFlow model from epoch {start_epoch + 1} to {total_epochs}")

if __name__ == "__main__":
    main()

4. Experiment Tracking & Metric Logging Contract

The platform autowrapper parses terminal output and syncs evaluation parameters with the centralized tracking server.

To guarantee that metrics appear in the live dashboard charts: 1. Print metrics in a structured Key=Value or Key: Value format (e.g. Loss: 0.1245 | Accuracy: 0.9850). 2. Alternatively, log metrics directly via MLflow Python API (the tracking URI and run ID are pre-configured automatically in the container environment):

import mlflow

# Track metrics directly during training loop
mlflow.log_metric("train_loss", loss_value, step=current_epoch)
mlflow.log_metric("val_accuracy", val_acc, step=current_epoch)


Back: Troubleshooting Guide