Runtime Variables & True Resume Platform Contract
This document serves as the technical reference for runtime environment variables, workspace file system paths, and the automated checkpoint & true resume protocol on PowerMind ComputeAI.
1. Platform Directory Variables
When a training job executes on a worker node, ComputeAI automatically mounts your workspace directories and injects standard environment variables into your process environment.
┌───────────────────────────────────────────────────────────────────────────┐
│ CONTAINER RUNTIME ENVIRONMENT │
│ │
│ $USER_WORKSPACE ────────► /workspace (Root user folder) │
│ $DATASET_DIR ────────► /workspace/datasets (Read training data) │
│ $CHECKPOINT_DIR ────────► /workspace/checkpoints (Save epoch states) │
│ $JOB_OUTPUT_DIR ────────► /workspace/output (Save final metrics/logs) │
│ $CODE_DIR ────────► /workspace/code (Supporting user modules) │
│ │
│ $COMPUTEAI_RESUME_EPOCH ──► Starting epoch integer (0 if fresh run) │
│ resume_epoch ──► Injected global variable for quick access │
└───────────────────────────────────────────────────────────────────────────┘
Core Directory Reference:
| Variable Name | Description | Default Fallback Pattern |
|---|---|---|
USER_WORKSPACE |
Absolute root directory of the user's workspace on the worker node. | os.getenv("USER_WORKSPACE", "./workspace") |
DATASET_DIR |
Absolute path containing uploaded datasets (<workspace>/datasets). |
os.getenv("DATASET_DIR", os.path.join(workspace, "datasets")) |
CHECKPOINT_DIR |
Dedicated directory for saving and loading epoch weights (<workspace>/checkpoints). |
os.getenv("CHECKPOINT_DIR", os.path.join(workspace, "checkpoints")) |
JOB_OUTPUT_DIR |
Directory reserved for saving final model artifacts, metrics summaries, plots, and logs. | os.getenv("JOB_OUTPUT_DIR", "./outputs") |
CODE_DIR |
Path to supporting modules and libraries in the workspace (<workspace>/code). |
os.getenv("CODE_DIR", os.path.join(workspace, "code")) |
2. Checkpoint & True Resume Contract
PowerMind ComputeAI provides resilience against hardware faults, worker node drops, preemption, or deliberate multi-phase training runs through an automated checkpoint/resume protocol.
graph TD
JobStart["Training Job Initial Run"] --> EpochLoop["Epoch Loop (1 to N)"]
EpochLoop --> CheckpointSave["Save Periodic Weights to CHECKPOINT_DIR"]
CheckpointSave -->|Naming Rule| FileRule["checkpoint_epoch_X.pth (PyTorch)<br/>checkpoint_epoch_X.h5 (TensorFlow)"]
Fault["Job Paused / Resumed"] -.-> ResumeCmd["User triggers: pm-compute resume <JOB_ID> or UI Resume"]
ResumeCmd --> ScanCheckpoints["ComputeAI Scans CHECKPOINT_DIR for latest checkpoint_epoch_X"]
ScanCheckpoints --> InjectVars["Injects COMPUTEAI_RESUME_EPOCH=X & resume_epoch=X"]
InjectVars --> ContinuedRun["Script restores model + optimizer and resumes from epoch X + 1"]
2.1 Checkpoint Naming Standard
To enable automatic checkpoint discovery and seamless resume indexing, your training script must save epoch weights using the standard naming format:
- PyTorch:
checkpoint_epoch_<N>.pthorcheckpoint_epoch_<N>.pt - TensorFlow / Keras:
checkpoint_epoch_<N>.h5orcheckpoint_epoch_<N>.keras
(Where <N> is the 1-indexed epoch integer, e.g. checkpoint_epoch_5.pth, checkpoint_epoch_10.pth).
2.2 Resume Variables Injected by Platform:
| Variable Name | Type | Description |
|---|---|---|
COMPUTEAI_RESUME_EPOCH |
int |
The detected starting epoch integer when resuming a job (0 if starting fresh). |
resume_epoch |
int |
Python global namespace variable injected by the runner for quick access. |
2.3 Resume Value Resolution Hierarchy
When resuming a job, your script retrieves the starting epoch using the following precedence:
# Check global namespace injected by platform, then OS environment, then fallback to 0
start_epoch = int(globals().get("resume_epoch", os.getenv("COMPUTEAI_RESUME_EPOCH", 0)))
3. Production Helper Templates
3.1 Training Script Template (PyTorch)
Save this template locally as train_pytorch.py and submit with pm-compute train ./train_pytorch.py --framework pytorch --resource gpu --name pytorch_run:
import os
import torch
import torch.nn as nn
import torch.optim as optim
def main():
# 1. Workspace and Directory Resolution
workspace = os.getenv("USER_WORKSPACE", "./workspace")
dataset_dir = os.getenv("DATASET_DIR", os.path.join(workspace, "datasets"))
checkpoint_dir = os.getenv("CHECKPOINT_DIR", os.path.join(workspace, "checkpoints"))
output_dir = os.getenv("JOB_OUTPUT_DIR", os.path.join(workspace, "outputs"))
os.makedirs(checkpoint_dir, exist_ok=True)
os.makedirs(output_dir, exist_ok=True)
# 2. Select Compute Device (GPU if available)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using compute device: {device}")
# 3. Define Model, Loss, Optimizer
model = nn.Sequential(
nn.Linear(784, 256),
nn.ReLU(),
nn.Dropout(0.2),
nn.Linear(256, 10)
).to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
# 4. Resume Contract: Extract starting epoch (injected automatically on resume)
start_epoch = int(globals().get("resume_epoch", os.getenv("COMPUTEAI_RESUME_EPOCH", 0)))
total_epochs = start_epoch + 20
if start_epoch > 0:
print(f"✓ Resuming training run starting from epoch {start_epoch + 1}")
else:
print("✓ Starting fresh training run from epoch 1")
# 5. Training Loop: Begins from start_epoch + 1
# Note: ComputeAI autowrapper AUTOMATICALLY persists epoch checkpoints to CHECKPOINT_DIR
print(f"Executing training loop from epoch {start_epoch + 1} to {total_epochs}...")
for epoch in range(start_epoch + 1, total_epochs + 1):
model.train()
# [Insert data loader batch training step here]
mock_loss = 0.5 / (epoch ** 0.5)
mock_acc = 0.70 + (0.014 * epoch)
# Standard stdout log formatting captured by ComputeAI monitoring:
print(f"Epoch {epoch}/{total_epochs} | Loss: {mock_loss:.4f} | Accuracy: {mock_acc:.4f}")
# 6. Final Model Export
final_model_path = os.path.join(output_dir, "model_final.pt")
torch.save(model.state_dict(), final_model_path)
print(f"✓ Training complete. Final model saved to: {final_model_path}")
if __name__ == "__main__":
main()
3.2 Training Script Template (TensorFlow / Keras)
Save this template locally as train_tf.py and submit with pm-compute train ./train_tf.py --framework tensorflow --resource gpu --name tf_run:
import os
import tensorflow as tf
def main():
# 1. Workspace Path Resolution
workspace = os.getenv("USER_WORKSPACE", "./workspace")
dataset_dir = os.getenv("DATASET_DIR", os.path.join(workspace, "datasets"))
output_dir = os.getenv("JOB_OUTPUT_DIR", os.path.join(workspace, "outputs"))
os.makedirs(output_dir, exist_ok=True)
# 2. Build Model
model = tf.keras.Sequential([
tf.keras.layers.Input(shape=(784,)),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(
optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
# 3. Resume Contract: Extract starting epoch (injected automatically on resume)
start_epoch = int(globals().get("resume_epoch", os.getenv("COMPUTEAI_RESUME_EPOCH", 0)))
total_epochs = start_epoch + 20
if start_epoch > 0:
print(f"✓ Resuming training from epoch {start_epoch + 1}")
else:
print("✓ Starting fresh training run from epoch 1")
# 4. Execute Training (initial_epoch handles resume index cleanly)
# model.fit(dataset, epochs=total_epochs, initial_epoch=start_epoch)
print(f"✓ Ready to train TensorFlow model from epoch {start_epoch + 1} to {total_epochs}")
if __name__ == "__main__":
main()
4. Experiment Tracking & Metric Logging Contract
The platform autowrapper parses terminal output and syncs evaluation parameters with the centralized tracking server.
To guarantee that metrics appear in the live dashboard charts:
1. Print metrics in a structured Key=Value or Key: Value format (e.g. Loss: 0.1245 | Accuracy: 0.9850).
2. Alternatively, log metrics directly via MLflow Python API (the tracking URI and run ID are pre-configured automatically in the container environment):
import mlflow
# Track metrics directly during training loop
mlflow.log_metric("train_loss", loss_value, step=current_epoch)
mlflow.log_metric("val_accuracy", val_acc, step=current_epoch)