Troubleshooting & Best Practices

This guide provides troubleshooting steps for common error scenarios and highlights operational best practices for developers using PowerMind ComputeAI.

1. Connection & Session Errors

1.1 `Cannot connect to API`

Symptom: CLI commands return connection timeout errors.
Resolution:
1. Verify the Controller API URL is correct by running (see CLI Reference Manual for options):
```
pm-compute config view
```
2. Test route access with ping or curl -I <API_URL>.
3. If the platform is deployed using internal self-signed TLS certificates, ensure you configured the endpoint with the --insecure option:
```
pm-compute config set --api-url <URL> --insecure
```

1.2 `No user logged in` or `Unauthorized`

Symptom: Commands fail with authentication errors.
Resolution:
1. Confirm your session state (see CLI Reference Manual for session management details):
```
pm-compute whoami
```
2. If the session has expired or is inactive, log in again:
```
pm-compute login
```

2. Training Job Failures

2.1 Job fails instantly with `File not found`

Symptom: Job transitions to failed immediately upon submission.
Resolution:
1. Ensure your python script is loading datasets using the dynamically-provided environment variable DATASET_DIR or workspace relative path, rather than hardcoded absolute directories from your local workstation (see Runtime Variables & Contracts Guide for definitions and templates):
```
# Correct path loading
dataset_path = os.path.join(os.getenv("DATASET_DIR"), "input_data.csv")
```
2. Check that the dataset file actually exists in the workspace:
```
pm-compute file browse datasets
```

2.2 Job fails during execution

Symptom: Job status changes to failed after running for some time.
Resolution:
1. Inspect training logs to review tracebacks:
```
pm-compute logs <JOB_ID>
```
2. Verify that you requested the correct resource framework matches. For instance, do not request --resource gpu if your PyTorch code does not contain .cuda() or .to('cuda') device mapping logic.

3. Checkpoint & Resume Discrepancies

3.1 Job fails to resume

Symptom: Running pm-compute resume <JOB_ID> returns an error.
Resolution:
1. Verify the target job is in a failed or cancelled state. Jobs in completed or running state cannot be resumed.
2. Confirm that checkpoint files exist under your checkpoint folder:
```
pm-compute checkpoints <JOB_ID>
```
  If no checkpoints are listed, the job must be submitted as a new run.

3.2 Resume starts from epoch 0

Symptom: Job resumes, but loops over the full epochs sequence instead of continuing from the checkpoint epoch.

Resolution:

Verify your training loop integrates the platform-provided resume_epoch global variable (for integration instructions and templates, see Runtime Variables & Contracts Guide):

# Check global scope for resume epoch injected by platform
start_epoch = int(globals().get("resume_epoch", 0))
for epoch in range(start_epoch, total_epochs):
    # training steps...

4. Serving & Inference Errors

4.1 Deployment remains in `submitted` or `failed` state

Symptom: Model serving endpoint is unhealthy or cannot be queried.
Resolution:
1. Inspect deployment configurations by listing status details:
```
pm-compute deployments
```
2. Ensure that the input and output tensor shapes you declared during model registration match the actual architecture shapes in the weights file.

4.2 Inference call returns `401 Unauthorized` or `403 Forbidden`

Symptom: Prediction requests are rejected.
Resolution:
1. Confirm the X-API-Key header is present and contains the exact sk_... key.
2. Check that the key has not been revoked:
```
pm-compute keys list <DEPLOYMENT_ID>
```

5. End-User Best Practices Checklist

Always Use Path Environment Variables: Never hardcode physical local machine directories (e.g. /home/user/data). Use DATASET_DIR, CHECKPOINT_DIR, and JOB_OUTPUT_DIR.
Set Realistic Checkpoint Intervals: Choose a checkpoint frequency (e.g., every 5 epochs) that balances disk writing overhead with protection against cluster node dropouts.
Secure API Keys: Treat your inference API keys as highly sensitive credentials. Keep keys environment-scoped (dev/staging/production) and rotate them regularly.
Clean Up Resources: Stop or delete serving deployments and keys that are no longer needed to free up slots in the GPU pool.