Jobs Overview

Jobs are the fundamental unit of computation in Magpie Cloud. A job represents a script or command that runs in an isolated environment with specified compute resources. Jobs can be ephemeral, stateful, or persistent depending on your use case.

Job Types

Ephemeral Jobs (Default)

Ephemeral jobs run once and terminate automatically when the script completes. They start with a clean filesystem and do not persist any data after completion. Perfect for stateless tasks like batch processing, CI/CD pipelines, and data transformations.

Stateful Jobs

Stateful jobs preserve the /workspace directory between runs. Use stateful jobs when you need to cache data, store intermediate results, or maintain state across multiple executions. Enable with stateful=True.

Persistent VMs

Persistent VMs stay running until explicitly stopped. They provide long-lived compute environments for interactive development, services, and applications. Access them via SSH for interactive work. Enable with persistent=True.

jobs.create()

Create and start a new job with the specified configuration.

client.jobs.create(name, script, vcpus=None, memory_mb=None, stateful=False, persistent=False, ipv6_lease=None, env=None, timeout=None)

Parameters

Parameter Type Description
name required str A unique name for the job. Used for identification and retrieval.
script required str The bash script or command to execute in the job environment.
vcpus optional int Number of virtual CPUs to allocate. Defaults to 1.
memory_mb optional int Memory allocation in megabytes. Defaults to 512 MB.
stateful optional bool If True, preserves the /workspace directory between runs. Defaults to False.
persistent optional bool If True, keeps the VM running until explicitly stopped. Defaults to False.
ipv6_lease optional str IPv6 address lease identifier for networking configuration.
env optional dict Environment variables to set in the job environment as key-value pairs.
timeout optional int Maximum execution time in seconds. Job will be stopped if exceeded.

Returns

Returns a job object with properties including id, name, status, vcpus, memory_mb, and created_at.

Example

from magpie import Magpie

client = Magpie(api_key="your_api_key_here")

# Create an ephemeral job
job = client.jobs.create(
    name="data-processor",
    script="python process_data.py",
    vcpus=4,
    memory_mb=2048
)

print(f"Job ID: {job.id}")
print(f"Status: {job.status}")

# Create a stateful job with environment variables
stateful_job = client.jobs.create(
    name="ml-training",
    script="""
        pip install torch
        python train.py --epochs 100
    """,
    vcpus=8,
    memory_mb=16384,
    stateful=True,
    env={
        "MODEL_PATH": "/workspace/models",
        "DATASET": "imagenet"
    },
    timeout=3600
)

# Create a persistent VM
persistent_vm = client.jobs.create(
    name="dev-environment",
    script="echo 'VM ready for development'",
    vcpus=2,
    memory_mb=4096,
    persistent=True
)

jobs.run_and_wait()

Create a job and wait for it to complete, polling for status updates at regular intervals.

client.jobs.run_and_wait(name, script, vcpus=None, memory_mb=None, stateful=False, env=None, timeout=None, poll_interval=5)

Parameters

Parameter Type Description
name required str A unique name for the job.
script required str The bash script or command to execute.
vcpus optional int Number of virtual CPUs to allocate.
memory_mb optional int Memory allocation in megabytes.
stateful optional bool If True, preserves the /workspace directory between runs.
env optional dict Environment variables as key-value pairs.
timeout optional int Maximum execution time in seconds.
poll_interval optional int Seconds to wait between status checks. Defaults to 5.

Returns

Returns a completed job object with the final status (completed or failed) and all job details.

Example

from magpie import Magpie

client = Magpie(api_key="your_api_key_here")

# Run a job and wait for completion
job = client.jobs.run_and_wait(
    name="batch-report",
    script="""
        echo 'Generating report...'
        python generate_report.py
        echo 'Report complete!'
    """,
    vcpus=2,
    memory_mb=1024,
    poll_interval=10
)

print(f"Job finished with status: {job.status}")

# Get the logs after completion
logs = client.jobs.get_logs(job.id)
print(logs)

jobs.get()

Retrieve detailed information about a specific job by its ID.

client.jobs.get(job_id)

Parameters

Parameter Type Description
job_id required str The unique identifier of the job to retrieve.

Returns

Returns a job object containing all job details including configuration, status, timestamps, and resource allocation.

Example

from magpie import Magpie

client = Magpie(api_key="your_api_key_here")

# Get job details
job = client.jobs.get("job_abc123xyz")

print(f"Job Name: {job.name}")
print(f"Status: {job.status}")
print(f"vCPUs: {job.vcpus}")
print(f"Memory: {job.memory_mb} MB")
print(f"Created: {job.created_at}")
print(f"Stateful: {job.stateful}")
print(f"Persistent: {job.persistent}")

jobs.get_status()

Get the current status of a job. Returns one of five possible states.

client.jobs.get_status(job_id)

Parameters

Parameter Type Description
job_id required str The unique identifier of the job.

Returns

Returns a string representing the current job status:

  • pending - Job is queued and waiting to start
  • running - Job is currently executing
  • completed - Job finished successfully
  • failed - Job terminated with an error
  • stopped - Job was manually stopped

Example

from magpie import Magpie
import time

client = Magpie(api_key="your_api_key_here")

# Create a job
job = client.jobs.create(
    name="long-task",
    script="sleep 60 && echo 'Done!'",
    vcpus=1,
    memory_mb=512
)

# Poll for status
while True:
    status = client.jobs.get_status(job.id)
    print(f"Current status: {status}")

    if status in ["completed", "failed", "stopped"]:
        break

    time.sleep(5)

jobs.get_logs()

Retrieve the console output logs from a job's execution.

client.jobs.get_logs(job_id, tail=None)

Parameters

Parameter Type Description
job_id required str The unique identifier of the job.
tail optional int Number of lines to retrieve from the end of the logs. If not specified, returns all logs.

Returns

Returns a string containing the job's console output. Includes both stdout and stderr streams.

Example

from magpie import Magpie

client = Magpie(api_key="your_api_key_here")

# Get all logs
logs = client.jobs.get_logs("job_abc123xyz")
print("Complete logs:")
print(logs)

# Get only the last 50 lines
recent_logs = client.jobs.get_logs("job_abc123xyz", tail=50)
print("\nLast 50 lines:")
print(recent_logs)

# Monitor logs for a running job
job = client.jobs.create(
    name="log-generator",
    script="""
        for i in {1..10}; do
            echo "Processing item $i"
            sleep 2
        done
    """,
    vcpus=1,
    memory_mb=512
)

import time
time.sleep(5)

logs = client.jobs.get_logs(job.id)
print(f"Current output:\n{logs}")

jobs.ssh()

Get SSH connection details for a persistent job, allowing interactive terminal access.

client.jobs.ssh(job_id)

Parameters

Parameter Type Description
job_id required str The unique identifier of the persistent job.

Returns

Returns an object containing SSH connection details including host, port, username, and command (a ready-to-use SSH command string).

Note: SSH access is only available for persistent jobs (persistent=True). Attempting to SSH into ephemeral or stateful jobs will result in an error.

Example

from magpie import Magpie

client = Magpie(api_key="your_api_key_here")

# Create a persistent VM
vm = client.jobs.create(
    name="interactive-dev",
    script="echo 'VM ready'",
    vcpus=4,
    memory_mb=8192,
    persistent=True
)

# Wait for VM to be ready
import time
while client.jobs.get_status(vm.id) != "running":
    time.sleep(2)

# Get SSH connection details
ssh_info = client.jobs.ssh(vm.id)

print(f"Host: {ssh_info.host}")
print(f"Port: {ssh_info.port}")
print(f"Username: {ssh_info.username}")
print(f"\nConnect with:\n{ssh_info.command}")

# Example output:
# Connect with:
# ssh -p 2222 magpie@vm-abc123.magpie.cloud

jobs.stop()

Stop a running job immediately. The job status will change to "stopped".

client.jobs.stop(job_id)

Parameters

Parameter Type Description
job_id required str The unique identifier of the job to stop.

Returns

Returns a confirmation object indicating the job has been stopped successfully.

Example

from magpie import Magpie

client = Magpie(api_key="your_api_key_here")

# Create a long-running job
job = client.jobs.create(
    name="background-task",
    script="while true; do echo 'Working...'; sleep 10; done",
    vcpus=1,
    memory_mb=512,
    persistent=True
)

# Work with the job...
import time
time.sleep(30)

# Stop the job when done
result = client.jobs.stop(job.id)
print(f"Job stopped: {result}")

# Verify status
status = client.jobs.get_status(job.id)
print(f"Current status: {status}")  # Output: stopped

jobs.delete()

Permanently delete a job and all associated data including logs and workspace files.

client.jobs.delete(job_id)

Parameters

Parameter Type Description
job_id required str The unique identifier of the job to delete.

Returns

Returns a confirmation object indicating the job has been deleted successfully.

Warning: Deleting a job is permanent and cannot be undone. All logs, outputs, and workspace data will be permanently removed. For stateful jobs, this includes all files in the /workspace directory.

Example

from magpie import Magpie

client = Magpie(api_key="your_api_key_here")

# Create a temporary job
job = client.jobs.create(
    name="temp-calculation",
    script="echo 'Quick calculation'",
    vcpus=1,
    memory_mb=512
)

# Wait for completion
job = client.jobs.run_and_wait(
    name="temp-calculation",
    script="echo 'Quick calculation'",
    vcpus=1,
    memory_mb=512
)

# Get logs before deletion
logs = client.jobs.get_logs(job.id)
print(f"Final output: {logs}")

# Delete the job
result = client.jobs.delete(job.id)
print(f"Job deleted: {result}")

# Attempting to get the job now will raise an error
try:
    client.jobs.get(job.id)
except Exception as e:
    print(f"Error: {e}")  # Job not found

Next Steps