Build a Parallel Job Runner in Python: Step-by-Step Guide (2026)

Running multiple tasks in parallel can significantly enhance the efficiency of your applications, especially when dealing with I/O-bound or high-latency operations. In this tutorial, we will build a simple yet effective parallel job runner in Python. This tool will read a list of shell jobs from a configuration file, execute them in parallel with a maximum concurrency limit, and generate a JSON report detailing the outcomes. Notably, the process will exit with a non-zero status if any job fails, ensuring robust error handling.

Key Takeaways

• Learn how to run shell commands in parallel using Python's concurrent.futures library.
• Implement job management with a concurrency limit to control execution.
• Create a JSON report summarizing job results for easy debugging.
• Ensure the process exits with an appropriate status based on job outcomes.

Python's concurrent.futures module provides a high-level interface for asynchronously executing callables, making it ideal for our task. By the end of this tutorial, you'll understand how to combine this module with shell commands to build a practical and efficient job runner.

Prerequisites

• Basic knowledge of Python programming (Python 3.10 or above recommended).
• Familiarity with shell commands and JSON.
• Python installed on your system.
• A code editor or IDE of your choice.

Step 1: Install Necessary Python Packages

To start, ensure you have Python installed on your system. We will primarily use Python’s standard library, so no additional packages are required. However, ensure your Python version is up to date to leverage the latest features.

Step 2: Create a Configuration File for Jobs

First, we need a configuration file containing the list of shell jobs to execute. This file will be in JSON format, listing each job as a command string.

{
  "jobs": [
    "echo 'Job 1'",
    "echo 'Job 2'",
    "echo 'Job 3'"
  ]
}

Save this file as jobs_config.json.

Step 3: Read and Parse the Configuration File

In this step, we will write a Python script to read the job configuration file. This will allow us to dynamically handle various job sets without hardcoding them into our script.

import json

with open('jobs_config.json', 'r') as file:
    config = json.load(file)

jobs = config['jobs']
print("Loaded jobs:", jobs)

This script reads the JSON file and extracts the list of jobs, preparing it for execution.

Step 4: Execute Jobs in Parallel

Next, we will use the concurrent.futures module to run our jobs in parallel. We will limit the concurrency to a maximum of 3 jobs at a time.

import subprocess
from concurrent.futures import ThreadPoolExecutor

MAX_CONCURRENT_JOBS = 3

# Function to execute a single job
def execute_job(command):
    try:
        result = subprocess.run(command, shell=True, capture_output=True, text=True, check=True)
        return {"command": command, "status": "success", "output": result.stdout}
    except subprocess.CalledProcessError as e:
        return {"command": command, "status": "failed", "output": e.output}

# Run jobs in parallel
with ThreadPoolExecutor(max_workers=MAX_CONCURRENT_JOBS) as executor:
    futures = [executor.submit(execute_job, job) for job in jobs]
    results = [future.result() for future in futures]

print("Job results:", results)

This code snippet efficiently executes the jobs concurrently, respecting the maximum concurrency limit.

Step 5: Generate a JSON Report

After executing the jobs, we need to generate a JSON report summarizing the results. This report will help us understand which jobs succeeded and which failed.

import json

report = {"jobs": results}

with open('job_report.json', 'w') as report_file:
    json.dump(report, report_file, indent=4)

print("Report generated: job_report.json")

The report contains each job's command, status, and output, providing a comprehensive overview of the execution.

Step 6: Exit with Appropriate Status

To ensure our tool indicates failure when any job fails, we need to check the results and exit with a non-zero status if necessary.

if any(job['status'] == "failed" for job in results):
    print("One or more jobs failed. Exiting with status 1.")
    exit(1)
else:
    print("All jobs succeeded. Exiting with status 0.")
    exit(0)

This step ensures that the tool exits with a status that reflects the success or failure of the jobs.

Common Errors/Troubleshooting

• Command not found: Ensure that the shell commands specified in the configuration file are correct and executable in your environment.
• Subprocess errors: If commands fail, check their syntax and ensure any required environment variables or dependencies are available.
• JSON decoding errors: Verify the JSON configuration file is correctly formatted.

Frequently Asked Questions

What Python version is required?

Python 3.10 or above is recommended for compatibility with the latest features.

Can I run more than 3 jobs concurrently?

Yes, you can adjust the MAX_CONCURRENT_JOBS variable to increase or decrease the concurrency limit.

What happens if a job continuously fails?

The tool will exit with a non-zero status, and the job report will indicate the failure, allowing for troubleshooting.

Frequently Asked Questions

What Python version is required?

Python 3.10 or above is recommended for compatibility with the latest features.

Can I run more than 3 jobs concurrently?

Yes, you can adjust the MAX_CONCURRENT_JOBS variable to increase or decrease the concurrency limit.

What happens if a job continuously fails?

The tool will exit with a non-zero status, and the job report will indicate the failure, allowing for troubleshooting.