Multiprocessing

About

Multiprocessing refers to the ability of a system to support more than one processor at the same time by utilizing multiple CPU cores for true parallelism. Unlike threading, which shares memory within a single process and is constrained by Python’s Global Interpreter Lock (GIL), multiprocessing creates independent processes. Each process has its own memory space and operates in parallel, achieving true parallelism for CPU-bound tasks.

This approach allows programs to handle computationally intensive operations efficiently, such as numerical simulations, data processing, or machine learning tasks, by distributing the workload across available CPU cores.

Essential Components

Multiprocessing in Python involves several key components that allow efficient parallel execution of tasks

Process

A process is the fundamental unit of execution in multiprocessing. Each process runs independently and can execute its own task on a separate CPU core. The Process class in Python is used to create and manage these processes

Queue

A Queue is a thread and process-safe FIFO (First In, First Out) data structure that enables data exchange and coordination between processes. It’s used for passing messages or results between process instances

Pipe

Pipes provide a way to establish a communication channel between processes. They are useful for bidirectional communication between two processes

Lock

A Lock is a synchronization primitive that ensures only one process accesses a shared resource at a time. It is critical for avoiding race conditions when multiple processes modify shared data

Example

Example taken from Multiprocessing in Python | Set 1 (Introduction)

from multiprocessing import Process, Queue, Pipe, Lock

def worker(q, lock):
    with lock:
        print("Task executed")
        q.put("Data")

if __name__ == "__main__":
    q = Queue()
    lock = Lock()
    p = Process(target=worker, args=(q, lock))
    p.start()
    p.join()
    print(q.get())  # Output "Data"

Implementation

from multiprocessing import Process

def worker_function(name):
    print(f"Hello from {name}")

if __name__ == "__main__":
    process1 = Process(target=worker_function, args=("Process 1",))
    process2 = Process(target=worker_function, args=("Process 2",))
    
    process1.start()
    process2.start()
    
    process1.join()
    process2.join()

    print("All processes completed.")

The output of the provided script depends on the order in which the processes are executed. Since Python's multiprocessing doesn't guarantee the order of execution for separate processes, the output may vary. However, it will generally look something like this:

Hello from Process 1
Hello from Process 2
All processes completed.