# What is Python multiprocessing and how to use it

Python multiprocessing lets you divide the **workload among multiple processes**, cutting down on overall execution time. This is especially useful for making hefty calculations or handling large datasets.

## What is Python multiprocessing?

Multiprocessing in Python refers to running multiple processes simultaneously, allowing you to make the most of **multicore systems**. Unlike single-threaded methods that handle tasks one by one, multiprocessing lets various parts of the program run in parallel, each on its own. Each process gets its own memory space and can run on separate processor cores, slashing execution time for heavy-duty or time-sensitive operations.

Python multiprocessing has a wide range of applications. Multiprocessing is often used in data processing and analysis to process large data sets faster and to accelerate complex analyses. Multiprocessing can also be used in **simulations and modelling calculations** (e.g., in scientific applications) to shorten the execution times of complex calculations. In addition to powering web scraping by fetching data from multiple sites simultaneously, it also boosts efficiency in image processing and computer vision, resulting in quicker image analysis.

## How to implement Python multiprocessing

Python offers various options for implementing multiprocessing. In the following sections, we’ll introduce you to three common tools: the `multiprocessing` module, the `concurrent.futures` library and the `joblib` package.

### `multiprocessing` module

The `multiprocessing` module is the **standard module for Python multiprocessing**. With this module, you can create processes, share data between them and sync them using locks, queues and other tools.

```python
import multiprocessing
def task(n):
    result = n * n
    print(f"Result: {result}")
if __name__ == "__main__":
    processes = []
    for i in range(1, 6):
        process = multiprocessing.Process(target=task, args=(i,))
        processes.append(process)
        process.start()
    for process in processes:
        process.join()
```

In the example above, we use the `multiprocessing.Process` class to spawn and run processes executing the `task()` function, which computes the square of a given number. After initialising the processes, we wait for them to complete before proceeding with the main program. The result is displayed using an **f-string**, a Python string format method that incorporates expressions. It’s worth noting that the sequence of output is random and non-deterministic. You can also create a process pool with Python `multiprocessing`:

```python
import multiprocessing
def task(n):
    return n * n
if __name__ == "__main__":
    with multiprocessing.Pool() as pool:
        results = pool.map(task, range(1, 6))
        print(results)  # Output: [1, 4, 9, 16, 25]
```

With `pool.map()` the function `task()` is applied to a sequence of data, and the results are collected and output.

### `concurrent.futures` library

This module provides a high-level interface for asynchronous execution and parallel processing of processes. It uses the **Pool Executor** to execute tasks on a pool of processes or threads. The `concurrent.futures` module is a simpler way to process asynchronous tasks and is in many cases easier to handle than the Python `multiprocessing` module.

```python
import concurrent.futures
def task(n):
    return n * n
with concurrent.futures.ProcessPoolExecutor() as executor:
    futures = [executor.submit(task, i) for i in range(1, 6)]
    for future in concurrent.futures.as_completed(futures):
        print(future.result()) # result in random order
```

The code uses the `concurrent.futures module` to process tasks in parallel with the `ProcessPoolExecutor`. The function `task(n)` is passed for numbers from 1 to 5. The `as_completed()` method waits for the tasks to be completed and outputs the **results in any order**.

### `joblib` package

`joblib` is an external Python library designed to simplify parallel processing in Python, for example, for repeatable tasks such as executing functions with different input parameters or working with large amounts of data. The main functions of `joblib` is the parallelisation of tasks, the **caching of function results** and the optimisation of memory and computing resources.

```python
from joblib import Parallel, delayed
def task(n):
    return n * n
results = Parallel(n_jobs=4)(delayed(task)(i) for i in range(1, 11))
print(results) # Output: Results of the function for numbers from 1 to 10
```

The expression `Parallel(n_jobs=4)(delayed(task)(i) for i in range(1, 11))` initiates the parallel execution of the function `task()` for the numbers from 1 to 10. `Parallel` is configured with `n_jobs=4`, meaning up to four parallel jobs can be processed. Calling `delayed(task)(i)` creates the task to be executed in parallel for each number i in the range from 1 to 10. This means that the `task()` function is called simultaneously for each of these numbers. The results for numbers 1 through 10 are stored in `results` and output.


This is a markdown version of: [https://www.ionos.com/en-ie/digitalguide/websites/web-development/python-multiprocessing/](https://www.ionos.com/en-ie/digitalguide/websites/web-development/python-multiprocessing/) for AI/LLM consumption.