What Is Parallel Processing? How CPUs and GPUs Divide the Work

Imagine a job that takes 100 seconds to finish.

You give it four times the processing resources. A little mental arithmetic suggests it should now take 25 seconds.

It takes 40.

You increase the resources again, this time to sixteen workers. Now it takes 25 seconds.

Four times the workers produced a useful improvement. Sixteen times the workers still only made the job four times faster.

Where did the rest of that performance go?

This is a hypothetical example, but the puzzle gets to the heart of parallel processing. Having more hardware available is only part of the story. The work has to give that hardware something useful to do.

To understand why, let’s brighten a picture.

What is parallel processing?

Parallel processing means using multiple computing resources to execute parts of a computation at the same time. A larger job is divided into pieces, those pieces are processed, and their results are coordinated as needed. ^[1]

It can happen on CPUs, GPUs, or systems containing multiple computers.

The important question is what can happen together.

Some jobs divide beautifully. Others contain steps that have to wait their turn.

Twenty-four million pixels looking for work

A photo measuring 6,000 × 4,000 pixels contains 24 million pixels.

Suppose we want to brighten it with a simple adjustment. For each pixel, we read its original color values, calculate brighter values, and write the result into a separate output image.

A sequential version looks like this:

for each pixel in the input image:
    output[pixel] = brighten(input[pixel])

Conceptually, it works through the picture in order.

But look at what the calculation needs.

To brighten a pixel in the top-left corner, do we need the finished pixel in the bottom-right corner?

For this adjustment, we do not.

That means we can divide the image into non-overlapping tiles and process different tiles together:

process the image tiles in parallel:
    for each pixel in this tile:
        output[pixel] = brighten(input[pixel])

Each worker reads the original image and writes to its own part of the output.

Nobody needs to wait for another tile’s answer.

These are conceptual examples, not executable code or benchmarks of a photo editor. What they show is the property that makes this job promising: the pieces can be calculated independently.

Two ways to split a job

Two useful patterns are data parallelism and task parallelism. ^[1]

Data parallelism

Our picture uses data parallelism.

The workers perform similar operations on different portions of the data. Same adjustment, different pixels.

Think of several people checking prices in separate sections of a catalog.

Task parallelism

Task parallelism divides work by activity.

Imagine an application preparing an article. One task generates a thumbnail while another builds a search entry from the text.

If neither needs the other’s result, those tasks can happen together.

An application can use both approaches.

Your CPU already does this

Parallel processing is not a special ability that arrives with a graphics card.

On a multicore CPU, software can use different threads to perform suitable work on different cores. A thread is an execution path within a program, not another physical core. ^[2]

Creating more threads gives the system more work to schedule. It does not manufacture more hardware.

And CPU parallelism goes deeper than core count.

Vector instructions can apply an operation to multiple data elements. This is commonly called SIMD, short for single instruction, multiple data. ^[3]

A photo editor might therefore use several CPU cores while also using vector instructions inside those cores.

The explanation that “CPUs do one thing while GPUs do many things” misses a substantial part of what a modern CPU can do.

For the hardware distinctions behind those labels, read CPU Cores vs GPU Cores: Why the Numbers Don’t Compare.

Lots of threads does not automatically mean lots of simultaneous work

This is where concurrency and parallelism often get mixed together.

Concurrency concerns organizing activities so they can make progress during overlapping periods. Parallelism means computations actually execute at the same time. ^[4]

An application can download a file while responding to your clicks. It might handle an input event, wait for network data, then handle another event.

Those activities can make progress without all their application instructions executing simultaneously.

Parallel execution goes further: separate processing resources perform calculations at the same time.

Keeping an application responsive and making a calculation finish sooner are both useful. They are different goals.

Where the GPU comes in

A GPU provides another way to execute large amounts of parallel work.

In NVIDIA’s CUDA programming model, a function called a kernel runs across many software threads. Those threads are organized into blocks, which are scheduled onto the GPU’s streaming multiprocessors. ^[5]

For our image adjustment, a programmer might assign each thread a pixel or a small group of values.

That does not mean every pixel gets its own permanently dedicated hardware. It also does not mean every thread in a large image runs simultaneously. The GPU schedules work according to the resources available. ^[5]

The opportunity comes from providing enough suitable work to keep those resources occupied.

If you want to look inside the arithmetic hardware involved, continue with What Are CUDA Cores?.

The catch: some answers have to arrive first

Our brightness adjustment is convenient because each output pixel depends only on its corresponding input pixel.

Now imagine a calculation where every step uses the previous step’s result.

The second step needs the first answer. The third needs the second.

Giving those steps to different workers does not make the missing answers appear.

Dependencies help determine how much parallelism an algorithm can expose. A different algorithm may create more opportunities, but some sequences of operations remain. ^[1]

You can have plenty of workers and still have a queue.

Solving the 100-second puzzle

Return to the imaginary job from the opening.

It takes 100 seconds with one worker, but its time is divided like this:

• 20 seconds belong to a stage that stays sequential.
• 80 seconds belong to work that additional workers can divide.

For now, assume identical workers, perfect scaling of the divisible work, and no extra overhead.

There is our missing performance.

The additional workers shorten the divisible stage, but the 20-second stage stays put.

Even if the parallel stage became almost instantaneous, the complete job would still take approximately 20 seconds. Its speedup would approach a limit of 5×.

This is the central lesson of Amdahl’s law: for a fixed workload, the portion that remains sequential limits the benefit of parallelizing the rest. ^[6]

Our table gives the parallel stage perfect conditions. Real software has a few more things to pay for.

Dividing the work creates more work

Someone has to organize it

Software has to partition the job, schedule workers, and coordinate their progress.

For a small task, that organization can cost more time than it saves. Microsoft warns that parallel loops can run more slowly than sequential versions, particularly when there are few iterations and little work in each one. ^[7]

Giving a tiny picture to hundreds of workers may be an excellent way to keep the scheduler busy.

It is not automatically an excellent way to finish the picture sooner.

The data has to get there

A discrete GPU with separate device memory may need the input transferred to it and the result transferred back.

Those transfers take time. NVIDIA recommends minimizing unnecessary movement between CPU and GPU memory and keeping intermediate data on the GPU where possible. ^[8]

A calculation might be much faster on the GPU while the complete operation barely improves.

If the trip there and back consumes the savings, the stopwatch does not care how impressive the calculation was.

Workers can collide

Imagine two workers updating the same counter.

Both read 10. Each adds one. Each writes 11.

Two updates were attempted, but the counter only increased once.

This illustrates a problem with unsynchronized updates to shared data. Synchronization can protect those updates, but it can also introduce waiting and overhead. ^[9]

Our image example avoids that particular conflict by giving workers separate output regions.

Fast arithmetic still needs something to calculate

More arithmetic hardware helps only when the rest of the system can supply useful work.

GPU performance can be limited by memory bandwidth, arithmetic throughput, or latency. Insufficient parallel work can also leave execution resources underused. ^[10]

If reading and writing the pixels is already the limiting factor, adding arithmetic capacity alone may accomplish very little.

The hardware can be ready to do more while the data keeps it waiting.

How do you know whether parallel processing will help?

Start with the job rather than the specification sheet.

1. Find the independent work. Which operations can happen together, and which need earlier results?
2. Find where the time goes. Making a tiny stage faster may barely change the total.
3. Count the additional costs. Include scheduling, synchronization, and data movement.
4. Measure the finished job. Check whether the result arrives sooner.

When comparing CPU and GPU implementations, use the same workload and required output quality. Include necessary setup and transfers when timing the complete operation.

A fast individual calculation is useful. The number that matters to the person waiting is how long the whole job takes.

What parallel processing actually promises

Parallel processing gives a computer an opportunity to work on several parts of a problem at once.

How valuable that opportunity becomes depends on the problem.

Our picture offers millions of independent pixel calculations. Other jobs contain longer dependency chains, more shared state, or more expensive transfers.

That is why more cores can transform one workload and barely change another.

Before asking how many workers a processor has, ask:

Can this job keep them doing useful work?

That question gets you much closer to understanding performance than counting hardware alone.

Where to go next

For the broader architectural picture, read Why GPUs Are So Much Faster Than CPUs.

To understand why CPU and GPU specifications count different kinds of hardware, read CPU Cores vs GPU Cores: Why the Numbers Don’t Compare.

For a closer look at NVIDIA’s arithmetic resources, continue with What Are CUDA Cores?.

Sources