Parallel Computing

It is the simultaneous use of multiple compute resources to solve a computational problem

Parallel Execution

Scalability refers to a parallel system’s (hardware and/or software) ability

Parallel computing can include

Serial Execution

Shared Memory is

Distributed Memory

Parallel Overhead is

Massively Parallel

Fine-grain  Parallelism is

In shared Memory

In shared Memory:

In shared Memory

In designing a parallel program, one has to break the problem into discreet chunks of work that can be distributed to multiple tasks. This is known as

Latency is

Domain Decomposition

Functional Decomposition:

Synchronous communications

Collective communication

Point-to-point communication referred to

Uniform Memory Access (UMA) referred to

Asynchronous communications

Granularity is

Coarse-grain Parallelism

Cache Coherent UMA (CC-UMA) is

Non-Uniform Memory Access (NUMA) is

It distinguishes multi-processor computer architectures according to how they can be classified along the two independent dimensions of Instruction and Data Each of these dimensions can have only one of two possible states: Single or Multiple.

In the threads model of parallel programming

These applications typically have multiple executable object files (programs). While the application is being run in parallel, each task can be executing the same or different program as other tasks. All tasks may use different data

Here a single program is executed by all tasks simultaneously. At any moment in time, tasks can be executing the same or different instructions within the same program. These programs usually have the necessary logic programmed into them to allow different tasks to branch or conditionally execute only those parts of the program they are designed to execute.