In the first phase, the DEEP project laid the ground for the DEEP programming environment. Ater all, programming heterogeneous systems is often considered challenging and cumbersome. Therefore, the DEEP programming environment provides a dedicated development and runtime system which makes porting applications to DEEP project prototypes a snap. If your application supports MPI. If your application does not use MPI, the task-based OmpSs programming model might even save you from explicitly parallelizing your application yourself.

 


DEEP-ER: Resiliency & highly scalable I/O

In the second phase, within the DEEP-ER project, the system software stack was extended to feature a highly scalable, efficient, and user-friendly parallel I/O system based on the Fraunhofer parallel file system BeeGFS (formerly known as FhGFS). Additionally, it provides a low overhead, unified user-level checkpointing system and exploiting the multiple levels of non-volatile memory and storage added to the DEEP architecture.

 


DEEP-EST: Resource management & job scheduling

Now in the third phase, the DEEP-EST project, the focus in system software development lies with resource management and job scheduling. The developments in this area will enable to determine optimal resource allocations for each combination of workloads, support adaptive scheduling and enable dynamical reservation of the resources. Added to the modularity of the system itself, this guarantees maximum usage of the overall system, since no component is “blocked” for an application if it is not utilised.


Furthermore, the DEEP‑EST unified programming environment, based on standard components such as MPI and OmpSs but also on the emerging programming paradigms of the data-analytic fields like map-reduce, will provide a model that fully supports applications to use combinations of heterogeneous nodes, and enables developers to easily adapt and optimise their codes while keeping them fully portable.

Benchmarking is an essential element in evaluating the success of a hardware prototyping project. In the DEEP-ER project we use the JUBE benchmarking environment to assess the performance of the DEEP-ER system.

The heterogenous nature of the MSA poses new challenges to job schedulers and resource management systems that will be approached in the DEEP-EST project.

The DEEP projects actively develop software in key areas such as programming software, I/O, resiliency, benchmarking, tools, resource management, and job scheduling. Most of the developments are open source.

Highly-scalable, parallel I/O will be an important building block for future Exascale systems. In the DEEP-ER project, the I/O subsystem relies on 3 components: Fraunhofer's parallel file system BeeGFS, the parallel I/O library SIONlib, and Exascale10.

Read more ...

Based on OpenMP, OmpSs employs a data-flow programming model to ease application porting to the heterogeneous architectures. It exploits task level parallelism and supports asynchronicity, heterogeneity and data movement.

Exascale systems will require a combination of powerful resiliency techniques that are also flexible enough to accommodate the heterogeneous nature of such systems.

Read more ...

The DEEP software stack includes the Parastation MPI by ParTec which has been modified to reflect the peculiarities of the DEEP system.