Contents

• Background and infrastructure
• Accounts, login, and file system
• Using Bash shell
• Compiling and running code on CPU nodes
• Job script for efficient utilization of hardware
• Using ThinLinc
• Compiling and running code on GPU nodes
• Using Singularity
• Using Matlab
• Materials theory codes
• Using Python virtual environment

Background and infrastructure

History of PDC

History of PDC

PDC is finansed by NAISS

• NAISS: National Academic Infrastructure for Supercomputing in Sweden

• Resources: Chalmers Tek. Högskola, Kungliga Tek. Högskola, Linköping Universitet, Uppsala Universitet

• Branches: Chalmers Tek. Högskola, Göteborg Universitet, Karolinska institutet, Kungliga Tek. Högskola, Linköpings Universitet, Lund Universitet, Stockholm Universitet, Umeå Universitet, Uppsala Universitet,

Collaboration with Industry

PDC's largest industrial partner is Scania. The figure shows a volume rendering of the instantaneous velocity magnitude on the leeward side of a Scania R20 Highline truck at crosswind conditions. (Source: Scania)

Collaboration with industry

• Business partners

  • https://www.pdc.kth.se/research/business-research/pdc-partners

• A small part of Dardel nodes will be dedicated to industry/business research.

• If you are interested in purchasing HPC compute time, contact PDC Support.

Broad range of training

• Summer School

  • Introduction to HPC held every year

• Workshops (see PDC events)

• University Courses that use PDC systems

  • KTH courses: DD2356, DD2365, SF2568, FDD3258, ...
  • SU courses: BL4018, BL8060

Support and System Staff

• First-Line Support

  • Provide specific assistance to PDC users related to accounts, login, allocations etc.

• System Staff

  • System managers/administrators ensure that computing and storage resources run smoothly and securely.

• Application Experts

  • Hold PhD degrees in various fields and specialize in HPC. Assist researchers in optimizing, scaling and enhancing scientific codes for current and next generation supercomputers.

Introduction to Dardel supercomputer

2023-04-13

About Dardel

Dardel is an HPE Cray EX system

• CPU partition

  • 1270 CPU nodes
  • Each node has
    • two AMD EPYC(TM) Zen2 2.25 GHz 64-core processors

• GPU partition

  • 56 GPU nodes
  • Each node has
    • one AMD EPYC(TM) processor with 64 cores
    • four AMD Instinct(TM) MI250X GPUs

More about the CPU nodes

Non-uniform memory access (NUMA)

• Memory access time depends on the relative location of memory

• Accessing local memory is faster compared to non-local memory

• On Dardel compute node: 8 NUMA domains (16 cores in each NUMA domain)

  salloc -N 1 ... srun -n 1 numactl --hardware

Non-uniform memory access (NUMA)

Non-uniform memory access (NUMA)

Account, Login and File System

2023-04-13

PDC support documentation

Getting a PDC account

• From SUPR

  • Get a SUPR account
  • Join a project with time allocation on Dardel
    • PI: create proposal for small/medium/large allocation
    • collaborators: added to project by PI
  • Request a PDC account from SUPR

• From PDC webpage

  • Attend a course or training event
  • Fill in the form and provide a copy of passport or national ID

Time allocation (project)

• Unit: core-hours per month

• NAISS projects are managed in SUPR.

• Course/Workshop allocations are managed locally at PDC.

• Use projinfo to list the projects that you have access to.

Login

• Method 1: Use Kerberos ticket

  • Windows: PuTTY + NIM
  • Linux: openssh-client
  • macOS: homebrew-openssh-gssapi

• Method 2: Use SSH keys

  • Upload your public key in PDC login portal

Login with Kerberos ticket

• Edit /etc/krb5.conf on Linux/macOS

• kinit -f <your-username>@NADA.KTH.SE

  • use NIM on Windows
  • use /usr/bin/kinit -f ... on macOS

• ssh -o GSSAPIDelegateCredentials=yes -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes <your-username>@dardel.pdc.kth.se

  • use PuTTY on Windows
  • you can save SSH options in ~/.ssh/config on Linux/macOS

Login with SSH keys

• Generate SSH key pair

  ssh-keygen -t ed25519 -f ~/.ssh/id-ed25519-pdc
  • Create ~/.ssh folder if it doesn't existmkdir ~/.ssh && chmod 700 ~/.ssh

• Upload SSH public key in PDC login portal

• Login using SSH key

  ssh -i ~/.ssh/id-ed25519-pdc <your-username>@dardel.pdc.kth.se

File System

Lustre File System (Klemming, total size 12 PB (12,000 TB))

• Open-source massively parallel distributed file system
• Optimized for handling data from many clients
• Home directory (25 GB quota, with backup)/cfs/klemming/home/[u]/[username]
• Project directory/cfs/klemming/projects/...
• Scratch directory/cfs/klemming/scratch/[u]/[username]

Home and project directories

• Home directory

  cd && pwd

  or

  echo $HOME

• Project directory

  projinfo

Groups

In addition to projinfo, your groups also indicate the projects that you have access to.

• groups starting with cac- are associated with compute projects
• groups starting with pg_ are associated with storage projects

Storage quota

Use of file system

• Good practice
  • Minimize the number of I/O operations
  • Avoid creating too many files
  • Avoid creating directories with a large numbers of files
• Bad practice
  • Small reads
  • Opening many files
  • Seeking within a file to read a small piece of data

Access Control Lists

To view the access for a folder:

The output looks like this:

Access Control Lists

To grant the access to another user (use "-R" for recursive):

To remove the access for another user (use "-R" for recursive):

Using Bash shell

Introduction for beginners

Tor Kjellsson Lindblom

Content

• Bash shell and basic commands
• Files and Folders
• Input/output
• File/directory permissions
• Environment variables

Bonus material for self-studying

• Processes (including programs)
• Searching in text
• Finding files
• Hotkeys
• File archiving

What is a shell?

• What you get when the terminal window is open

• A "layer" (shell) around the operating system

• Frequently used to interact with remote systems, such as Dardel

• Multiple types of shells exist - this is about bash shell

• This presentation contains the very basics with hands-on exercises and type-alongs.

• No need to finish all exercises now.

Getting a bash shell

• Linux and Mac users - just open a terminal window (or login to Dardel)
• Windows users - please login to Dardel.

Your very first commands (type along)

Explore the following commands, one at a time.

The last two lines downloads and extracts tutorial material we have borrowed from Software Carpentry
https://carpentries.org/

Exercise 1 (3 min)

verify that the new file was created.
NB! Do not use whitespace in file/folder names.

Exercise 2: Some more commands (5 min)

Relative vs. absolute paths

You can specify a location in two ways:

Both ways are useful. Determine which is best for the situation at hand.

Text editors

Very good idea to master at least one (non-graphical) editor in the terminal.

List of common editors:

• nano - easiest, minimal functionality
• vi/vim - a bit more involved, but more functionality
• emacs - even a bit more involved, but a lot of functionality
• HOMEWORK: get used to one text editor.

Input & Output: redirect and pipes

• Programs can display something, e.g. echo hello world

• Programs can take input, e.g. less

• $ cat numbers_copy.txt

• $ cat numbers_copy.txt | less

Try it: pipes (3 min)

Redirects

• Like pipes, but data is sent to/from files and not processes

• Replace a file:

• $ command > file.txt

• Append to a file:

• $ command >> file.txt (be careful to not mix them up!)

• Redirect file as STDIN:

• command < file (in case program accepts STDIN only)

Try it: Redirects (3 min)

File/directory permissions

The basics

Important to set access permission on shared objects

Tell you who can: read, write and execute a file/list a directory

d is for directory - then 3 groups of triple fields: user, group, others

Basic file permission manipulation

NB: On Dardel, we also use ACLs (next slide)

Access control lists (ACLs)

In many support cases we ask users to apply the last line so we can access files.

Environment variables

Defined text strings that your programs may use

Bonus material

• Processes (including Programs)
• Initialization and configuration
• Finding files and text patterns within files
• Hotkeys
• File archiving

Jump to next section

Processes [bonus slide]

Uptil now we only discussed files/folders.
But we also want to run programs.

• All running programs and commands are processes
• Processes have:
  • Process ID, NAME, Command line arguments
  • input and output, Return code (integer) when complete
  • Working directory, Environment variables
• These concepts bind together all UNIX programs
• To see some runnings processes, type top

Foreground and background processes [bonus slide]

Foreground

• Example: top
• Keyboard is connected as input, screen to output.
• Only one such process active at a time.
• Kill it: Ctrl-c

Background

• No input connected
• You can have as many as resources allow
• Add an & after a command to put it in background
• To kill: use kill or pkill, or do it from within top

Foreground and background processes [bonus slide cont]

NB: You will most likely not use Dardel like this, but it is possible to do so by logging into a compute node.

Exercise: some more redirects [bonus slide]

Initialization and configuration [bonus slide]

• When the shell first starts (e.g. at login) it reads shell config files.

• The config files give you power to customize your shell to your liking.

• You can always manually test things in an open shell before putting it in the config files (recommended!)

Config files are located in $HOME and are called:

• .bashrc
• .bash_profile

Example to try [bonus slide]

Findings patterns in files: grep [bonus slide]

This command is for searching keyword inside files.

Exercise 5 [grep] [bonus slide]

Finding things [bonus slide]

Command: find

Exercise 5 [bonus slide]

Bonus (for interested to do later):

• On a Lustre system, lfs find is faster. Same syntax.

• On a workstation: locate may be useful. Read manual for information.

• Type find . -name animals.csv.

• Type find . -name *.pdb

• Make a pipe that counts number of files/directories in the
  data-shell directory.

• Count unique logged in users on Dardel.

Tip: w or users give you a list of all currently login users,
many of them have several sessions open.

Tip: You may have to use uniq, tr -s, cut -f 1 -d " ", and wc -l

Hotkeys [bonus slide]

• Shortcuts
• Most important key: tab for autocompletion
• You should never type full filenames or command names - tab can complete almost anything.

Some common commands

Exercise 6 [bonus slide]

TAB autocompletion

We will here display contents of a file using its full path, but try to type as few characters as possible.

First find your absolute path to "numbers.txt"

File archiving [bonus slide]

• tar is the standard tool to save many files or directories into a single archive
• Archive files may have extensions .tar, .tar.gz etc depending on compression used.
• "f" is for filename, "a" selects compression based on suffix
• With no compression, files are simply packed
• "r" will append files to end of archive, "t" will list archive
• Individual files can be compressed directly with e.g. gzip. (gzip file, gunzip file.gz)

Exercise 7 [bonus slide]

Compiling and running code on CPU nodes

Xin Li

Cray programming environment (CPE)

Reference page: Compilers and libraries

The Cray Programming Environment (CPE) provides consistent interface to multiple compilers and libraries.

• In practice, we recommend

  • ml cpeCray/23.12
  • ml cpeGNU/23.12
  • ml cpeAOCC/23.12

• The cpeCray, cpeGNU and cpeAOCC modules are available after ml PDC/23.12

• No need to module swap or module unload

Compiler wrappers

• Compiler wrappers for different programming languages

  • cc: C compiler wrapper
  • CC: C++ compiler wrapper
  • ftn: Fortran compiler wrapper

• The wrappers choose the required compiler version and target architecture optinons.

• Automatically link to MPI library and math libraries

  • MPI library: cray-mpich
  • Math libraries: cray-libsci and cray-fftw

Compile a simple MPI code

• hello_world_mpi.f90

  program hello_world_mpi include "mpif.h" integer myrank,mysize,ierr call MPI_Init(ierr) call MPI_Comm_rank(MPI_COMM_WORLD,myrank,ierr) call MPI_Comm_size(MPI_COMM_WORLD,mysize,ierr) write(*,*) "Processor ",myrank," of ",mysize,": Hello World!" call MPI_Finalize(ierr) end program ftn hello_world_mpi.f90 -o hello_world_mpi.x

What flags does the ftn wrapper activate?

• Use the flag -craype-verbose

  ftn -craype-verbose hello_world_mpi.f90 -o hello_world_mpi.x

Compile a simple MPI code

Compile a simple linear algebra code

Link to the code

Use cray-libsci

Compile a simple linear algebra code

Use openblas

where the environment variable EBROOTOPENBLAS had been set when loading the OpenBLAS module. Its module file includes a statement

which corresponds to an export statement

Check the linked libraries

Check the linked libraries

Test run the dgemm_test code

• Run on a single core in the shared partition

  salloc -n 1 -t 10 -p shared -A <name-of-allocation> srun -n 1 ./dgemm_test_craylibsci.x srun -n 1 ./dgemm_test_openblas.x exit

• Expected output:

  2.700 4.500 6.300 8.100 9.900 11.700 13.500 4.500 8.100 11.700 15.300 18.900 22.500 26.100 6.300 11.700 17.100 22.500 27.900 33.300 38.700

Exercise: Compile and run fftw_test code

Compilation of large program

• Examples at https://www.pdc.kth.se/software

Environment variables for manual installation of software

• Environment variables for compilers

  export CC=cc export CXX=CC export FC=ftn export F77=ftn

• Environment variables for compiler flags

  • add -I, -L, -l, etc. to Makefile

• Environment variables at runtime

  • prepend to PATH, LD_LIBRARY_PATH, etc.

What happens when loading a module

When running your own code

• Load correct programming environment (e.g. cpeGNU)
• Load correct dependencies (e.g. openblas if your code depends on it)
• Properly prepend to environment variables (e.g. PATH, LD_LIBRARY_PATH)
• Choose correct SLURM settings

SLURM settings for hybrid MPI/OpenMP code

• --nodes number of nodes

• --ntasks-per-node number of MPI processes

• --cpus-per-task 2 x number of OpenMP threads (because of SMT)

• OMP_NUM_THREADS number of OpenMP threads

• OMP_PLACES cores

Example job script

• 64 MPI x 2 OMP per node (main parition)#!/bin/bash #SBATCH -A ... #SBATCH -J my_job #SBATCH -t 01:00:00 #SBATCH -p main #SBATCH --nodes=2 #SBATCH --ntasks-per-node=64 #SBATCH --cpus-per-task=4 module load ... export OMP_NUM_THREADS=2 export OMP_PLACES=cores srun ...

Example job script

• 2 MPI x 2 OMP per node (shared partition)#!/bin/bash #SBATCH -A ... #SBATCH -J my_job #SBATCH -t 01:00:00 #SBATCH -p shared #SBATCH --ntasks=2 #SBATCH --cpus-per-task=4 module load ... export OMP_NUM_THREADS=2 export OMP_PLACES=cores srun ...

Exercise: Hybrid MPI/OpenMP code for matrix-matrix multiplication

• Preparation

  mkdir -p matmul_test && cd matmul_test

• Copy python code matmul_mpi_omp_test.py to the same folder

Exercise: Hybrid MPI/OpenMP code for matrix-matrix multiplication

• Copy job script job-n1.sh

  • for running on 1 MPI processes with different number of OpenMP threads

• Copy job script job-n2.sh

  • for running on 2 MPI processes with different number of OpenMP threads

• Submit two jobs

Exercise: Hybrid MPI/OpenMP code for matrix-matrix multiplication

• Result

  Setting	Timing
  1 MPI x 16 OMP	Time spent in matmul: 2.307 sec
  1 MPI x 8 OMP	Time spent in matmul: 3.924 sec
  1 MPI x 4 OMP	Time spent in matmul: 6.626 sec
  2 MPI x 8 OMP	Time spent in matmul: 2.034 sec
  2 MPI x 4 OMP	Time spent in matmul: 3.287 sec
  2 MPI x 2 OMP	Time spent in matmul: 6.188 sec

Running jobs with efficient utilization of hardware

Xavier Aguilar

How do we run jobs on a Supercomputer?

• Edit files
• Compile programs
• Run light tasks
• Submit jobs
• Do not run parallel jobs on the login node!

What is SLURM ?

Simple Linux Utility for Resource Management

• Open source, fault-tolerant, and highly scalable cluster management and job scheduling system
  • Allocates access to resources
  • Provides a framework for work monitoring on the set of allocated nodes
  • Arbitrates contention for resources

Types of jobs?

• Batch jobs
  • the user writes a job script indicating the number of nodes, cores, time needed, etc.
  • the script is submitted to the batch queue
  • The user retrieves the output files once the job is finished
• Interactive jobs:
  • the user runs a command that allocates interactive resources on a number of cores
  • this creates an interactive job that awaits in the queue as any other job
  • when the job reaches the front of the queue, the user gets access to the resources and can run commands there

How jobs are scheduled?

• Age: the time the job has been in the queue
• Job size: number of nodes or cores requested
• Partition: a factor associated with each node partition
• Fair-share: the difference between the computing resources promissed and the amount of resources computed

SLURM basic commands

Submit a job to the queue:

List queued/running jobs belonging to a user:

Cancel a job:

Get information on partitions and nodes

Job scripts (pure MPI)

Partitions

• Nodes are logically grouped into partitions
• There are several partitions that can be used on Dardel
  • main: Thin nodes (256 GB RAM), whole nodes, maximum 24 hours job time
  • long: Thin nodes (256 GB RAM), whole nodes, maximum 7 days job time
  • shared: Thin nodes (256 GB RAM), job shares node with other jobs, maximum 24 hours job time
  • memory: Large/Huge/Giant nodes (512 GB - 2 TB RAM), whole nodes, 24 hours job time
  • gpu: GPU nodes, 512 GB RAM, and 4 AMD MI250X GPU cards

Job scripts (MPI + OpenMP)

Interactive jobs

Request an interactive allocation

Once the allocation is granted, a new terminal session starts (typing exit will stop the interactive session)

It is also possible to ssh into one of the allocated nodes.

Interactive jobs

We can check what nodes have been granted with:

or inspecting the environment variable:

Exercise 1: Basic SLURM commands

• In this exercise we are going to test some basic SLURM commands.

• You will:

  • Create and compile a simple MPI program
  • Submit it to the queues
  • Inspect the queues
  • Inspect the partitions
  • Check the output of the job

MPI Hello world

Exercise 1

• You can find the code from the previous slide here

• Save the file on Dardel, compile the code and generate a binary called hello_mpi.x

Exercise 1

• Take the job script that you can find here and modify it accordingly to:
  • Use one node for the job
  • Use 4 cores from that node

Exercise 1

• Submit this script using sbatch

• Check the queue using squeue -u <your_username>

  • What's the ID of the job?
  • Is it already running? If so, which node was allocated for the job?

• Once the job finishes check the job output. Where is it saved?

Note:

Notice that we run our program with just:

It would be also possible to run our program with:

However, we don't need to specify those flags because SLURM takes the -N and -n values from the BATCH directives in the script

Exercise 1

• Use sinfo to check the partitions
  • How many different partitions are defined? What are their names?
  • What's the partition with the highest number of nodes?
  • Name 1-2 nodes included in that same partition

SLURM advanced commands

Get detailed information of a running job:

Filter the information provided by sstat

Tip: Use sstat -e to see all possible fields for the format flag

SLURM advanced commands

To get information on past jobs:

Get detailed information of a certain job:

SLURM advanced commands

Quick performance summary for a finished job:

Common reasons for inefficient jobs

• Not all nodes allocated are used
• Not all cores within a node are used (if it's not intentional)
• Many more cores than the available are used
• Inefficient use of the file system

Exercise 2

• In this exercise we are going to use some more advanced SLURM commands to explore job performance.

• You will:

  • Create and compile a simple MPI program
  • Submit it to the queues
  • Check job data using sacct

Exercise 2

• You can find the sample code for this exercise here

• Compile the code and generate a binary called vector_mpi.x

Exercise 2

• You can find the job script for this exercise here

• Save the script on Dardel and submit the job. Once the job finishes check its output.

• Inspect the job performance data using sacct

  • Use: -j <job-id> --format=JobID,JobName,Elapsed,ReqMem,MaxRSS

    Tip: use flag "--unit=M" to see memory units in MB

Exercise 2

• Use the seff <job-id> command for quick job efficiency overview.

  Job ID: 211499 Cluster: dardel User/Group: xaguilar/xaguilar State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 8 CPU Utilized: 00:00:01 CPU Efficiency: 0.37% of 00:04:32 core-walltime Job Wall-clock time: 00:00:34 Memory Utilized: 549.25 MB (estimated maximum) Memory Efficiency: 7.73% of 6.94 GB (888.00 MB/core)

SLURM good practices

• Avoid too many short jobs
• Avoid massive output to STDOUT
• Try to provide a good estimate of the job duration before submitting

Packing short jobs

Several short jobs can be packed together into a single job where they run serially

The job is submitted with:

Using fewer cores

Reduce the number of cores used per job and run multiple instances of the job in a single node

Using fewer cores

Using ThinLinc

Mustafa Arif

Link to download the slides: ThinLinc_Tutorial.pdf

Compiling and running code on GPU nodes

Johan Hellsvik

Reference pages:
Building for AMD GPUs
Introduction to GPUs course, September and October 2023

Generalized programming for GPUs

Central processing units (CPU) and graphics processing units (GPU) do different work

• CPUs have large instruction sets and execute general code.

• GPUs have smaller instructions sets. Runs compute intensive work in parallel on large number of compute units (CU).

• Code execution is started and controlled from the CPU. Compute intensive work is offloaded to the GPU.

Dardel GPU nodes

Dardel has 56 GPU nodes, each of which is equipped with

• One AMD EPYC™ processor with 64 cores

• 512 GB of shared fast HBM2E memory

• Four AMD Instinct™ MI250X GPUs (with an impressive performance of up to 95.7 TFLOPS in double precision when using special matrix operations)

AMD Radeon Open Compute (ROCm)

The AMD Radeon Open Compute (ROCm) platform is a software stack for programming and running of programs on GPUs.

• The ROCm platform supports different programming models

  • Heterogeneous interface for portability (HIP)
  • Offloading to GPU with OpenMP directives
  • The SYCL programming model

• AMD ROCm Information Portal

Setting up a GPU build environment

• Load the PDC/23.12 module and version 5.7.0 of ROCm with

  • ml PDC/23.12
  • ml rocm/5.7.0

• Set the accelerator target to amd-gfx90a (AMD MI250X GPU)

  • ml craype-accel-amd-gfx90a

• Choose one of the available toolchains (Cray, Gnu, AOCC)

  • ml cpeCray/23.12
  • ml cpeGNU/23.12
  • ml cpeAOCC/23.12

The ROCM info command

Information on the available GPU hardware can be displayed with the rocminfo command. Example output (truncated)

The CRAY_ACC_DEBUG runtime environment variable

For executables that are built with the compilers of the Cray Compiler Environment (CCE), verbose runtime information can be enabled with the environment variable CRAY_ACC_DEBUG which takes values 1, 2 or 3.

For the highest level of information

export CRAY_ACC_DEBUG=3

Offloading to GPU with HIP

The heterogeneous interface for portability (HIP) is a hardware close (low level) programming model for GPUs. Example lines of code:

• Include statement for the HIP runtime

• HIP functions have names starting with hip

• Explicit handling of memory on the GPU

• Call to run the compute kernel on the GPU

Offloading to GPU with OpenMP

The OpenMP programming model can be used for directive based offloading to GPUs.

Example: A serial code that operates on arrays vecA, vecB, and vecC

Implement OpenMP offloading by inserting OpenMP directives. In Fortran the directives starts with !$omp

Exercise 1: Hello world with HIP

Build and test run a Hello World C++ code which offloads to GPU via HIP.

• Download the source code

  • wget https://raw.githubusercontent.com/PDC-support/introduction-to-pdc/master/example/hello_world_gpu.cpp

• Load the ROCm module and set the accelerator target to amd-gfx90a (AMD MI250X GPU)

  • ml rocm/5.7.0
  • ml craype-accel-amd-gfx90a

• Compile the code with the AMD hipcc compiler on the login node

  • hipcc --offload-arch=gfx90a hello_world_gpu.cpp -o hello_world_gpu.x

Run the code as a batch job

• Edit job_gpu_helloworld.sh to specify the compute project and reservation

• Submit the script with sbatch job_gpu_helloworld.sh

with program output written to output.txt

Exercise 2: Dot product with OpenMP

Build and test run a Fortran program that calculates the dot product of vectors.

• Activate the PrgEnv-cray environment ml PrgEnv-cray

• Download the source code

  • wget https://github.com/ENCCS/openmp-gpu/raw/main/content/exercise/ex04/solution/ex04.F90

• Load the ROCm module and set the accelerator target to amd-gfx90a

  • ml rocm/5.7.0 craype-accel-amd-gfx90a

• Compile the code on the login node

  • ftn -fopenmp ex04.F90 -o ex04.x

Run the code as a batch job

• Edit job_gpu_ex04.sh to specify the compute project and reservation

• Submit the script with sbatch job_gpu_ex04.sh

• with program output The sum is: 1.25 written to output.txt

Optionally, test the code in interactive session.

• First queue to get one GPU node reserved for 10 minutes

  • salloc -N 1 -t 0:10:00 -A <project name> -p gpu

• wait for a node, then run the program srun -n 1 ./ex04.x

• with program output to standard out The sum is: 1.25

• Alternatively, login to the reserved GPU node (here nid002792) ssh nid002792.

• Load ROCm, activate verbose runtime information, and run the program

  • ml rocm/5.7.0
  • export CRAY_ACC_DEBUG=3
  • ./ex04.x

• with program output to standard out

Using Singularity

Henric Zazzi

Link to download the slides: singularity.pdf

Using MATLAB

Johan Hellsvik

Using MATLAB

MATLAB is a proprietary multi-paradigm programming language and numeric computing environment developed by MathWorks.

MATLAB allows matrix manipulations, plotting of functions and data, implementation of algorithms, creation of user interfaces, and interfacing with programs written in other languages.

MATLAB at PDC

Licensing

In order to use MATLAB you need to have a MATLAB license.

KTH has a MATLAB site license valid for academic users that are affiliated with KTH or with other universities. The license includes all MATLAB toolboxes.

To access the installation of MATLAB on Dardel, please contact PDC support and request access.

Interactive and batch use of MATLAB

MATLAB can be run on Dardel both in interactive sessions, with or without a graphical user interface, or in batch jobs.

Running interactively

Matlab can be run interactively on allocated cores. To book 24 cores for one hour

Wait for cores to be reserved

Log in to the node where the cores reside

and start MATLAB with graphical user interface (GUI)

if you have not got X11 forwarding active, start MATLAB without GUI

Parallel computation with Matlab

MATLAB implements a large set of solutions for parallel computing.

• Vectorization
• Automated parallel computing support
• Parallel execution of loops with parfor
• Running multiple serial batch jobs

Parallel execution with parfor

Calculation of the spectral radius of a matrix

Consider first serial code: spectral_serial.m

Execution time on one core: 297 s

Can this be parallelized? How large is the overhead?

A parfor loop can be used to execute loop iterations in parallel on workers in a parallel pool.

Replace for statements in serial code with parfor to obtain spectral_parfor.m

Execution time on 24 cores: 30 s

Speedup as compared to execution time on 1 core: 9.9

Parallel execution of MATLAB in batch jobs

Also for batch job use of MATLAB, consider to use the shared partition of Dardel. Example job script jobMATLABn16.sh

Running multiple serial batch jobs

Exercise 1

Calculating spectral radius in interactive session

• Request node / cores for interactive session
• Log in to the node, and start MATLAB
• Experiment with spectral_serial.m and spectral_parfor.m
  • Compare runtimes serial code vs parallel code for n = 200; A = 500;
  • Change values of n and A. How does relative runtime change?

Exercise 2

Calculating spectral radius in batch job

• Edit the template script jobMATLABn16.sh. You will need to

  • Specify allocation and reservation
  • Change the name of the MATLAB program
  • Consider to change the time for the job and number or cores

• Launch jobs for both the serial and the parallel versions of the spectral radius code

Materials theory codes on Dardel

Johan Hellsvik

Reference page: Available software

Types of computer codes for materials theory modelling

• Density functional theory (DFT) programs. Often with the DFT extended with for instance dynamical mean field theory (DMFT), GW, hybrid functionals.
• Atomistic modelling of magnetic or lattice degree of freedom with Monte Carlo or equations of motion solvers
• Modelling of quantum mechanical many body Hamiltonians
• ... and more

Density functional theory (DFT) codes

Challenges

• Often large complex codes. Difficult to grasp for individual developers and users
• High memory demand, scaling to different order with number of atoms/electrons/orbitals, plane wave cut-off, angular momentum cut-off, etc
• High memory bandwidth demand, for instance for large Fourier transforms

Strategies on heterogeneous supercomputers

• Make use of optimized CPU/GPU libraries
• Modularization of code - enabling for multiple backends
• Develop algorithms with scaled down memory bandwidth demand

The Vienna Ab initio Simulation Package (VASP)

• General purpose program for electronic structure calculations and quantum-mechanical molecular dynamics from first principles.

• VASP is licensed software. For NAISS systems, the VASP licenses are managed on the SUPR web portal.

• To use VASP on the Dardel CPU nodes

Reference page: General information about VASP

ABINIT

• General purpose program for electronic structure calculations and quantum-mechanical molecular dynamics, from first principles, using pseudopotentials and a planewave or wavelet basis. Example ABINIT job script

Reference page: General information about ABINIT

Elk

• Elk is an all-electron full-potential linearised augmented-planewave (FP-LAPW) code. Example job script for a hybrid MPI and OpenMP Elk calculation on Dardel

Reference page: General information about Elk

How to build Elk

• For maintaining and installing (new versions) of materials theory codes on Dardel, we are mainly using the EasyBuild system. To build Elk 9.5.14 under CPE 23.12, load and launch an EasyBuild with

• A program that has been EasyBuilt and installed on Dardel can (often) be straightforwardly ported to a build configuration for LUMI. Or vice versa, a build on LUMI can be ported for Dardel. The easyconfig build configuration for Elk on Dardel has been ported to LUMI. See and compare the easyconfigs

  • Dardel elk-9.5.14-cpeGNU-23.12.eb
  • LUMI Elk-8.7.10-cpeGNU-22.12.eb

Reference page: Installing software using EasyBuild

Other DFT codes available on Dardel (selected)

• The Relativistic Spin Polarized tookit (RSPt), a code for electronic structure calculations based on the Full-Potential Linear Muffin-Tin Orbital (FP-LMTO) method.

• The Quantum ESPRESSO integrated suite of open-source computer codes for electronic-structure calculations and materials modeling at the nanoscale

• CP2K, a program to perform atomistic and molecular simulations of solid state, liquid, molecular, and biological systems.

Materials theory codes available on Dardel via Spack

Some codes can be built and installed in your own file area using Spack. Load a Spack user module with ml PDC/23.12 spack-user/0.21.2.

Examples of Spack specs for building on Dardel

• Siesta spack install siesta@4.0.2%gcc@12.3.0
• Sirius spack install sirius@7.4.3%gcc@12.3.0 ^spla@1.5.5%cce@15.0.1
• BerkeleyGW spack install berkeleygw@3.0.1%gcc@12.3.0
• Yambo spack install yambo@5.1.1%gcc@12.3.0 +dp +openmp
• BigDFT spack install bigdft-core@1.9.2%gcc@12.3.0

Reference page: Installing software using Spack

Other materials theory programs and tools

• The Spglib library for finding and handling crystal symmetries
• The Uppsala Atomistic Spin Dynamics (UppASD) software package, a simulation suite to study magnetization dynamics by means of the atomistic version of the Landau-Lifshitz-Gilbert (LLG) equation.
• The Wannier90 open-source code for generating maximally-localized Wannier functions and using them to compute advanced electronic properties of materials with high efficiency and accuracy.
• Phonopy for modelling of phonons spack install py-phonopy@1.10.0%gcc@12.3.0

Exercise 1: Run a DFT simulation with ABINIT

Perform a calculation on two Dardel CPU compute nodes with the ABINIT package for modeling of condensed matter. The example calculation is a DFT simulation of the properties of the material SrVO3.

Exercise instructions: See Submit a batch job to the queue and scroll down to the heading Example 2: Submit a batch job to queue for a center installed software

Exercise 2: Build the most recent version of Elk

As of 20240320, the most recent version of Elk globally installed on Dardel is 9.5.14. How to build and make local intall of the newer version 10.0.15?

• First make a local installation of Elk 9.5.14. Why is the --rebuild flag needed?

• Use the easyfonfig elk-9.5.14-cpeGNU-23.12.eb as a template to construct a file elk-10.0.15-cpeGNU-23.12.eb. Then build and install locally with eb elk-10.0.15-cpeGNU-23.12.eb --robot.

Reference page: Installing software using EasyBuild

Exercise 3: Calculate the magnetic phase diagram of bcc Fe with UppASD

UppASD is a program for simulating atomistic spin dynamics at finite temperatures, which makes it possible to describe magnetization dynamics on an atomic level. Magnetic phase diagrams and thermodynamical properties of a magnetic Hamiltonian can be investigated with techniques for Monte Carlo simulations.

In this exercise you will calculate the magnetic phase diagram of bulk bcc Fe. Exercise instructions: Determination of Tc of a ferromagnetic material

Reference pages:

• UppASD manual
• UppASD tutorial
• UppASD autumn school 2022

Using Python virtual environment

Xin Li, Juan de Gracia

Why Use a Virtual Environment?

• Python packages are essential tools that help us accomplish various tasks in programming. Examples include:import numpy from pandas import DataFrame import matplotlib.pyplot as plt
• Projects have unique needs: Different projects may require different versions of the same package, leading to conflicts.
• Shared systems complicate package management: On High-Performance Computing (HPC) platforms, multiple users' needs might clash due to different package requirements.

Why virtual environment?

Without virtual environment, Python packages are installed globally:

• System site directory: Where Python is installed and accessible by all users.python3 -c 'import site; print(site.getsitepackages())'
• User base directory: A user-specific area to avoid needing system-wide permissions.echo $HOME/.local

Exercise: Check python site directory

Python virtual environment

• What is a Virtual Environment? An isolated environment allowing you to manage packages for individual projects without conflicts and without interfering with the outside world.
• How to Manage Virtual Environments?
  • Using Python's built-in venv module.
  • Using conda, a powerful package manager and environment manager.

Virtual environment with venv

• Recommendation: use with cray-pythonml cray-python/3.11.5 cd $HOME python3 -m venv myenv source myenv/bin/activate Your prompt changes to indicate the active environment:(myvenv) user@uan01:~>

Virtual environment with venv

• Check site-packages directory

  (myenv) user@uan01:~> python3 -c 'import site; print(site.getsitepackages())' ['/cfs/klemming/home/u/user/myenv/lib/python3.9/site-packages']

• Install a Python package

  (myenv) user@uan01:~> python3 -m pip install yapf (myenv) user@uan01:~> which yapf /cfs/klemming/home/u/user/myenv/bin/yapf

Virtual environment with venv

• Deactivate a virtual environment(myenv) user@uan01:~> deactivate user@uan01:~>

Virtual environment with conda

• Load Anaconda: Prepares your system to use Anaconda packages and tools.

  ml PDC/23.12 anaconda3/2024.02-1

• Initialize Conda: Sets up Conda in your current shell session.

  source conda_init_bash.sh

• Your prompt changes to show (base), indicating Conda is ready.

  (base) user@uan01:~>

Virtual environment with conda

• Content of conda_init_bash.sh

  # >>> conda initialize >>> # >>> conda initialize >>> # !! Contents within this block are managed by 'conda init' !! __conda_setup="$('//pdc/software/23.12/eb/software/anaconda3/2024.02-1-cpeGNU-23.12/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" if [ $? -eq 0 ]; then eval "$__conda_setup" else if [ -f "/pdc/software/23.12/eb/software/anaconda3/2024.02-1-cpeGNU-23.12/etc/profile.d/conda.sh" ]; then . "/pdc/software/23.12/eb/software/anaconda3/2024.02-1-cpeGNU-23.12/etc/profile.d/conda.sh" else export PATH="/pdc/software/23.12/eb/software/anaconda3/2024.02-1-cpeGNU-23.12/bin:$PATH" fi fi unset __conda_setup # <<< conda initialize <<<

• conda init bash may append the above script to your ~/.bashrc but this is not recommended on an HPC system.

• Note: Alternatively we can create a ~/.bashrc.conda.dardel file and source it.

Virtual environment with conda

• Create and activate a conda environment:

  (base) user@uan01:~> conda create --name my-conda-env (base) user@uan01:~> conda activate my-conda-env

• Your prompt now indicates the active environment, e.g., (my-conda-env).

  (my-conda-env) user@uan01:~>

Virtual environment with conda

• Check Installation Path: Understand where Conda places your environment's packages.(my-conda-env) user@uan01:~> python3 -c 'import site; print(site.getsitepackages())'
• Anaconda's Default Directory: Packages are placed in Anaconda's directory unless specified otherwise.['/pdc/software/23.12/eb/software/anaconda3/2024.02-1/lib/python3.11/site-packages']
• Important: We do not want to create our environments under the anaconda3 instalation directory!

Virtual environment with conda

• We have seen that we cannot rely on the default behaviour of conda create.

• To circumvent this issue we will add a ~/.condarc file specifying where environments and packages are installed

  pkgs_dirs: - /cfs/klemming/home/u/username/conda-dirs/pkgs envs_dirs: - /cfs/klemming/home/u/username/conda-dirs/envs

• Verify Package Location: Confirm that packages are in your environment's directory.

  (my-conda-env) user@uan01:~> python3 -c 'import site; print(site.getsitepackages())' ['/cfs/klemming/home/u/user/conda-dirs/envs/my-conda-env/lib/python3.9/site-packages']

Virtual environment with conda

• Deactivate: Return to the base environment or your system's default settings.

  (my-conda-env) user@uan01:~> conda deactivate (base) user@uan01:~>

Customizing Environment Location with --prefix

• Replace /path/to/myenv with your desired location.
  Activating with --prefix:

• Advantage: One can install the environment in a project directory and not be bounded to the memory limitation of the $HOME

Working with Jupyter Notebooks

• Disclaimer: Jupyter notebooks can be run trough Thinlinc.

• Make sure you have installed jupyter notebooks in your conda environment:

  conda install jupyterlab

• Then start a Jupyter notebooks without browser:

  jupyter-notebook

• Important: Do not run a Jupyter notebook in the login node, instead get an interactive node allocation:

  salloc --nodes=<n> -t 1:00:00 -A <project>