nf-core/configs: vsc_kul_uhasselt
genius profile for use on the genius cluster of the VSC HPC.
nf-core/configs: KU Leuven/UHasselt Tier-2 High Performance Computing Infrastructure (VSC)
NB: You will need an account to use the HPC cluster to run the pipeline.
- Install Nextflow on the cluster
conda create --name nf-core python=3.12 nf-core nextflow
:::note
A nextflow module is available that can be loaded module load Nextflow but it does not support plugins. So it’s not recommended
:::
- Set up the environment variables in
~/.bashrcor~/.bash_profile:
:::note If you have access to dedicated nodes, you can export these as a command separated list. These queues will only be used if specified task requirements are not available in the normal partitions but they are available in dedicated partitions. AMD is considered a dedicated partition. :::
export SLURM_ACCOUNT="<your-credential-account>"
# Comma-separated list of available dedicated partitions (if any)
# For example: export VSC_DEDICATED_QUEUES="dedicated_big_bigmem,dedicated_big_gpu"
export VSC_DEDICATED_QUEUES="<available-dedicated-partitions>"
# Needed for running Nextflow jobs
export NXF_HOME="$VSC_SCRATCH/.nextflow"
export NXF_WORK="$VSC_SCRATCH/work"
# Needed for running Apptainer containers
export APPTAINER_CACHEDIR="$VSC_SCRATCH/.apptainer/cache"
export APPTAINER_TMPDIR="$VSC_SCRATCH/.apptainer/tmp"
export NXF_CONDA_CACHEDIR="$VSC_SCRATCH/miniconda3/envs"
# Optional tower key
# export TOWER_ACCESS_TOKEN="<your_tower_access_token>"
# export NXF_VER="<version>" # make sure it's larger then 24.10.1
:::warning The current config is setup with array jobs. Make sure nextflow version >= 24.10.1, read array jobs in nextflow you can do this in
export NXF_VER=24.10.1
:::
- Make the submission script.
NB: you should go to the cluster you want to run the pipeline on. You can check what clusters have the most free space using following command
sinfo --cluster wice|genius.
$ more job.pbs
#!/bin/bash -l
#SBATCH --account=...
#SBATCH --chdir=....
#SBATCH --partition=batch_long
#SBATCH --nodes="1"
#SBATCH --ntasks-per-node="1"
# module load Nextflow # does not support plugins
conda activate nf-core
nextflow run <pipeline> -profile vsc_kul_uhasselt,<CLUSTER> <Add your other parameters>
NB: You have to specify your credential account, by setting
export SLURM_ACCOUNT="<your-credential-account>"else the jobs will fail!
Here the cluster options are:
- genius
- genius_gpu
- wice
- wice_gpu
- superdome
NB: The vsc_kul_uhasselt profile is based on a selected amount of SLURM partitions. The profile will select to its best ability the most appropriate partition for the job. Including modules with a label containing
gpuwill be allocated to a gpu partition when the ‘normal’geniusprofile is selected. Select thegenius_gpuorwice_gpuprofile to force the job to be allocated to a gpu partition. NB: If the module does not haveacceleratorset, it will determine the number of GPUs based on the requested resources.
Use the --cluster option to specify the cluster you intend to use when submitting the job:
sbatch --cluster=wice|genius job.slurmÂ
All of the intermediate files required to run the pipeline will be stored in the work/ directory. It is recommended to delete this directory after the pipeline has finished successfully because it can get quite large, and all of the main output files will be saved in the results/ directory anyway.
- Optional use nf-co2footprint
You can monitor the CO2 usage of your pipeline using the nf-co2footprint plugin using a nextflow version =>24.10.6. Monitoring the CO2 usage is fully optional and will only be activated when running the following command-line.
nextflow run <pipeline> -profile <CLUSTER> -plugins nf-co2footprint@1.0.0 --outdir your_output_folder <Add your other parameters>
Config file
// Default to /tmp directory if $VSC_SCRATCH scratch env is not available,// see: https://github.com/nf-core/configs?tab=readme-ov-file#adding-a-new-config// NOTE: All helper variables and functions have been inlined into closures for// compatibility with Nextflow config parser v2 (Nextflow >= 25.10).
// Perform work directory cleanup when the run has succesfully completed// cleanup = true
// Reduce the job submit rate to about 50 per minute, this way the server won't be bombarded with jobs// Limit queueSize to keep job rate under control and avoid timeoutsexecutor { submitRateLimit = '50/1min' queueSize = 50 exitReadTimeout = "10min"}
// Add backoff strategy to catch cluster timeouts and proper symlinks of files in scratch to the work directoryprocess { executor = 'slurm' stageInMode = "symlink" stageOutMode = "rsync" errorStrategy = { sleep(Math.pow(2, task.attempt ?: 1) * 200 as long); return 'retry' } maxRetries = 3}
// Specify that singularity should be used and where the cache dir will be for the imagessingularity { enabled = true autoMounts = true cacheDir = "${System.getenv('VSC_SCRATCH') ?: '/tmp'}/.singularity" pullTimeout = "30 min"}
params { config_profile_contact = 'GitHub: @Joon-Klaps - Email: joon.klaps@kuleuven.be' config_profile_url = 'https://docs.vscentrum.be/en/latest/index.html'}
co2footprint { traceFile = "${params.get('outdir', "$launchDir")}/pipeline_info/co2footprint_trace_${new java.util.Date().format('yyyy-MM-dd_HH-mm-ss')}.txt" summaryFile = "${params.get('outdir', "$launchDir")}/pipeline_info/co2footprint_summary_${new java.util.Date().format('yyyy-MM-dd_HH-mm-ss')}.txt" reportFile = "${params.get('outdir', "$launchDir")}/pipeline_info/co2footprint_report_${new java.util.Date().format('yyyy-MM-dd_HH-mm-ss')}.html" location = 'BE' //pue = 1.33 // replace with PUE of your data center machineType = 'compute cluster' // set to 'compute cluster', 'local', or 'cloud'}
env { APPTAINER_TMPDIR="${System.getenv('VSC_SCRATCH') ?: '/tmp'}/.apptainer/tmp" APPTAINER_CACHEDIR="${System.getenv('VSC_SCRATCH') ?: '/tmp'}/.apptainer/cache"}
// AWS maximum retries for errors (This way the pipeline doesn't fail if the download fails one time)aws { maxErrorRetry = 3}
/* * Queue Selection Logic for HPC Environments * =========================================== * Each process directive closure below contains inlined queue selection logic * for compatibility with Nextflow config parser v2 (no top-level def allowed). * * Constants used throughout: * TIME_THRESHOLD = 72.h (threshold for long-running jobs) * MEMORY_THRESHOLD_GENIUS = 175.GB (bigmem threshold for GENIUS) * MEMORY_THRESHOLD_WICE = 239.GB (high-memory threshold for WICE) * * Queue selection rules: * GENIUS CPU: memory >= 175GB → bigmem/bigmem_long; else → batch/batch_long * GENIUS GPU: memory >= 175GB → gpu_v100; else → gpu_p100/amd (with dedicated queue checks) * WICE CPU: memory >= 239GB → bigmem,hugemem (time capped at 72h); else → batch/batch_icelake * WICE GPU: memory >= 239GB → gpu_h100; else → gpu_a100,gpu (time capped at 72h unless dedicated) */
// Define profiles for each clusterprofiles { genius { params.config_profile_description = 'genius profile for use on the genius cluster of the VSC HPC.'
process { // 768 - 65 so 65GB for overhead, max is 720000MB resourceLimits = [ memory: 703.GB, cpus: 36, time: 168.h ] beforeScript = { // determineGeniusQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 175.GB def isLongRunning = task.time >= 72.h def hasDedicatedBigmem = availQueues.contains('dedicated_big_bigmem') def queueName if (isHighMemory) { queueName = isLongRunning ? (hasDedicatedBigmem ? 'dedicated_big_bigmem' : 'bigmem_long') : 'bigmem' } else { queueName = isLongRunning ? 'batch_long' : 'batch' } 'module load cluster/genius/' + queueName.split(',')[0] } queue = { // determineGeniusQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 175.GB def isLongRunning = task.time >= 72.h def hasDedicatedBigmem = availQueues.contains('dedicated_big_bigmem') if (isHighMemory) { return isLongRunning ? (hasDedicatedBigmem ? 'dedicated_big_bigmem' : 'bigmem_long') : 'bigmem' } isLongRunning ? 'batch_long' : 'batch' } clusterOptions = { // determineGeniusQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 175.GB def isLongRunning = task.time >= 72.h def hasDedicatedBigmem = availQueues.contains('dedicated_big_bigmem') def queueName if (isHighMemory) { queueName = isLongRunning ? (hasDedicatedBigmem ? 'dedicated_big_bigmem' : 'bigmem_long') : 'bigmem' } else { queueName = isLongRunning ? 'batch_long' : 'batch' } queueName =~ /dedicated/ ? "--clusters=genius --account=lp_big_genius_cpu" : "--clusters=genius --account=${System.getenv('SLURM_ACCOUNT') ?: null}" }
withLabel: '.*gpu.*'{ resourceLimits = [ memory: 703.GB, cpus: 36 , time: 168.h ] beforeScript = { // determineGeniusGpuQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 175.GB def isLongRunning = task.time >= 72.h def hasDedicatedGpu = availQueues.contains('dedicated_rega_gpu') def hasAmdGpu = availQueues.contains('amd') def queueName if (isHighMemory) { queueName = isLongRunning ? 'gpu_v100_long' : 'gpu_v100' } else if (isLongRunning) { queueName = hasDedicatedGpu ? 'dedicated_rega_gpu' : (hasAmdGpu ? 'amd_long' : 'gpu_p100_long') } else { queueName = hasAmdGpu ? 'amd' : 'gpu_p100' } 'module load cluster/genius/' + queueName.split(',')[0] } apptainer.runOptions = '--containall --cleanenv --nv' singularity.runOptions = '--containall --cleanenv --nv' queue = { // determineGeniusGpuQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 175.GB def isLongRunning = task.time >= 72.h def hasDedicatedGpu = availQueues.contains('dedicated_rega_gpu') def hasAmdGpu = availQueues.contains('amd') if (isHighMemory) { return isLongRunning ? 'gpu_v100_long' : 'gpu_v100' } if (isLongRunning) { if (hasDedicatedGpu) return 'dedicated_rega_gpu' if (hasAmdGpu) return 'amd_long' return 'gpu_p100_long' } hasAmdGpu ? 'amd' : 'gpu_p100' } clusterOptions = { def gpus = task.accelerator?.request ?: Math.max(1, Math.floor((task.cpus ?:1)/9) as int) "--gres=gpu:${gpus} --clusters=genius --account=${System.getenv('SLURM_ACCOUNT') ?: null}" } } } }
genius_gpu { params.config_profile_description = 'genius_gpu profile for use on the genius cluster of the VSC HPC.' apptainer.runOptions = '--containall --cleanenv --nv' singularity.runOptions = '--containall --cleanenv --nv'
process { // 768 - 65 so 65GB for overhead, max is 720000MB resourceLimits = [ memory: 703.GB, cpus: 36, time: 168.h] beforeScript = { // determineGeniusGpuQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 175.GB def isLongRunning = task.time >= 72.h def hasDedicatedGpu = availQueues.contains('dedicated_rega_gpu') def hasAmdGpu = availQueues.contains('amd') def queueName if (isHighMemory) { queueName = isLongRunning ? 'gpu_v100_long' : 'gpu_v100' } else if (isLongRunning) { queueName = hasDedicatedGpu ? 'dedicated_rega_gpu' : (hasAmdGpu ? 'amd_long' : 'gpu_p100_long') } else { queueName = hasAmdGpu ? 'amd' : 'gpu_p100' } 'module load cluster/genius/' + queueName.split(',')[0] } queue = { // determineGeniusGpuQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 175.GB def isLongRunning = task.time >= 72.h def hasDedicatedGpu = availQueues.contains('dedicated_rega_gpu') def hasAmdGpu = availQueues.contains('amd') if (isHighMemory) { return isLongRunning ? 'gpu_v100_long' : 'gpu_v100' } if (isLongRunning) { if (hasDedicatedGpu) return 'dedicated_rega_gpu' if (hasAmdGpu) return 'amd_long' return 'gpu_p100_long' } hasAmdGpu ? 'amd' : 'gpu_p100' } clusterOptions = { def gpus = task.accelerator?.request ?: Math.max(1, Math.floor((task.cpus ?:1)/9) as int) "--gres=gpu:${gpus} --clusters=genius --account=${System.getenv('SLURM_ACCOUNT') ?: null}" } } }
wice { params.config_profile_description = 'wice profile for use on the Wice cluster of the VSC HPC.'
process { // max is 2016000 resourceLimits = [ memory: 1968.GB, cpus: 72, time: 168.h ] beforeScript = { // determineWiceQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 239.GB def isLongRunning = task.time >= 72.h def hasDedicatedQueue = availQueues.contains('dedicated_big_bigmem') def queueName if (isHighMemory) { if (isLongRunning && hasDedicatedQueue) { queueName = 'dedicated_big_bigmem' } else { task.time = task.time > 72.h ? 72.h : task.time queueName = 'bigmem,hugemem' } } else { queueName = isLongRunning ? 'batch_long,batch_icelake_long,batch_sapphirerapids_long' : 'batch,batch_sapphirerapids,batch_icelake' } 'module load cluster/wice/' + queueName.split(',')[0] } queue = { // determineWiceQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 239.GB def isLongRunning = task.time >= 72.h def hasDedicatedQueue = availQueues.contains('dedicated_big_bigmem') if (isHighMemory) { if (isLongRunning && hasDedicatedQueue) { return 'dedicated_big_bigmem' } task.time = task.time > 72.h ? 72.h : task.time return 'bigmem,hugemem' } isLongRunning ? 'batch_long,batch_icelake_long,batch_sapphirerapids_long' : 'batch,batch_sapphirerapids,batch_icelake' } clusterOptions = { // determineWiceQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 239.GB def isLongRunning = task.time >= 72.h def hasDedicatedQueue = availQueues.contains('dedicated_big_bigmem') def queueName if (isHighMemory) { if (isLongRunning && hasDedicatedQueue) { queueName = 'dedicated_big_bigmem' } else { task.time = task.time > 72.h ? 72.h : task.time queueName = 'bigmem,hugemem' } } else { queueName = isLongRunning ? 'batch_long,batch_icelake_long,batch_sapphirerapids_long' : 'batch,batch_sapphirerapids,batch_icelake' } queueName =~ /dedicated/ ? "--clusters=wice --account=lp_big_wice_cpu" : "--clusters=wice --account=${System.getenv('SLURM_ACCOUNT') ?: null}" }
withLabel: '.*gpu.*' { resourceLimits = [ memory: 703.GB, cpus: 64, time: 168.h ] apptainer.runOptions = '--containall --cleanenv --nv' singularity.runOptions = '--containall --cleanenv --nv' beforeScript = { // determineWiceGpuQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 239.GB def isLongRunning = task.time >= 72.h def hasDedicatedQueue = isHighMemory ? availQueues.contains('dedicated_big_gpu_h100') : availQueues.contains('dedicated_big_gpu') if (isLongRunning && !hasDedicatedQueue) { task.time = task.time > 72.h ? 72.h : task.time } def queueName = isHighMemory ? ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') : ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu') 'module load cluster/wice/' + queueName.split(',')[0] } queue = { // determineWiceGpuQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 239.GB def isLongRunning = task.time >= 72.h def hasDedicatedQueue = isHighMemory ? availQueues.contains('dedicated_big_gpu_h100') : availQueues.contains('dedicated_big_gpu') if (isLongRunning && !hasDedicatedQueue) { task.time = task.time > 72.h ? 72.h : task.time } isHighMemory ? ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') : ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu') } clusterOptions = { // determineWiceGpuQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 239.GB def isLongRunning = task.time >= 72.h def hasDedicatedQueue = isHighMemory ? availQueues.contains('dedicated_big_gpu_h100') : availQueues.contains('dedicated_big_gpu') if (isLongRunning && !hasDedicatedQueue) { task.time = task.time > 72.h ? 72.h : task.time } def gpus = task.accelerator?.request ?: Math.max(1, Math.floor((task.cpus ?:1)/16) as int) def queueValue = isHighMemory ? ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') : ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu') queueValue =~ /dedicated_big_gpu_h100/ ? "--clusters=wice --account=lp_big_wice_gpu_h100 --gres=gpu:${gpus}" : queueValue =~ /dedicated_big_gpu/ ? "--clusters=wice --account=lp_big_wice_gpu --gres=gpu:${gpus}" : "--clusters=wice --account=${System.getenv('SLURM_ACCOUNT') ?: null} --gres=gpu:${gpus}" } } } }
wice_gpu { params.config_profile_description = 'wice_gpu profile for use on the Wice cluster of the VSC HPC.' apptainer.runOptions = '--containall --cleanenv --nv' singularity.runOptions = '--containall --cleanenv --nv'
process { // 768 - 65 so 65GB for overhead, max is 720000MB resourceLimits = [ memory: 703.GB, cpus: 64, time: 168.h ] beforeScript = { // determineWiceGpuQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 239.GB def isLongRunning = task.time >= 72.h def hasDedicatedQueue = isHighMemory ? availQueues.contains('dedicated_big_gpu_h100') : availQueues.contains('dedicated_big_gpu') if (isLongRunning && !hasDedicatedQueue) { task.time = task.time > 72.h ? 72.h : task.time } def queueName = isHighMemory ? ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') : ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu') 'module load cluster/wice/' + queueName.split(',')[0] } queue = { // determineWiceGpuQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 239.GB def isLongRunning = task.time >= 72.h def hasDedicatedQueue = isHighMemory ? availQueues.contains('dedicated_big_gpu_h100') : availQueues.contains('dedicated_big_gpu') if (isLongRunning && !hasDedicatedQueue) { task.time = task.time > 72.h ? 72.h : task.time } isHighMemory ? ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') : ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu') } clusterOptions = { // determineWiceGpuQueue logic def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',') def isHighMemory = task.memory >= 239.GB def isLongRunning = task.time >= 72.h def hasDedicatedQueue = isHighMemory ? availQueues.contains('dedicated_big_gpu_h100') : availQueues.contains('dedicated_big_gpu') if (isLongRunning && !hasDedicatedQueue) { task.time = task.time > 72.h ? 72.h : task.time } def gpus = task.accelerator?.request ?: Math.max(1, Math.floor((task.cpus ?:1)/16) as int) def queueValue = isHighMemory ? ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') : ((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu') queueValue =~ /dedicated_big_gpu_h100/ ? "--clusters=wice --account=lp_big_wice_gpu_h100 --gres=gpu:${gpus}" : queueValue =~ /dedicated_big_gpu/ ? "--clusters=wice --account=lp_big_wice_gpu --gres=gpu:${gpus}" : "--clusters=wice --account=${System.getenv('SLURM_ACCOUNT') ?: null} --gres=gpu:${gpus}" } } }
superdome { params.config_profile_description = 'superdome profile for use on the genius cluster of the VSC HPC.'
process { clusterOptions = {"--clusters=genius --account=${System.getenv('SLURM_ACCOUNT') ?: null}"} beforeScript = 'module load cluster/genius/superdome' // 6000 - 228 so 228GB for overhead, max is 5910888MB resourceLimits = [ memory: 5772.GB, cpus: 14, time: 168.h]
queue = { task.time <= 72.h ? 'superdome' : 'superdome_long' } } }}