Using Scratch Space (and Pixi/Conda) Effectively with Slurm

Ed Davis
Bioinformatics Manager

Center for Quantitative Life Sciences, Oregon State University

2026-01-28

Today’s Agenda

  1. The /scratch drive — What’s changed
  2. Slurm makes things easier — No more workarounds
  3. The new workflow — Check out, set up, submit
  4. Pixi/Conda environments — They just work now
  5. One thing to watch — Architecture matching

Note: Room location (ALS 3005) may change in future meetings — check announcements!

The /scratch Drive

A quick orientation

/scratch is the Local Drive

What changed:

  • Local drive is now /scratch (was /data)
  • Makes more sense — “scratch” implies temporary workspace
  • /data on login nodes was used for web resources, which caused confusion

The good news: Your $TMPDIR variable points to /scratch automatically on compute nodes.

No action needed on your part — it just works.

Volume Layout

Volume Purpose Notes
/tmp Small OS temp space Login/file nodes
/data Non-temporary storage Web server resources; legacy symlink on some nodes
/scratch Your workspace Fast local storage for processing

Bottom line: Use $TMPDIR in your scripts and it’ll do the right thing everywhere.

Slurm Makes Things Easier

The old workarounds are gone

Remember SGE?

SGE had some frustrating limitations:

  • Couldn’t keep conda/pixi environments activated through job submission
  • Difficult to copy data to scratch before jobs ran
  • Had to submit from specific “submit hosts”

We built workarounds into hqsub (--local, --conda) to deal with these.

Slurm doesn’t have these problems.

What Slurm Fixed

  • Environment inheritance works — activate your env, submit, it stays activated

  • Submit from anywhere — including compute nodes and /scratch itself

  • $TMPDIR just works — set automatically whether you use salloc or hqsub

The goal was always to reduce barriers to your research. Slurm lets us do that.

The New Workflow

Check out, set up, submit

The Pattern

# 1. Check out a node
salloc -c 1 -p core -w chrom1

# 2. Set up your workspace
mkdir -p $TMPDIR/$USER/project
cd $TMPDIR/$USER/project
cp /nfs4/my_lab/data.fastq .

# 3. Activate your environment (if using pixi/conda)
pixi shell

# 4. Submit your job
hqsub "my_analysis.sh data.fastq" -r job.my_analysis -q core -w chrom1 -P 1

That’s it. No special flags, no workarounds.

When to Use /scratch

Heavy input I/O — copy inputs first:

cp /nfs4/my_lab/big_reference.fa $TMPDIR/$USER/project/
hqsub "my_tool --db $TMPDIR/$USER/project/big_reference.fa --query input.fastq" \
  -r job.my_tool -q core -w chrom1

Repeated reads from NFS are slow; one copy up front saves time overall.

Heavy output I/O — write output to scratch, read from NFS:

hqsub "my_tool --input /nfs4/my_lab/reads.fastq \
  --output $TMPDIR/$USER/project/results/" -r job.my_tool -q core -w chrom1
# Then copy results back when done
cp -r $TMPDIR/$USER/project/results/ /nfs4/my_lab/results/

Avoids hammering NFS with many small writes; copy back when done.

Always remember to specify a node when using scratch

Important: Use -w <node> to pin your job to the same node where you copied data. Each node has its own /scratch — if your job lands on a different node, your files won’t be there.

Light I/O — skip scratch entirely:

If your job does a single read and single write, NFS is fine. No need to add complexity.

Tip: You can save any of these commands to a shell script and submit that instead:

hqsub "bash my_analysis.sh" -r job.my_analysis -q core

Why This Works

When you submit from within your salloc session:

  • Your environment stays activated
  • You’re already on /scratch — no copying needed during the job
  • Job output lands right where you are (assuming you request the same partition and node)

You can watch progress, check outputs, and submit follow-up jobs interactively.

salloc Examples

# Basic checkout
salloc -c 1

# Specific partition
salloc -c 1 -p core

# Specific node with more CPUs
salloc -c 8 -p core -w chrom1

# ARM64 node (GH200)
salloc -c 1 -p grace -A grace

Using GPUs

When I need to use a GPU, I use this workflow:

  • checkout a GPU when testing interactively
    • salloc -G 1 -p grace -A grace
  • relinquish the GPU reservation when I’m sure the job runs
  • checkout a CPU on the node so I can monitor the job
  • submit job using hqsub, reserve GPU with -G flag

Pixi/Conda Environments

They just work now

Using Pixi with Slurm

Option 1: Activate then submit

cd /path/to/my/project
pixi shell
hqsub -c 4 "my_command --threads 4"

Option 2: Use pixi run

hqsub -c 4 "pixi run my_command --threads 4"

Both work. Pick whichever fits your workflow.

Check out https://pixi.prefix.dev/latest/first_workspace/ and https://docs.hpc.oregonstate.edu/cqls/software/conda/pixi/ for information about using pixi in place of conda.

Complete Example

# Check out a node
salloc -c 1 -p core -w chrom1

# Set up on scratch
mkdir -p $TMPDIR/my_analysis
cd $TMPDIR/my_analysis
cp /nfs4/my_lab/samples/*.fastq.gz .

# Set up pixi environment
pixi init .
pixi add fastp seqkit

hqsub "pixi run fastp -i sample.fastq.gz -o clean.fastq.gz -w $SLURM_CPUS_ON_NODE" -p 8 # can also use $NPROCS

One Thing to Watch

Architecture matching

Different CPU Architectures

We have nodes with different architectures:

  • x86_64 — Traditional Intel/AMD (most nodes)
  • aarch64 — ARM64 (GH200 nodes)
  • ppc64le — IBM Power

Pixi/Conda packages are architecture-specific. If you install on x86_64 and try to run on ARM64, it won’t work.

The Solution

Install and submit from the same architecture.

# Want to run on ARM64? Check out an ARM64 node first.
salloc -c 1 -p sharpton_gh

# Now pixi will fetch the correct binaries
cd /path/to/project
pixi install

# And your job will run on the right architecture
pixi shell
hqsub "my_command" -P 8 -r job.my_command -q sharpton_gh # Make sure the partition/node matches

When you are using multiple CPU arch, this is the primary thing you need to think about to be successful.

What About Those hqsub Options?

--local and --conda

You might see these options in hqsub --help:

➜ hqsub -h

 Usage: queue submit [OPTIONS] "COMMAND"

 Alias: hqsub. Submit queueing job "COMMAND". Multiple lines of commands supported using piped input from STDIN. If providing as argument and there are spaces in the command, use quotes to provide
 the command, either '' or "" will work!  Do not use -c '', just provide the command as argument.

...

 -= Examples =-
 • Batch job:
     hqsub -q '*' -r - 'blastp -db nr -query ...'
 • Batch job, watching for job submission:
     hqsub -q '*' -r - 'blastp -db nr -query ...' --watch
 • Batch job with multiple processors (use $NPROCS in command):
     hqsub -q '*' -r - -p 8 'blastp -db nr -num_threads $NPROCS -query ...'
 • Array job with multiple processors, reading commands from stdin:
     bash submit_jobs.sh | hqsub -q '*' -r - -p 8 -t array -
 • Using conda activate:
     hqsub --conda -q '*' -r - 'conda activate bakta; which bakta'
 • Using /data drive:
     hqsub --local-drive pertask -q '*' -r - ...

The --conda and --local-drive are no longer necessary (although –local-drive still does work if you don’t want to manually copy).

These are generally for SGE compatibility.

Documentation Updates

We’re working on updating the docs on the website and in hpcman to make these changes more obvious.

If something’s unclear or you run into issues, let us know, so we can improve the docs for everyone.

Summary

Key Takeaways

  1. /scratch is your local workspace — use $TMPDIR, it’s automatic

  2. Slurm removed the old barriers — environment activation, submitting from scratch, it all just works

  3. The workflow is simple: salloc → set up → pixi shell → hqsub

  4. Match your architecture for pixi installs on ARM/Power nodes

  5. --local and --conda? You don’t need them on Slurm

More Information

Documentation:

Get Help:

  • Support Request
  • Slack: BUG and AI channels
  • Email: Ed Davis, Brent Kronmiller

Winter 2026 Schedule

Date Topic Presenter(s)
Jan 14 Introducing the RCO Users Group Ed, Chris, Dirk
Jan 28 Using scratch space with Slurm Ed Davis (CQLS)
Feb 11 HCIC/AI focus (TBD) Dirk Petersen (UIT)
Feb 25 HPC focus (TBD) Fabian Wittmers (Micro)
Mar 11 HCIC/AI focus (TBD) Dirk Petersen (UIT)

Wednesdays, 12-1pm • ALS 3005 • Zoom available

Questions?

Join us: