Fine-tuning LLMs with Multi-GPU Training

This hands-on workshop is designed to provide participants with practical experience in fine-tuning large language models (LLMs) with a special focus on LLaMA on high-performance computing () systems.

The course is open to all and free of charge, but registration is mandatory.

Date and time

Start: Sep 02 2025 09:30
End: Sep 02 2025 15:30

Organised by

NRIS

Target audience

Researchers, developers, and students who are familiar with Python and want to gain practical insights into scalable LLM training with LLaMA.

More information about registration and the GPU-course

Learning outcomes

By the end of this workshop, participants will be able to:

Learn the basics of LLaMA.
Set up a distributed training environment using PyTorch, Torchao, and Torchtune on an HPC system.
Fine-tune a LLaMA model using LoRA techniques on a single GPU and then scale up to multiple GPUs.
Perform inference tasks like summarization using the fine-tuned model.
Monitor GPU usage and GPU memory utilization.