← Back to Projects

Model Fine-Tuning — LoRA, QLoRA & RAG

WIP
PythonPyTorchLoRAQLoRAPhiGradioRAG

Overview

A personal, from-scratch deep dive into fine-tuning open-source LLMs on consumer hardware, with no prior ML background going in. Started as an attempt to build a locally-run assistant, evolved through several model and technique changes as constraints and results forced rethinking the approach at each step. The goal was never a polished product — it was to learn the full pipeline: model selection, fine-tuning, inference, and retrieval, end to end, by hitting every wall along the way.

Current state: multi-adapter QLoRA setup on Phi 2.7B (moved from plain LoRA once Phi’s size stopped fitting the 6GB card), served through Gradio, with a RAG-based context system in progress to solve adapter-switching context loss.

Hardware & Constraints

  • Lenovo LOQ gaming laptop
  • GPU: RTX 3050, 6GB VRAM
  • This is the ceiling for every decision on this project — model size, technique (LoRA vs full fine-tune vs QLoRA), and batch size were all picked around what fits in 6GB.

Concepts

Brief, plain explanations — written for someone who, like me at the start, doesn’t know this world yet.

Base vs Instruct models A base model is a raw next-token predictor — give it a prompt and it continues the text rather than answering it. An instruct model is a base model further tuned to behave conversationally (follow instructions, answer questions). Mixing these up wastes time — learned this the hard way early on.

LoRA (Low-Rank Adaptation) Instead of updating all of a model’s weights, LoRA freezes the original model and trains a small set of additional parameters (often ~1%) that get added on top. Drastically cuts the VRAM and compute needed to fine-tune, which is what makes it viable on a 6GB card at all.

QLoRA LoRA applied on top of a quantized (lower-precision) version of the base model. Same idea, smaller memory footprint — the difference that made training on 6GB actually practical rather than borderline.

Adapters The output of a LoRA/QLoRA training run is an “adapter” — a small file of weights, not a full model. Adapters can be swapped in and out of the same base model at inference time, and (with the right library support) more than one can be attached at once.

RAG (Retrieval-Augmented Generation) Instead of baking knowledge into the model’s weights, RAG retrieves relevant information at inference time and feeds it into the prompt as context. Used here for two things: (1) the original plan of feeding in live stock data, and (2) a newer use — patching the context the model loses when adapters are swapped.

Timeline

A short version of the model/technique changes, each driven by a specific result:

Stage Model Technique Why the change
v1 Qwen3 1.7B (base — mistakenly downloaded instead of instruct) LoRA Starting point; wrong model variant discovered after setup
v1.1 Qwen3 1.7B LoRA, trained on Claude-generated stock-data JSON First real training run; worked only when prompts closely matched training data
v2 (current) Phi 2.7B (microsoft/phi-2) QLoRA, multi-adapter (general adapter trained on UltraChat; task adapters planned) Qwen3 outputs weren’t coherent enough; Phi is larger and gave better results, but too big for plain LoRA to fit in 6GB VRAM — switched to QLoRA (4-bit quant) specifically to accommodate the bigger model

(Groq’s cloud API was tested briefly as an alternative to local training/inference, but the project intentionally stayed local — the point was learning to run and train everything on this hardware, not to outsource it.)

Training

Base model: microsoft/phi-2, loaded 4-bit quantized (BitsAndBytesConfig, nf4, double quant) — this is what makes QLoRA training fit on 6GB.

Data prep (prepare_dataset.py) — pulls 8,000 examples (seeded, shuffled) from HuggingFaceH4/ultrachat_200k, keeps the first user/assistant turn from each, formats as Instruct: ... \nOutput: / response pairs, splits 200 off for eval.

Training (train_general_adapter.py) — QLoRA via PEFT:

  • r=16, lora_alpha=32, lora_dropout=0.05
  • target modules: Wqkv, out_proj, fc1, fc2 (Phi-2’s attention/MLP layers)
  • batch size 2, gradient accumulation 4, 2 epochs, cosine LR schedule, bf16
  • checkpoint-resumable — picks up from the latest checkpoint-* automatically
  • labels masked over the prompt span so loss is only computed on the response
  • saves to adapters/general_adapter/

This produces the general-purpose adapter — the always-attached one, trained on broad conversational data rather than a specific task.

Multi-Adapter Setup

Design (project internally named Robin):

  • Base model — Phi-2.7B, never modified
  • General-purpose adapter — LoRA, always active, never merged into base weights
  • Task-specific adapters — one per task, trained with the general adapter attached and active, so training matches actual runtime behavior rather than training against the bare base model
  • Switching — manual only for now, no automatic routing/classifier
  • RAG pipeline (planned, not yet built) — embedding model + vector store (FAISS/Chroma) to index model output on task switch and inject retrieved context into the next adapter’s input, addressing the context-loss problem

Planned structure:

Robin_LoRA/
├── src/
│   ├── model.py       # base model + adapter loading
│   ├── train.py       # training loop
│   ├── inference.py   # multi-adapter inference
│   ├── rag.py         # embeddings, vector store, retrieval
│   ├── switch.py      # manual adapter switching
│   └── config.py
├── adapters/
├── data/
└── requirements.txt

Inference (Gradio)

inference.py — loads the quantized base model, attaches an adapter via PEFT, and serves a Gradio ChatInterface:

  • --adapter flag to pick which adapter to load (defaults to adapters/general_adapter), --share for a public link
  • Chat-style generation: wraps each message as Instruct: ...\nOutput:, strips the prompt back out of the decoded response
  • Adjustable sliders for max new tokens, temperature, top-p — exposed directly in the UI rather than hardcoded

Eval / test scripts (CLI, used during development rather than for serving):

  • test_inference.py — sanity-checks the quantized base model loads and generates, no adapter involved
  • verify_adapter.py — loads base model + general_adapter, runs a fixed set of reasoning/coding/math prompts through it
  • compare_adapter.py — same prompt set, run twice per prompt with enable_adapter_layers() / disable_adapter_layers() toggled, to directly compare base vs. adapted output

Results

  • [Space for before/after output from compare_adapter.py — with vs. without adapter, same prompts]
  • [Space for demo of the trained model responding]

Open Problems / What’s Next

Context loss on adapter switch — swapping adapters mid-conversation loses track of prior context. Planned fix: rag.py — index output into a vector store on switch, inject retrieved context into the next adapter’s input. Designed but not yet implemented.

No automatic adapter routing — switching is manual for now; a classifier to pick the right task adapter automatically is a later step.

Task-specific adapters — only the general adapter has been trained so far; task adapters (coding, prompting, etc.) are planned but no training data or scripts exist yet.

VRAM requirement mismatch — docs state 12GB+ target vs. 6GB actual hardware; needs resolving/clarifying.

RAG for live data — the original stock-market RAG use case is separate from the adapter-switch RAG use case and still pending.

Code & Links

  • Repo: github.com/Ronit9320/LoRA-RAG
  • prepare_dataset.py — builds train/eval JSONL from UltraChat
  • train_general_adapter.py — QLoRA training for the general adapter
  • inference.py — Gradio chat UI, adapter-selectable
  • test_inference.py — base model sanity check
  • verify_adapter.py — adapter output on fixed prompt set
  • compare_adapter.py — base vs. adapter side-by-side comparison
  • AGENTS.md, lora_adapter_pipeline.md — architecture/design docs
  • Related posts: Part 0, Part 1, Part 2 — narrative version of this same journey