Traversing the ML World Without a Map — Part 2

It has been some time since I wrote a follow-up on this project — it has changed quite a few times now. The first idea was to train a small model to perform certain tasks using LoRA and retrieve data using RAG. As it went on I discovered a lot of new things and tried to implement them into this project.

After training the model (Qwen) for the first time, I realized that getting good enough output from a model self-hosted on consumer hardware is harder than it sounds — in my case a gaming laptop with 6GB VRAM. The outputs were not coherent enough and the model was not behaving the way I wanted it to. This led me to reading more about the techniques I was using, specifically LoRA. I have explained this in the last two parts, but essentially it gives you the power to train adapters for a model. Then I got to know that you can have more than one adapter attached to the model at inference time to change its behaviour — which was something that could make this whole thing possible. There are Hugging Face libraries that handle this, so no new configuration needed.

Around this time I also got to know about Groq — a cloud inference API that lets you run models remotely and gives you access to a lot of flagship models. I tested it and it worked well, but this was not the original plan, which was to learn more about LLMs and run and train everything locally. So I came back.

I have also changed the model and the training technique. The new model is Phi 2.7B — I switched because I was not getting the desired output from Qwen, and Phi is a stronger model with more parameters, so why not. The technique has also moved from LoRA to QLoRA, a quantized version of LoRA that keeps memory usage manageable on a 6GB GPU. For inference we are now using Gradio, which gives a much better web interface where you can tune things to your preference. More details on all of this are on the project page.

Now for the training. One adapter is trained on UltraChat examples — not all, but 8000 of them. This was more of a general-purpose adapter and also a test to see if my laptop could handle training at all — it can, but the results were not great. More training data is needed and it needs to be concentrated on one specific thing. Then I found a pre-trained instruct adapter on Hugging Face which works noticeably better — yes, it does some weird things sometimes, but it is a much better starting point than training from scratch with limited data. The takeaway here is that finding a good base adapter and fine-tuning on top of it is a more practical approach than training a general one yourself, at least with the hardware I have.

The main plan now is to concentrate training to one task per adapter — coding, prompting, and so on — alongside one instruct adapter that stays permanently attached to the model. The second adapter would be swappable on the fly, and I am thinking about having more than two in the mix eventually. The open problem right now is context loss — when you switch adapters, the model loses track of what was said before. The plan is to index the conversation into a RAG pipeline and feed it back as context when needed. This has not been tested yet but it is the next thing on the list.

One last thing — when I started this project I did not know anything about how LLMs actually work. I just assumed what most people assume: that it is AI in broad terms, something almost magical. It is quite different. At the very core, a model is a next-token predictor — it predicts the most likely continuation of a sequence, where words or parts of words are broken into tokens. The fact that this produces something that feels like reasoning or understanding is what makes it fascinating. There is a separate post on the history of how we got here if you want to go deeper into that.

I’ll end this here — more things are being implemented and the next part will have a lot more to show.

Part 3 coming soon.