Epoch 9: Calibration Compiler, Part 1

sydney-moore-FTbxkhiPPKM-unsplash

Intro

Hi, this is chefcinnamon. Here is Epoch 9.

I intend to dip my toes further into ML!

I have been learning ML from scratch, and I decided to learn more.

There are many resources here and there, and the ecosystem is growing rapidly. This can make a learner unsure about their learning path and get lost among so many resources to try and learn.

But I believe: try to begin, and you will find the path sooner or later.

I’m an adventurer! Here we go.

Calibration compiler

My curiosity led me to model compression. Models are expensive to run, so teams shrink them through quantization. An interesting fact about them is that a compressed model can still look fine in ordinary tests while becoming worse at the exact behaviours agents need, such as long instructions, tool picking, argument construction, and more.

Cohere wrote about seeing this in production. Standard, short, generic calibration can be misleading and cause regressions, especially in long-context, tool-calling workflows. Even real traces are not enough, since they can dominate the calibration statistics in a specific area. Source

I found that there is still quite a lot of room to improve this multistep process: quantizers shrink models, tracing tools record agent runs, evaluation tools score behaviours, and calibration corpora provide data for proving whether our shrunken model is performing well.

I want to experiment with a calibration compiler: turning structured agent traces into calibration data and providing evidence for whether compression preserved or damaged the behaviour. We can test the result by checking the scores against generic calibration tools.

The process

  1. Original large model
  2. Calibration corpus
  3. Quantizer performs calibration
  4. Smaller / cheaper model
  5. Evaluation test set checks whether it still works

Definitions

Before we begin in the middle of nowhere, let me explain some definitions:

  1. Calibration corpus: a set of representative agent traces: instructions, tools, requests, tool results, and next actions.
  2. Calibration: the quantizer runs the corpus through the original model and measures its internal numerical activity.
  3. Shrinking a model: usually changing its stored numbers from high precision, such as 16-bit, to 4-bit. Shrinking can make inference cheaper and faster, or it can damage quality.
  4. Quantizer: a tool such as AWQ or GPTQ. It takes a model plus a calibration corpus to produce a compressed model.
../