← Blog · 2026-11-05 · IoT / AIoT · ~12 min read

Int8 quantization for edge AI: from the formula to measured error

How an 8-bit integer grid replaces 32-bit floats in a neural network, the arithmetic that runs on the device, and what it costs in accuracy, measured on 12 independent runs of a small vibration-monitoring classifier.

1. Context

A model that fits on a microcontroller has two budgets to respect: flash/RAM for the weights and energy per multiply-accumulate. Storing each weight in 8 bits instead of 32 cuts the weight memory by about four, and integer multiply-accumulate units are cheaper than floating-point ones; Jacob et al. (2018) describe a scheme in which both weights and activations are 8-bit integers and inference uses integer arithmetic only. Microcontrollers announced this month with on-chip neural accelerators, such as the STM32H7P with its Neural-ART NPU, are aimed at running models on the device. This article derives the quantizer, shows the integer arithmetic of one dense layer, and measures how much accuracy a post-training int8 conversion costs on a condition-monitoring task, including what happens when the grid is made much coarser.

2. The affine quantizer

A real value \(x\) in a range \([\alpha,\beta]\) is mapped to an integer \(q\) with \(b\) bits, scale \(s\) and zero-point \(z\):

With \(\alpha\le0\le\beta\) the value 0 is represented exactly (\(q=z\)), which matters because ReLU outputs and zero padding are exact zeros. For unsigned activations \(q\in[0,255]\). For weights I use the symmetric variant, \(z=0\), \(q\in[-(2^{b-1}-1),\,2^{b-1}-1]\) and \(s=\max|w|/(2^{b-1}-1)\); for \(b=8\) that is \([-127,127]\).

Error. Inside the range, rounding moves \(x\) by at most half a step, \(|x-\hat x|\le s/2\). If the rounding error is roughly uniform on \([-s/2,s/2]\) its variance is \(s^2/12\). For a signal that is itself uniform over the whole range \(R=\beta-\alpha\), the variance is \(R^2/12\) and, with \(s\approx R/2^{b}\), the signal-to-quantization-noise ratio is

So each extra bit buys about 6 dB. Real weights are bell-shaped and the range is set by the largest value, so most weights use only a small part of the grid and the measured SQNR sits below \(6.02\,b\) (Section 7).

3. Integer arithmetic of one dense layer

For output \(i\) of a dense layer, \(y_i=\sum_j W_{ij}x_j+b_i\). Substitute \(W_{ij}=s_{w,i}\,q^w_{ij}\) (symmetric, one scale per output row, or the same scale for all rows) and \(x_j=s_x\,(q^x_j-z_x)\):

The bracket is an integer computed with 8-bit operands and a 32-bit accumulator. Each term is at most \(127\cdot255=32{,}385\) in magnitude, so a 32-bit signed accumulator cannot overflow for a dot product with fewer than \(2^{31}/32{,}385\approx66{,}000\) terms; the layers here have 64, 32 and 16 inputs. The next layer's input is obtained by multiplying the accumulator by \(M=s_{w,i}s_x/s_y\) and adding the output zero-point; real devices store \(M\) as a fixed-point integer multiplier and a shift (Jacob et al., 2018). In this article's simulation the product by \(M\) is done in floating point, everything before it in integers.

Granularity. Per-tensor quantization uses one scale \(s_w\) for the whole matrix; per-channel uses one scale per output row, so a row with small weights is not forced onto the grid of the row with the largest weight. Krishnamoorthi (2018) recommends per-channel weights with per-layer activations as the preferred scheme for hardware. Activation ranges \([\alpha,\beta]\) come from calibration: run a few hundred representative inputs through the float model and record each layer's minimum and maximum.

4. Implementation

The three core functions in NumPy. The complete script (data generator, training, all three experiments and both figures) is downloadable.

def quant_weights(W, bits, per_channel):
    """Symmetric signed quantization, zero-point 0. Returns integers and the scale(s), shape (out,)."""
    qmax = 2 ** (bits - 1) - 1
    amax = np.abs(W).max(1) if per_channel else np.full(W.shape[0], np.abs(W).max())
    s = np.maximum(amax, 1e-12) / qmax
    q = np.clip(np.round(W / s[:, None]), -qmax, qmax).astype(np.int32)
    return q, s

def act_params(lo, hi, bits=8):
    """Asymmetric unsigned quantization of a range that always contains 0."""
    lo, hi = min(lo, 0.0), max(hi, 0.0)
    s = max(hi - lo, 1e-12) / (2 ** bits - 1)
    z = int(np.clip(round(-lo / s), 0, 2 ** bits - 1))
    return s, z

def quant_forward(P, X, wbits=8, per_channel=True, abits=8, cal=None):
    cal = cal or calibrate(P, X, abits)
    qmax_a = 2 ** abits - 1
    s_x, z_x = cal[0]
    xq = np.clip(np.round(X / s_x) + z_x, 0, qmax_a).astype(np.int32)
    for i, (W, b) in enumerate(P):
        wq, s_w = quant_weights(W, wbits, per_channel)
        bq = np.round(b / (s_w * s_x)).astype(np.int64)                 # int32 bias, scale s_w*s_x
        acc = (xq - z_x).astype(np.int64) @ wq.T.astype(np.int64) + bq  # integer accumulation
        y = acc * (s_w * s_x)
        if i == len(P) - 1:
            return y
        y = np.maximum(y, 0)
        s_x, z_x = cal[i + 1]
        xq = np.clip(np.round(y / s_x) + z_x, 0, qmax_a).astype(np.int32)

Sanity check

Quantizing 100,000 values drawn uniformly from \([-1.3,2.1]\) to 8 bits gives \(s=0.013333\), \(z=97\) and a maximum round-trip error of 0.006667, equal to \(s/2\) to the printed digits: the bound of Section 2 is attained, not just respected.

5. Experiment setup

Task (synthetic, fully reproducible). Windows of 128 samples at 1,024 Hz from a rotating machine, three classes: normal (a sine at a rotation frequency drawn from 28–32 Hz), imbalance (the same plus a second harmonic with amplitude 0.2–0.6), and bearing fault (the same plus decaying 300 Hz impulses repeating at 3.2–3.8 times the rotation rate, amplitude 0.5–1.5). Gaussian noise with standard deviation 0.3 is added and the phase is random. The features are \(\log(1+|\mathrm{FFT}|)\) of the Hann-windowed window, the first 64 bins, standardised with training statistics.

Model. MLP 64→32→16→3 with ReLU (2,659 parameters), trained with Adam (learning rate 0.003, batch 128, 40 epochs) in NumPy. For each of 12 seeds (0–11) a new training set and a new test set of 3,000 windows each, and a new initialisation, are generated. The first 256 training windows calibrate the activation ranges. All reported accuracies are on the 3,000 test windows. The floating-point reference is NumPy float64; memory sizes below count float32 storage.

6. Experiment 1: int8 against floating point

• int8 is practically lossless here. Per seed, the accuracy drop against floating point is at most 0.13 points (per-tensor) and 0.07 points (per-channel); the mean drop is −0.011 and 0.000 points, i.e. no measurable change. Predictions agree with the float model on 99.84–99.87 % of the test windows.
• Per-channel is slightly more accurate numerically (mean relative logit error 0.0080 against 0.0092, 13 % lower), but at 8 bits the accuracy difference is far below the seed-to-seed spread (0.28–0.31 points).
• Size: 2,659 parameters take 10,636 bytes as float32; int8 weights, int32 biases and one float32 scale per output channel take 3,016 bytes, 3.53 times smaller (not exactly 4, because biases and scales stay wide). Per-layer \(\max|W|\) is 1.03, 1.07 and 1.06; in layer 1 the largest row maximum is 2.5 times the smallest.

7. Experiment 2 and 3: making the grid coarser

Same trained networks and same calibration, weights quantized to fewer bits, activations kept at 8 bits.

The SQNR columns are measured on the weights of the seed-0 network (all layers pooled); the accuracy columns are mean ± std over the 12 seeds.

• About 6 dB per bit, with an offset. Per-tensor SQNR falls from 40.0 dB at 8 bits to 27.7 dB at 6 bits and 14.9 dB at 4 bits, about 6.2–6.4 dB per bit, but it lies 8.2 dB below \(6.02\,b\) at 8 bits, because the grid is sized for the largest weight. Per-channel quantization recovers 4.1 dB at 8 bits.
• Accuracy is flat down to 5 bits and loses less than half a point at 4 bits (93.35 and 93.56 % against 93.83 %) for this small task, without any retraining.
• Granularity matters once the grid is coarse. At 3 bits per-channel keeps 92.68 % against 91.36 % per-tensor; at 2 bits 82.67 ± 4.84 % against 69.58 ± 12.36 %, with a much larger spread across seeds for per-tensor.

Limitations. The data are synthetic and the task is easy to separate in the frequency domain; the network is tiny (2,659 parameters, dense layers only). Convolutions with batch normalisation and depthwise layers have much larger differences in weight range between channels, which is where Krishnamoorthi (2018) found per-channel scales to matter most; this experiment does not cover them. Only weights were swept; activations stayed at 8 bits with min/max calibration, which is sensitive to outliers in real data. The requantization multiplier is applied in floating point, not as a fixed-point multiply-shift, and no latency or energy was measured: this is a numerical simulation of integer arithmetic, not a benchmark on a device. Whether per-channel scales are available depends on the runtime and the accelerator. Quantization-aware training, which Jacob et al. (2018) pair with the integer scheme, was not tried.

8. Practical takeaways

1. Start with post-training int8: per-channel symmetric weights, per-tensor asymmetric activations, ranges from a few hundred representative inputs. On this task it changed accuracy by less than 0.15 points on every run.
2. Calibrate on data that looks like deployment data. The activation ranges decide the step size of every layer.
3. Compare against the float model on the same test set and count agreement, not only accuracy: agreement (99.8 % here) is a sensitive check that nothing broke.
4. Expect a loss of 6 dB of weight SQNR per bit removed; go below 8 bits only after measuring on your own task, and prefer per-channel scales when you do.
5. Check the memory arithmetic including biases and scales (3.53× here, not 4×), and check that your runtime supports the granularity you chose.

References

1. B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam and D. Kalenichenko. “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 2704–2713. doi:10.1109/CVPR.2018.00286.
2. R. Krishnamoorthi. “Quantizing deep convolutional networks for efficient inference: A whitepaper.” arXiv:1806.08342, June 2018. arxiv.org/abs/1806.08342. (Per-channel weights with per-layer activations; post-training 8-bit results; quantization-aware training.)
3. P. Warden and D. Situnayake. TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers. O’Reilly Media, 2020. ISBN 978-1-4920-5204-3.

Code: int8_quantization.py (Python 3.12, NumPy, Matplotlib; about 80 seconds for 12 seeds). All numbers in Sections 4–7 are the output of this script.