Efficient Deep Learning Deployment
Fitting CNNs into fixed memory and latency budgets without losing the accuracy that matters.
- Research selects models by accuracy; deployment selects them by accuracy subject to fixed memory and latency budgets. Most applied vision projects fail in that gap.
- Combined hardware-aware architecture search, structured pruning, quantization and distillation, measuring the accuracy cost of each step on the target device rather than estimating it.
- Core theme of the CIFRE PhD, transferred into industrial use. Quantitative results belong to that context and are not reproduced here.
Introduction
This is the core engineering theme of my doctoral work: taking convolutional vision models that perform well without constraints, and making them run within the memory, latency and compute budget of a target system without an unacceptable loss of accuracy.
The work was carried out in an industrial research setting, in a CIFRE arrangement between SEGULA Technologies and the CNRS Hubert Curien Laboratory.
Context and problem
Model selection in research optimises accuracy. Model selection in deployment optimises accuracy subject to hard constraints that are set by hardware, not by the modeller.
The gap between those two activities is where most applied computer vision projects fail. A model that satisfies the accuracy requirement and violates the latency requirement has not solved the problem.
- Target hardware sets fixed memory and latency budgets
- Accuracy loss from compression is task-dependent and not predictable in advance
- Compression techniques interact; applying them independently is not additive
- Deployment runtime affects which optimizations are actually realisable
Objectives
- Reduce model size and inference cost under explicit hardware constraints
- Quantify the accuracy cost of each optimization rather than assuming it
- Combine neural architecture search, pruning, quantization and distillation coherently
- Make the accuracy-versus-latency trade-off an explicit, inspectable decision
- Transfer the resulting methods into industrial use
Constraints
Hardware
- Fixed memory ceiling on the target system
- Latency budget defined by the application, not negotiable
- Limited or absent floating-point acceleration on some targets
Method
- Optimizations must survive export to the deployment runtime
- Results must be reproducible to be publishable
- Comparisons must hold hardware and runtime constant to mean anything
Architecture
Baseline Model
Unconstrained reference and accuracy ceiling
Hardware Constraint Definition
Memory, latency and compute budget of the target
Architecture Search
Candidate architectures within the constraint envelope
Compression
Pruning, quantization, knowledge distillation
Export and Runtime Validation
Verify the optimization survives export
On-target Measurement
Latency, memory and accuracy on real hardware
Data and model pipeline
Search
- Neural architecture search constrained by the hardware budget rather than by parameter count alone
- Candidate evaluation on proxy tasks before full training
Compression
- Structured pruning, chosen over unstructured where the runtime cannot exploit sparsity
- Post-training and quantization-aware approaches compared per task
- Knowledge distillation from the unconstrained baseline into the constrained student
Validation
- Accuracy measured after export, not before
- Latency and memory measured on the target device, never estimated from FLOPs
Optimization
- Hardware-aware architecture search rather than proxy metrics such as parameter count
- Structured pruning matched to what the deployment runtime can actually accelerate
- Quantization applied with per-layer sensitivity analysis instead of uniformly
- Distillation used to recover accuracy lost to compression
- Techniques composed in a fixed order, with the cost of each step measured separately
Deployment
- Export to deployment runtimes with verification that optimizations are preserved
- Measurement on the target device under representative input conditions
- Industrial technology transfer of the resulting methods
Reliability
- Accuracy verified after every transformation, so regressions are attributable to a specific step
- Fixed evaluation protocol across all candidates
- Export validation to catch operations unsupported by the target runtime early
- Fallback to a less aggressive configuration when a target constraint cannot be met without unacceptable accuracy loss
Benchmark methodology
FLOPs and parameter counts are poor predictors of on-device latency; they were used only as coarse filters. Every reported comparison is intended to be measured on the target hardware with the deployment runtime, holding input resolution and evaluation set constant.
- Fixed evaluation set across all model variants
- Latency measured on-device after warm-up, reported as a distribution
- Peak memory recorded on-target
- Each optimization step measured independently as well as in combination
Results
Quantitative results belong to the associated doctoral research and its industrial context. They are not reproduced here.
Benchmarking in progress. Results will be published with the hardware, model, runtime and measurement conditions that produced them.
Trade-offs
Aggressive compression
- Gains: meets tight hardware budgets
- Costs: accuracy loss that grows non-linearly past a task-dependent point
Architecture search
- Gains: finds architectures a human would not select under the constraint
- Costs: substantial search compute before any deployable result exists
Quantization-aware training
- Gains: better accuracy retention than post-training quantization
- Costs: requires retraining, so it is not applicable to every model
Lessons learned
- Optimize against measurements on the target device. Proxy metrics mislead often enough to be actively harmful.
- Compression techniques interact; the order in which they are applied changes the outcome.
- Distillation is most valuable as a recovery step after compression, not as a standalone technique.
- Defining the hardware constraint precisely at the start is more valuable than any individual optimization technique.
- An optimization that does not survive export to the deployment runtime does not exist.
Future improvements
- Extension to transformer-based vision architectures
- Automated sensitivity analysis to guide per-layer compression
- Tighter integration between search and the deployment runtime's actual capabilities
Technology stack
Training and optimization
- PyTorch
- TensorFlow
- Keras
Techniques
- Neural architecture search
- Structured pruning
- Quantization
- Knowledge distillation
Deployment
- ONNX Runtime
- Embedded targets
PyTorch · TensorFlow · Quantization · Pruning · NAS · Knowledge Distillation · Embedded AI