ONNX inference: comparing CPU and GPU
Published on March 11, 2026, the article compares ONNX Runtime inference on an NVIDIA L4 GPU and an AMD EPYC 9965 CPU. The author reports that OCaml and memory placement with libnuma made CPU inference faster than using scarce GPUs at TESSERA.
Source: Comparação de desempenho de CPU e GPU em inferência ONNX (tunbury.org). Text prepared with AI from this source.
What happened and what to do
Published on March 11, 2026, the article compares ONNX Runtime inference on an NVIDIA L4 GPU and an AMD EPYC 9965 CPU. The author reports that using OCaml and memory placement with libnuma made CPU inference faster than using scarce GPUs at TESSERA.
A company assessing inference infrastructure can measure latency, throughput, and resource consumption on representative workloads, comparing CPU and GPU while testing memory configurations. A benchmarking and monitoring pipeline can record these results and inform capacity and model-placement decisions.
How the consultancy can help
Wendelmaques can assess the inference workload and available infrastructure, define a scope for comparative tests, and implement a benchmarking and monitoring pipeline, with support for its operation.
Next step
Send a short description of your inference workload and current infrastructure to receive a scoped proposal.
Let's assess your project
I reply with scope and a proposal.