CONTEXT
Why this comes up
UT describes nine months of work evaluating and operating AI in a Box, moving through Ollama, vLLM and NVIDIA NIM. Customer feedback and hardware and software integration problems informed the work. Exact transition dates, deployment manifests and comparable measurements are not attached to this record.
THE USEFUL LESSON
Document each runtime decision against the application requirement, machine and version that motivated it. A serving stack is part of an operating system of applications, dependencies and recovery procedures.
ON YOUR MACHINE
What to record and test
- 01
Capture the machine, driver, OS, model revision, precision and serving version for each evaluated configuration.
- 02
Record the concrete reason for each change: compatibility, application features, performance, reliability or maintenance.
- 03
Separate model-loading issues from application integration and network dependencies.
- 04
Compare configurations on the same task-quality and latency requirements, with the request mix disclosed.
- 05
Publish a selected reproducible lesson, while keeping proprietary automation and customer-specific assets outside its scope.