Every AIoT project reaches the same fork: run the model on the device, or ship the data somewhere with a bigger GPU. The answer is rarely about model accuracy. It is about four constraints that are easier to measure than to argue about.
1. What does a round trip actually cost you?
Write down the deadline your system has to meet. A door that unlocks on face recognition can absorb a second; a robot arm deciding whether to close a gripper cannot. Then measure the round trip you would need — not the inference time, the round trip, including the network on its worst day. If the worst day exceeds the deadline, the decision is made for you.
2. How much video do you want to pay for, forever?
A single 1080p stream at a modest bitrate is roughly a terabyte a month. Multiply by camera count, then by the number of months the system is meant to run. Edge inference changes the shape of that number completely, because what leaves the device is an event — a few hundred bytes — rather than every frame in which nothing happened.
3. Who is allowed to see the frames?
In a workplace, a school or a residential building, the question of whether images leave the premises is often what decides the project, and it gets decided by someone who is not an engineer. On-device inference lets you answer honestly that no video leaves the building. That is a much easier conversation than explaining a retention policy.
4. What happens when the link drops?
Cloud inference means the feature stops when the connection stops. If your system has to keep working through an outage — and most systems that control something physical do — the model has to be local, even if a copy also runs in the cloud for analytics.
The usual answer is both
In practice most deployments end up hybrid: detection and control on the device, aggregation and long-term analytics in the cloud. Design for that split early. Retrofitting it after you have chosen a compute module that cannot run your model is the expensive path.
Sizing the hardware
Once you know inference runs locally, quantise your model to INT8 and benchmark it before choosing the part. The accuracy loss is usually small and the throughput difference is large — and a measured number beats a datasheet TOPS figure, which is measured on a model that is probably not yours.