Object Detection at 1840 FPS with TorchScript, TensorRT and DeepStream

Object Detection at 1840 FPS with TorchScript, TensorRT and DeepStream(paulbridger.com)

162 points by briggers 5 years ago | 55 comments

mozak1111 5 years ago |

I see this and I immediately think of "trash sorting" at ultra high speed. If one can combine this with a bunch of accurate (laser precision) air guns, to shoot and move individual pieces of trash you can sort through a truck load of trash in a matter of seconds, perhaps in the air while they are being dumped! compare this approach with how we are currently doing it [0] - Somebody should get Elon Musk on this project right away!

[0] - https://www.youtube.com/watch?v=QbKA9uNgzYQ

Cerium 5 years ago | |

Sorting by optical recognition and air guns to separate a falling curtain of product into two output streams is already a product. The development of these machines are the reason that 10 or 15 years ago you stopped seeing bad beans in bulk bean bags. I am involved in the tea industry where they are used to sort tea by grade - stems, bad leaves, broken leaves, full leaves.

Here is a diagram: https://www.satake-usa.com/what-is-optical-sorting.html

SequoiaHope 5 years ago | | |

Yes but when I went to the recycling (sorting) facility in San Mateo they remarked that their plastic sorting systems work by infrared reflection and so cannot see black plastic. They said that because of this they are unable to process black plastic entirely. I got the sense that there’s room for improvement.

Bonus slow motion footage of their processing machines: https://imgur.com/gallery/IK5zKkO

handol 5 years ago | | |

Reminds me of the library modernization drive in 'Rainbows End'. The book digitizer is basically a wood chipper with lights and high speed cameras in the debris chute.

ebalit 5 years ago | |

Not related to Elon but there is a company called Pellenc ST in the south of France that work exactly on this kind of problem. You can see a video of one of their machines here [0].

I work at an AI consultancy [1] that help them use deep neural nets in these high throughput and low latency conditions. It's an interesting challenge and the performance than can be squeezed from modern hardware are indeed impressive.

0: https://youtu.be/XLciSGE82DY?t=280

1: https://neovision.fr

MauranKilom 5 years ago | |

This is a thing already. In my understanding, it's a staple in several kinds of recycling processes. Random sample of related links (there's a seemingly infinite amount of these though):

https://youtu.be/mLya2NuY4Yk

https://youtu.be/GJeOfHxMWQo?t=87

https://youtu.be/bWUuBz2hWc0?t=83

tyfon 5 years ago | |

Trash sorting is probably better than self driving cars. I only see speed talk on this page and nothing about accuracy.

Musk needs like 99.9999% accuracy at near zero latency over several hours of operation. I think Tesla currently is at maybe 99.995% from driving my car. The last 0.005% results in phantom braking etc. It's actually a very hard nut to crack and I don't expect them to achieve the full self driving in all conditions for another 10-15 years maybe. The edge cases are just too many.

I like the trash idea though (or a Q/A robot at a factory etc).

cm2187 5 years ago |

Out of curiosity, what are the possible use cases for object detection at >100 fps? I assume it would have to be objects that move very fast, i.e. nothing ordinary that I can think of.

[edit] actually stupid question. I assume it's more about throughput than fps, i.e. be able to process lots of streams on the same machine, for instance for doing mass analysis of CCTV streams.

janimo 5 years ago |

How portable are these techniques to other architectures? Could >100 FPS be realistically achieved today using only CPUs or mobile phones?

nl 5 years ago | |

> Could >100 FPS be realistically achieved today using only CPUs or mobile phones?

Not yet.

Google's MediaPipe object detector (which is one of the most optimised mobile solutions around) can do "26fps on an Adreno 650 mobile GPU"[1].

The Adreno 650 is the GPU in the Snapdragon 865, ie the current high end SOC used by most non-Apple phones. This gives roughly the same performance as an iPhone 11.

[1] https://google.github.io/mediapipe/solutions/objectron.html

[2] https://www.tweaktown.com/news/69097/qualcomm-adreno-650-gpu...

[3] https://www.tomsguide.com/news/snapdragon-865-benchmarks

janimo 5 years ago | | |

Thanks for the links. I think there also isn't an API for accessing high FPS cameras on Android devices that support slow motion video capture.

motoboi 5 years ago | |

Converting to ONNX gives you advantage in Intel CPU too, if you convert from ONNX to OpenVINO.

briggers 5 years ago | |

Mobile phones definitely since these days most of them have pretty powerful GPUs.

gcanyon 5 years ago |

A weird question, but since there's another article on HN right now about programming language energy efficiency https://news.ycombinator.com/item?id=24816733 any idea whether going from 9fps to 1840fps consumes the same power, 200x the power, or somewhere in between?

briggers 5 years ago | |

Great question, now I wish I'd recorded power consumption for all these experiments. Judging from cumulative hours of watching the output of nvidia-smi I've definitely seen a linearish relationship between utilization and power draw (with a non-zero floor of 30-40W).

eggy 5 years ago | |

I see Rust is almost equal to C if not better in the graph, however, I think equally-skilled programmers in either language would show the Rust programmer spending more 'energy' programming and iterating than the C programmer, but then make the argument that the C program will use more 'energy' downstream if bugs slip in. In any case, an eye-opening metric on what I, and I am sure many, take for granted. Cool.

eggy 5 years ago | | |

EDIT: I think Zig would come up pretty good here too:

https://twitter.com/andy_kelley/status/1317586767260774400

Grimm1 5 years ago |

Good work getting TensorRT running we had a real pain in the butt recently when working with it and just opted to go with ONNXRuntime, their graph optimizer and their TensorRT backend -- may not be as fast as straight TensorRT from comparisons I've seen but it got us to a competitive inference and latency so we're happy with it.

briggers 5 years ago | |

Nice one! I've long been interested in the ONNX serving path.

moron4hire 5 years ago |

Any word on latency? I didn't see anything in the article. I guess, since this is a synthetic test just pumping a single image file through repeatedly instead of an actual video stream, then it wouldn't realistically be measurable. But if latency is particularly low, this would be a boon for AR systems.

briggers 5 years ago | |

BTW, this is pumping the same video file through the network - not just a single file. I don't measure latency, but this is not a deep pipeline so it's easy to calculate.

moron4hire 5 years ago | | |

Ok, I guess I misread that part.

liuliu 5 years ago | |

Latency fundamentally limited by the model processing single frame, all-in-all, probably somewhere around 10 to 15ms depends on your input size (assuming VGA type of input). This is a great article talks about system-engineering for the vision pipeline, but to solve the latency issue, you need either a beefier processor (or more specialized processor) or a better tuned algorithm.

stabbles 5 years ago |

> There is evidence (measured using gil_load) that we were throttled by a fundamental Python limitation with multiple threads fighting over the Global Interpreter Lock (GIL).

Can anyone comment on how often this is a problem and if this problem is truly fundamental to Python? Could it be solved in a Python 3.x release?

shepardrtc 5 years ago | |

Yes, this is a fundamental part of Python. By default, a single Python process is single-threaded in the traditional sense. So, using "threads" (i.e. the Threading module) in Python is actually more like using fibers in some other language. They're not OS threads. So, if you're not waiting on I/O, then yes, the threads will fight over the GIL and performance will suffer. This is inherent to Python and will not be changed.

But there's a few more things that can be said about this. Python "threads" are really just a mental construct for designing programs. The selling point is that you can share variables and data between "threads" without having to worry about locks or data corruption or anything like that. It just works. But, even with that advantage, you're relying on Python to switch between "threads" on its own, and that could easily slow things down. If you're willing to drop the mental construct and go for better performance but still use a single process and be able to share variables, the asyncio module will let you control when the main Python process will move between points of code flow.

However, if you really want to use traditional multiple processes/threads just use the Multiprocessing module. It actually launches multiple Python processes and links them together. It's called in a similar fashion to Threading, so there isn't much code change for that part. But because it's no longer a single process - and no longer bound by the GIL - you can't share data between the processes as easily. With Multiprocessing, you'll need to create slightly more complex data structures (like a multiprocessing manager namespace) to share that data. It's not that hard, but it requires a bit of planning ahead of time.

the-dude 5 years ago | |

The subject has been debated to death. Google it.

indeyets 5 years ago |

Name clash again… I thought about https://deepstream.io/