The Camera Is the Lidar(medium.com) |
The Camera Is the Lidar(medium.com) |
There are about a dozen companies in this space now. Nobody has the price down yet. Continental, the European auto parts company, is the most likely winner. Quanergy made a lot of noise but didn't ship much.[2] There's a conference on automotive LIDAR this month in Detroit.[3] Many of the exhibitors are major semiconductor packaging companies, with various approaches to putting lots of little LIDAR units in a convenient package at a reasonable price.
[1] https://www.ouster.io/faq/ [2] https://news.ycombinator.com/item?id=17755183 [3] http://www.automotivelidar.com/
Flash lidar seems to be a fundamentally broken concept to me wrt range.
There's a tradeoff between field of view and range. Automotive systems will probably include a long-range narrow field of view unit and a shorter range wide field of view unit.
Flash LIDAR has some advantages. No moving parts. Can be fabbed by semiconductor processes. The one big laser is separate from the sensor array, which helps with cooling. Also, you can spread the outgoing beam, which helps with eye safety. (Eye safety involves how much energy is in an eye iris sized, 1/4" or so, cross section of the beam. If the beam is spread out, energy density is lower.)
Because you're collecting NIR ambient light, your optics are wideband. Meaning that daylight would have a more pronounced negative effect on system range (easier to saturate the photocells). It's also low resolution (as most LIDARs are), and there is no color segmentation data.
In an automotive application, I can't see a justification to unify both visual and LIDAR into a single sensor, rather than having an extrinsically calibrated array of sensors. You can improve the calibration out of the data over time if you're very concerned about system stability.
It seems like a nice party trick, but the vehicle LIDAR game focuses on solid state long range units, as this will be what gets into mass production. The visual band imagers in the car are a given for many other reasons anyway.
> We are not sacrificing lidar performance by adding ambient imaging functionality. The lidar subsystem has a short integration time that avoids saturation, and if anything our approach outperforms other lidars.
> As proof, the example videos linked to in the article show raw unedited point cloud data with the lidar operating in extremely sunny environments with plenty of specular reflectors. You can see lens flare in the ambient imagery as any camera would exhibit, but the lidar signal and range data are unaffected. In addition, if you point a velodyne directly at the sun its false positive rate increases significantly while our sensor's FPR does not. No lidar will return the distance to the sun so the only thing that matters is FPR in this scenario.
> We've independently verified the OS-1's range performance with customers under all levels of solar exposure and I guarantee you can't get a smaller, cheaper lidar with even close to this combination of resolution and performance. If you have any doubts, download the raw pcap files from our github page and play them back yourself. We stand behind our data, our pricing, and our spec!
Second, even if you do plan on adding extra cameras, the extrinsic calibration between camera and lidar may become easier if you have good quality ambient light measurement from the lidar. For example Jesse Levinson, cofounder of Zoox, computes extrinsic calibration between camera and Velodyne lidar by assuming that depth discontinuities are correlated with visual features [1]. But obviously the correlation between 850 nm images and visible light images would be way better.
[0] https://www.reddit.com/r/SelfDrivingCars/comments/9c60pe/the...
They are probably wider band than would be required to read only the sensor self illumination.
>> Second, even if you do plan on adding extra cameras, the extrinsic calibration between camera and lidar may become easier if you have good quality ambient light measurement from the lidar. For example Jesse Levinson, cofounder of Zoox, computes extrinsic calibration between camera and Velodyne lidar by assuming that depth discontinuities are correlated with visual features [1]. But obviously the correlation between 850 nm images and visible light images would be way better.
I agree with that - but you could probably go the other way around and coorelate the LIDAR depth map with depth obtained through stereo imaging. The temporal synchronization between NIR and depth provided by this unit is nice though.
Let me phrase this differently - while the videos are cool to watch, I don't think calibration is the problem in vehicles, and nor are baseline artifacts between sensors when operating at such far ranges (whether your camera and LIDAR are perfectly aligned or translated 10cm apart, it won't matter much looking 10m down the road).
Having moving parts, however, won't get this system into a production model.
Solid state lidar has issues. The cofounder of Ouster, Angus Pacala previously cofounded Quanergy, a solid state lidar startup.
There was an announcement on a cooperation between BMW and Innoviz (an Israeli maker of solid state LIDARs) with Magna being their OEM sponsor.
I'm not sure calibration is that big of a deal for this application. Sensors are going to be calibrated and tested in the factory or at a module level regardless, and the accuracy requirements in automotive are much lower than consumer products using similar technology.
You can't overcome not having colors (traffic lights, anyone?), limited ranging distance or sensor saturation due to ambient conditions.
here's an old comparison of algorithms. I imagine the state of the art has improved with Deep Neural Nets recently.
http://vision.middlebury.edu/stereo/eval3/
edit: surprise! the page appears to be kept up to-date with new algorithms and recent techniques, and indeed the top performer is from 2018.
What does this bring that's new ?
On the other hand if you fuse lidar and camera data such as with Waymo and others, there may be issues with the sensors being out of sync (as they run at different framerates, and the lidar continually spins) or physically offset (leading to parallax issues). Dealing with such issues is very difficult. Having a single sensor output both accurate range information and camera data makes it much nicer to work with.
So right now, with very few LIDARs produced, we have a high price, which will start dropping as more are produced.
You might find this interesting: a single transistor used to sell for roughly the equivalent of $8 USD in today's money; today the cheapest ones are 6 Cents USD (price checked today from Mouser.com) in qty 1 pricing...
https://spectrum.ieee.org/tech-talk/semiconductors/devices/h...
However, none of those matches the capabilities of the Ouster OS-1 exactly.
The Ouster OS-1 in the article, as well as all other automotive lidars that I know of, are class 1 laser eye-safe, meaning that it is safe even if you put your eye right up to it for hours.
The power also decreases dramatically once you get far away from it, since the laser beams spend most of their time pointed in different directions, and the collimation is not perfect.
Think of the 1920s-1950s version of robots, for example. They were machines shaped like people and that acted like people. In retrospect, they seem not scary but silly. The human shape isn't particularly useful or easy to build; our most common robots are vacuums shaped like hockey pucks.
Skynet is another "what if machines acted like people" fairy tale. It makes sense if you imagine yourself as a computer that wakes up; we wake up all the time, so it seems normal to us. But self-awareness and self-preservation are biological systems that evolved over very long time scales. Those are intricate systems, again not really useful or easy to build. And also not likely to randomly occur.
It could be that we'll build those kinds of systems, of course. But I think it will take a long time to get them right, and then it's not really the skynet story, it's the mad scientist with the robot army story.
It's technology that already exists, but must be reinvented in a non-military context from scratch, since the tech transfer between weapons systems and civilian applications is likely locked up in policy. So, we know that this technology exists, and is proven, but we have to reinvent the wheel, because reasons.
The reason we see this interminable slow motion public struggle to bring it to consumer applications, is likely because there are no controls in place that can actually prevent "contemporaneous discovery" wink, wink.
That level of testing is well beyond what today's software and hardware is capable of. Waymo has to override their cars (disengagement) approximately every 6000 miles [1] which equates to about 200 hours of driving. To reach a confidence level of 1 in a million hours you would need a test fleet of a thousand vehicles operating for a whole year without any incident occurring that requires human intervention. The costs for such testing would run into the hundreds of millions of dollars which makes me feel like only the largest corporations in the world could develop this technology.
[1] https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disen...
We don't know how humans work and under what conditions they break down, either.
In constrast to the whole self driving stuff this DL is popular for: User input overrides DL input.
Passive self-driving systems that take over when the human gets distracted/unwell are great because human vision exceeds computers where as computers are always alert. This would capture the case you describe, I think it would also have a massive improvement for when bus/lorry drivers should collapse at the wheel (Elon Musk used this as a valid use case for Tesla auto-pilot in the Tesla Semi unveiling).
However active self-driving systems (e.g. Tesla's auto-pilot) are currently worse because they rely on computer vision and humans to be always alert.
By the way, although the article focuses on deep learning, there are many applications that don't involve deep learning. For example, although you can run the deep neural network based SuperPoint on the intensity data, you can also run any classical feature extraction algorithm such as SIFT, SURF, ORB, BRISK, FAST, AGAST, etc. Doing so provides an elegant solution to the problem of localizing in a geometrically sparse but visually rich environment, such as a smooth but well-illuminated tunnel.
And with the amount of drivers snapchatting behind the wheel, I'd rather take my chances with a self-driving car.
I’d really like to see a demonstration that human behavior is just neural nets. Sadly, I think that’s still an open question.
What would you rather use to interpret sensor data?
A friend of mine was doing that in a new car with some pedestrian detection system that decided to detect the barrier as a human and slam the brakes to the complete stop. From what I've heard it was not exactly pleasant.
Image orthicons for TV cameras once cost $10,000 each, and color cameras needed three of them, plus another $50K or so of electronics to drive them. Today a cell phone camera costs about $10.
And no, it's my understanding that it's just a matter of volume.
Even if we don't understand deep learning to the degree that we would like, we can observe its safety record and compare it to humans'.
That's AU$0.000,000,062,395,833 (US$0.000,000,044,791,667) per transistor;
- FLIR thermal camera
- 3 different small cameras manually set at different settings, models chosen for their qualities handling light levels.
Those 4 live feeds were fed into a small black magic design quad layout device, that turned them into a single hd feed via hardware/real-time. That was fed into a hardware capture, that stacked the quad arrangement, applied some other filters and did hardware compression. At that point almost no latency was introduced but had a nice working base video feed. That was fed into the Linux box for processing.
The quad device created a sort of super hdr video, and the thermal layer took it to the next level. All of the cameras had drawbacks, but combined they were minimized.
Late and a bit rough but here is my idea:
Install redundant self driving units in at least a good number number of the first few thousand cars in each generation.
When planning a release, push to the redundant unit in the cars already running in "production".
Use only primary unit as input to car as usual, but log the diffs between the new version and old version in the same way they now log driver intervention.
I think there is a number of issues this won't catch, off the top of my head what if the new self driving unit attempts to turn slightly faster on slippery road etc.
But it should be able to collect up realistic feedback really fast i.e. in a few months (crazy slow for modern application developers like me but more than fast enough for anything that should be allowed to drive unsupervised I guess :-)
Definitely not : )
> but is there a name for that style of testing and do their exist categories of systems for which it is commonly used
I don't know but I guess parallell deployment, production testing (hehe) or something.
I guess I learned about it here on HN but I've heard similar approaches elsewhere.
Three concrete systems I can think of that has been tested in similar but not identical ways:
- trajectory calculation systems for some space probe (I forgot the details) where two separate vendors where tasked with writing software and their software where then run in parallel in a simulation of possible trajectories to root out any bugs. Mentioned as an example to point out an extreme variant of testing, probably by someone here on HN or in something linked to from HN.
- a vendor running testing of bag sorting equipment on an airport. Probably told me from a colleague of mine at the time who knew them. AFAIK they'd rerun batches and verify the outputs, making sure the new system produced similar or better results.
- an API testing service, probably mentioned here on HN as well, tjat worked by firing three similar requests for each resource and method: two to a deployment of the old system and one to the new. The two that were fired against the old would be used to find parts of the response that changed from request to request (time stamps etc) and the the rest of the response would be used as a template to verify the response from the new system.
more generally capturing and reusing production input or running two sets of services in parallel in a data center, - or across datacenters: the existing system as usual and the system under test with only input data.
Of course they might cheat, but who really knows if their car has an airbag where it is supposed to.
Somehow I doubt the authorities have the foresight.
Velodyne's 64-channel is 60 W