Thoughts on DIY Lane detection CNNs for openpilot

One of the most limiting factors of openpilot for developers is the snpe’ified TensorFlow vision model. Qualcomm SNPE is defined as Snapdragon Neural Processing Engine SDK, which is a platform to convert existing TensorFlow, Caffe and maybe some other models to a dlc file container format for acceleration on Qualcomm hardware, most commonly found in mobile phones.

The reason why this is limiting is that running openpilot on different hardware such as x86, Google Coral and Raspberry Pi with TPU is not immediately possible as there isn’t a snpe GPU accelerated runtime available for non-Qualcomm platforms. Newer platforms such as the Snapdragon 855 are benchmarked to have a 5x snpe speedup vs the comparatively ancient 821 in the LEX727 and 3T.

With all this said, there’s really nothing stopping you from running OP on another SNPE supported device but camera interfacing. It’s rumored that getting visiond to work with a phone’s camera is the reason why we still use IMX298 based hardware as camera manufacturers don’t publish specs, drivers and configs.

It’s become more complicated to train your own model now, as comma now performs vision radar, laneless, and coming lane change in their models; which isn’t trivial to develop as the subject matter is not well published online or in academia. You may want to target openpilot 0.5.8.

I’m not going to go into too much detail in training your own models, but I will point you to which is an excellent resource for lane models, complete with papers. Your objective is to input a camera image and output polys, and probabilities. model inputs. If you can push 256×512 pixels in, and get these out, you have a OP compatible lane finding model

Specifically, all you need is a left lane poly, right lane poly, and the probabilities of each. You don’t even really need polys, 50 points each will suffice, as the conversion from points to polys is in code, not from the model.

visiond model inputs and outputs. input tensor size is 256×512 for 0.6.4 model (medmodel)

Here’s a good dataset, or create your own with OP data:

I’m excited to see your results!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s