One of the most limiting factors of openpilot for developers is the snpe’ified TensorFlow vision model. Qualcomm SNPE is defined as Snapdragon Neural Processing Engine SDK, which is a platform to convert existing TensorFlow, Caffe and maybe some other models to a dlc file container format for acceleration on Qualcomm hardware, most commonly found in mobile phones.
The reason why this is limiting is that running openpilot on different hardware such as x86, Google Coral and Raspberry Pi with TPU is not immediately possible as there isn’t a snpe GPU accelerated runtime available for non-Qualcomm platforms. Newer platforms such as the Snapdragon 855 are benchmarked to have a 5x snpe speedup vs the comparatively ancient 821 in the LEX727 and 3T.
With all this said, there’s really nothing stopping you from running OP on another SNPE supported device but camera interfacing. It’s rumored that getting visiond to work with a phone’s camera is the reason why we still use IMX298 based hardware as camera manufacturers don’t publish specs, drivers and configs.
It’s become more complicated to train your own model now, as comma now performs vision radar, laneless, and coming lane change in their models; which isn’t trivial to develop as the subject matter is not well published online or in academia. You may want to target openpilot 0.5.8.
I’m not going to go into too much detail in training your own models, but I will point you to https://paperswithcode.com/task/lane-detection which is an excellent resource for lane models, complete with papers. Your objective is to input a camera image and output polys, and probabilities.
Specifically, all you need is a left lane poly, right lane poly, and the probabilities of each. You don’t even really need polys, 50 points each will suffice, as the conversion from points to polys is in code, not from the model.
Here’s a good dataset, or create your own with OP data: https://bdd-data.berkeley.edu
I’m excited to see your results!