CarPlay using RasberryPi on RMC

[Carl0s]

So... I want CarPlay, but i have RMC headunit... And i have RasberryPi to spare. This made think, would it be possible to use RasPi as a "CarPlay bridge" - similar like the aftermarket solutions available.

I know there are ready made solutions available for RasPi to run CarPlay whit 8$ CarPlay USB-adapter from ebay. Then the only issue is, how to get the image to MMI display, and how to use the MMI controls.

I got 3 ideas:

1. Exploit the factory fitted AMI port. If it would be possible to pull the "ipod" controlls and the video input to RasPi, it would be neat solution where CarPlay activation would be similar to external audio activation and all audio, video and controlls would go trough the AMI port. What is still unclear is how the iPod identification is handled ( trough the resistor in pin 21&22 like USB/Aux/Video) or trough the iPod data/iPod detected signal on pins 13,14 and 16.

2. Interrupt the video signal going to the screen / Use the external video feed and video in motion. Tag in to the "controll button loop" and read the button presses for RasPi and use them to controll the CarPlay.

Also, one thing i've been thinking is to ripout the internals of the main unit, solder the RS232 port in place and try to figure out if the normal boot is possible to interupt before the serial terminal goes to MMI CLI, and gain access to system. If system access is possible, then try to run custom software inside the MMI... But i think that one is a longshot.

Comments, ideas? Anyone tried something similar before ?

And before anyone asks why not just buy RSnav / unique auto or similar: Because i like to tinker :)

[Carl0s]

Ok, so last evening i did some digging, and found out there is 2 ways to inject composite video to the MMI:

1. Control unit J794 (in dash) 12-pin connector B pins 5(-) & 11(+). This is the reverse camera input, and the MMI will automaticly switch to it once signal is detected. This is what i assume most of the aftermarket solutions are exploiting. Upside using this method is automaticly activating the Carplay view once phone is connected, downside is that inorder to receive audio, you must manually remember to select AUX as audio source. Exitting from the "Reversing camera" imaging is still unknown.

2. Control unit J794 (in dash) 12-pin connector C (from unit R199 / AMI in centre console). Pins 9(+) & 10(-). This would have the upside of single selection would activate the carplay view, and select the propper audio source on on go. This method requires having 900ohm / 1Kohm (information varies) detection resistor on pins 21 & 22 to activate AMI AUX. Downside is, that Video In Motion has to be activated using VCDS / OBDeleven.

Next step for me is to purchase 4-pole 3.5mm jack to Extract audio and composite video from Rasberry Pi and connect them to AMI, whit the detection resistor and see if it works as planned.

If the above testing goes as planned, next step would be determing how the MMI controll dial works: Can the information be read from CAN whit fairly inexpensive CAN-chip. Other way, if it works anything like the Ford focus mk2 cruise control that i build - whit resistors in loop and each button changes the total resistance value of the loop: it should be fairly simple to read the resistance of the loop, and translate them to commands commanding the Rasberry Pi.

[edmondhk]

I got a Chinese CarPlay box, it use something like your method 2. Hard cut the LVDS signal and intercept the button signal.
For the LVDS signal switching, it use a relay to do it as I heard the click sound when switching.

I don't recommend you use the composite video from Rasberry Pi, just cause you more problem to do the A/D convert. I suggest you use the LVDS directly, but you need to find out the setting, resolution, dead zone, etc. But you need this information anyway.

[Carl0s]

I got a Chinese CarPlay box, it use something like your method 2. Hard cut the LVDS signal and intercept the button signal.
For the LVDS signal switching, it use a relay to do it as I heard the click sound when switching.

I don't recommend you use the composite video from Rasberry Pi, just cause you more problem to do the A/D convert. I suggest you use the LVDS directly, but you need to find out the setting, resolution, dead zone, etc. But you need this information anyway.

Reasons why I'm going for composite video instead of hooking directly to the LVDS signal:

1. All the inputs for the MMI are composite, you can buy AMI to Composite adapter to input Auxiliary video + audio to MMI
2. Rasberry Pie offers Composite TV-out and analog audio directly from 3.5mm jack. So its direct Analog to analog. LVDS would require additional HW layer to function.

Small update: Due to covid-19 all the necessary parts had to be ordered by mail, so everything takes sooo long. Today I did some tests on the AMI connecter, and the 1kOhm resistor only activates AUX audio in. AUX video in requires different detector resistor, that needs to be figured out. Unfortunately, that information seems to be not available. Now I'm thinking if I should just buy adapter and measure the resistance, or buy bunch of potentiometers and try to find the correct resistor value.

But, my Rasberry CAN-hat should arrive early next week, so then I get to map the CAN-commands for the dial and the surrounding 4 buttons for navigation.

[edmondhk]

Reasons why I'm going for composite video instead of hooking directly to the LVDS signal:

1. All the inputs for the MMI are composite, you can buy AMI to Composite adapter to input Auxiliary video + audio to MMI
2. Rasberry Pie offers Composite TV-out and analog audio directly from 3.5mm jack. So its direct Analog to analog. LVDS would require additional HW layer to function.

Small update: Due to covid-19 all the necessary parts had to be ordered by mail, so everything takes sooo long. Today I did some tests on the AMI connecter, and the 1kOhm resistor only activates AUX audio in. AUX video in requires different detector resistor, that needs to be figured out. Unfortunately, that information seems to be not available. Now I'm thinking if I should just buy adapter and measure the resistance, or buy bunch of potentiometers and try to find the correct resistor value.

But, my Rasberry CAN-hat should arrive early next week, so then I get to map the CAN-commands for the dial and the surrounding 4 buttons for navigation.

So you build this on top of existing MMI instead of independently? Sounds like a good idea as it also kept the parking camera work normally. Will you plan to do this open source or close source?

[Acejam]

Any updates on this?

[Carl0s]

If images are not showing they can be seen at: https://ibb.co/mHzY2Xm https://ibb.co/1Z6LqJM
Progress is slow, but atleast it's still progress :)

Now the setup is Raspberry 4 on KonstaT's LineageOS: https://konstakang.com/devices/rpi4/LineageOS17.1/ which is runnig the AutoPlay.apk for the Carplay usb-dongle. Image/analog audio is taken from raspberrys TV-out and feeded to MMI trough AMI-port. I enabled video in motion using OBD11 so the image is visible during driving. Tests shown on images were still controlled using USB-keyboard.

Next step is connect the RPI4's serial port to the MMI control panel. The control panel communicates with the mainunit using rs232 not CAN. I started mapping the commands using some cheap USB-serial adapter, but there were too many dirty characters to make any use of it. I need to give it another try using RPI native serial, and then figure out a way for:

a) Use these serial commands and keyboard inputs for android
b) Build logic that these command only controll the android when "media input" is selected. I dont want the wheel to controll the rpi if im on radio mode :)

[Carl0s]

Updates!

I did some small refinements. Inorder to have Raspberrys TV-out working i was required to use Swifthshader software rendered instead of the hardware renderer. This caused significant dip in performance. To counter this, i overclocked the pi to 1.9ghz and added cooling fins for the chips. Now the operation is smooth enough and the pi stays cool. I installed all the components to vented box that fits nicely inside the lower partment of the armrest (pics follow later).

Now when i turn on the car, the PI boots and the carplay comes up automaticly. If i pull the phone off mid use, android home screen is shown, but when phone is re-inserted the Carplay comes back up.

I also finished mapping the e380 control knob commands:

Connection: RS232
Bitrate: 19200
Data bits: 8
Stop bits: 1
Parity: Odd

Using these connection settings i get following messages from knob actions:

Wheel left: ”Qw”
Wheel right: ”Pv”
Wheel press: ”0V”
Wheel press release: ”1W”

Whats left is to deep dive to android development on how i can use serial port in android, and make small app that when it reads "Qw" from serial port, it send ie. "T" keyboard event to android system. Why "T" instead of left arrow? The autokit usb-dongle has its own keyadaptation where i can map that "T" -> "next item on list" etc. Using "T" that has no function on android unlike ie. left arrow, i can eliminate the need to monitor the MMI input status so i wont mess around in the android when ie. controlling the radio using the control wheel.

If anyone has any ideas how this can be done, i would more then apreciate it :)

[Carl0s]

This is startin to look like a monolog but anyways:

I made some changes:
In the last keymapping i complitely overlooked the "next track" "previous track" buttons located bellow the volume controll. Yesterday i was soldering the rs232 connection wires for the Raspberry to the E380 harnes, and while i was there i completed the keymapping. The full controll-message-map follows:
Wheel counter clockwise: ”Qw”
Wheel clockwise: ”Pv”
Wheel:
On press: ”0V”
On release: ”1W”
Media: ”0W1X”
Previus song:
on press: ”a”
on release: ”b”
Next song:
on press: ”0u”
on relese: ”1v”

Also i made some changes to how i power the Raspberry. On first install i used the 12v socket inside the arm rest for power. This was sub-optimal solution since the 12v outlet is tied to ignition power not MMI state. This caused when ever i turned off the car, the Raspberry turned off even tho the MMI was still on. Also, i've noticed that when you unlock the car, the MMI starts to boot in the background.

So, i purchased DC-DC converter board and adjusted the output to 5V. I ditched the original USB-C based power delivery, and hooked the DC-DC converter directly to the 5v GPIO pins. I Also added 12v 50mm fan running on 5v to the box for better cooling.

BUT!
I cant find any 12v signal that follows the MMI state... The wakeup signal for E380 is 6V and other wakeups are trough CAN or the optical bus. I could build a separete mosfet controlled cirquit that uses the 6v to turn on the 12v for the DC-DC converter, but the space inside the box is starting to run out. One option would be to use the iPod power delivery from AMI (5v) but i dont know the current limit for that.

The software part is still big questionmark. Started looking into android development, but onfortunately cant even run the Android studios simulator on my mac :/

[andymassey]

A monologue yes, but only because you’ve done amazing work @Carl0s!

Did you completely finish the job? Is it something you think the rest of us mere mortals could manage..? Wondering whether to try…

Thanks for your feedback!

Andy

[Audisthesia]

If you look at the btres, it also uses a pi to pull obd data to a little lcd display.. You can do a lot with a pi. Ive got one sitting on the shelf at home, and you got me thinking what can i do with it...

[Carl0s]

Sooo, this project was putted to ice mainly due to two reasons:

1. Performance. Running Android on Raspberry pi was usable but not as fluid as I've would've wanted. This was mainly due to lack of Android support for hardware encoding on tv-out.
2. Controlling the box. I couldn't find the time and motivation to immerse my self into Android development on a level where I could create piece of code that would translate the MMI controls to Android commands.

Due to these reasons, after running the "dummy proto unit" like a month I parted the box and forgot this project.


Time goes by, and by some miracle I stumbled on this on GitHUB: https://github.com/rhysmorgan134/react-carplay somebody had similar ideas as I did, and created python based bridge for the CarLinkIt CarPlay usb dodge, and this app is nearly plug and play on RasbperryPI!

Filled with excitement I rebooted my project. Blowed the dust away from my Raspberry, throw a fresh RaspberryPI OS on it and gave it a test. And its almost everything I needed and definitely something I can build on top. Performance on bone stock RaspberryPI OS is on a whole different level compared to Android and since its LINUX, I can use python to read the serial input of Raspberry and convert them to any fucking system command I want! Fuck yeah! (I'm pretty exited if it won't translate trough my text :D )

BUT.... There's always that god forsaken but that diminishes everything that came before it... :(

When running the CarlinkIt usb dongle on the Android .apk as it was intended, there is settings section that has key mapping. In this keymapping you can map any input to pre-determined function like next item, previous item, SIRI, Home, Back... you get the point. Feature that is essential for my intended use case. Unfortunately the fine fellow who did the majority of the work on React CarPlay have not implemented this key mapping functionality on his software, since he intended to use it with Jaguar system that has touchscreen, and apparently can tag into Jag's touch input.

So at first, I contacted the developer if he has any plans to implement the key mapping (there is file called key mappings in he's project, but the mapped keys do not function) but he never did respond to me. Now I'm juggling should I try to develop the key mapping feature my self or maybe try find some other friend who could do this.

But, from the experience I had on the previous tests I feel confident that using the AUX in on MMI was the best way of doing this in DIY terms. Its easy, the picture quality is good enough and the necessary parts are easily sourceable and implementable. Also the operating logic is simple, select the AMI input and you have CarPlay, instead of the other systems where you use NAV to toggle the image and then select audio source differently. (just my opinion)

But, the saga shall continue and I will report if I have any news :)

[andymassey]

Thanks both for the replies.

I’d actually starred that GitHub repo, but it didn’t look like it had enough detail for my fundamental RPi skills to follow…

Wish I could help you with the key mapping Carl0s. Really seemed like you’d basically nailed it previously. But yeah, I tend to steer clear of Android too.

Maybe if a genius like you hasn’t nailed it, I’d probably be better just going for a dumbass kit from AliExpress. (This one seems like it should work: https://www.aliexpress.com/item/1005002465584418.html ) I have way too many half finished projects where I’ve either got disinterested or bottled out, and don’t fancy messing up the Audi or my wife will divorce me… Motorbike is fine, and doesn’t affect her!

I had had dreams of using the RPi to also do the RoadApplePi (https://github.com/JVital2013/RoadApplePi) too, but should know where my limits lie by now!

I’ll follow your progress vicariously through you if you continue to post here Carl0s! Good luck.

[Carl0s]

Thanks, but I'm no genius on any standard. Just building a puzzle from peaces of information that I've managed to dig from different sources (mainly from Audis self study programs and via try and error).

That React CarPlay repo from GitHub has nearly fully functioning AppImage that can be run on PI and if you have touchscreen its fully functional. Too bad that Audis does not have touchscreen - maybe in that case it would've been possible to piggy bag it 😅

There is updated version on GitHub that has the key mapping project started, but that version is not yet compiled to AppImage. It's electron app and you should be able to run it on your own computer via electron for testing purposes and for developing it further. But, there seems to be some issues with Mac compatibility - not all required decencies can be fulfilled so can't get the app run properly trough electron.

But, still digging for solution, and I will keep it updated here :)

[andymassey]

But, still digging for solution, and I will keep it updated here :)

Thank you! (And yes, genius.)