If you use H3VR, you don`t need to convert to 2nd or 3rd order. You need to make sure if your microphone records in B-Format. (AmbiX should be in the recorder's settings). If that's the case, you don't need to use Reaper to convert audio, only for fx, equalization etc. if need. It's simple, you just need to follow step 6 of the manual, where you need to select
Spatial Audio: B-format (1nd order ambiX)
Spatial audio file: select 4-channel audio
Head-Locked Stereo: empty or choose stereo audio file (if needed)
Video file: select the video file
After that, follow the next step and add the stereo file as a separate track to the mkv file.
Please, if this is still not working, let me know so that they can fix this problem.