VIDEO WORKS.
NOW FIX AUDIO.
If the active EDID advertises stereo, the player may only offer stereo. Automix/custom audio capability must match the real downstream audio system.
Use basic PCM/stereo to prove the HDMI audio path. Then move to the multichannel/bitstream format your AVR or processor actually supports.
Test Source → AVR directly if needed, then put Vertex back in the chain. This separates an AVR decoding problem from an EDID/routing problem.
Vertex can de-embed HDMI audio and provides its audio output options. Configure extraction only after the incoming HDMI signal/audio is known-good.
The 3.5 mm D/A combo jack gives analog stereo (3.5 mm jack) or up to 5.1 Dolby/DTS (mini-TOSLINK optical). Lossless formats (TrueHD, DTS-HD MA, Atmos) only pass over HDMI. Do not assume every HDMI lossless format can be carried by an optical 5.1 connection.
If you change advertised audio capability, force the source to read the new EDID before deciding the setting did not work.