Following up.... WMVHD at 1080p with Dolby 5.1 (AC3) audio is working great.
Unfortunately the xbox can't play x264 with multichannel audio, so I've got to reencode to WMVHD, which is a little tedious, but works well.
I demux the MKV file, and pull out the DTS audio if I have it, and quickly convert it over to Dolby (AC3). Then I use tmpgenc to encode the x264 along with the Dolby audio to WMV, about 10000 bitrate, 1080p.
There is some minor quality loss, but I'm also ending up with slightly smaller files, I suspect if I wanted to make them larger I would lose less quality. That being said, it is reasonably comparable to watching a bluray/hddvd directly, and superior to OTA HD broadcasts.