Pixelization With XP, VMR9, and FSE

[Straykat]

Problem: overlay looks great with normal DVDs (.TS) and SD shows but FSE+VMR9 is pixelized. Text looks blocky, faces have jaggy edges. Crawlers on TV have blocky text. Switch back to overlay, everything looks great again. Dropping the same .TS into Graphedit using the VMR9 renderer, in any scaling, looks fantastic. This happens to all SD video. TV, DVDs, MPEGs.

Resolution attempts:
* New install of Sage 6.5.7, 6.5.9, and 6.5.12.
* Nvidia Decoder
* Cyberlink standard decoder (power dvd 7.3)
* With and without Haali
* VMR9 without FSE (blue screens the second I play a .TS)
* All sorts of deinterlacing options (maybe I missed one that works?)
* Different deinterlacing options in nVidia's decoder, Cyberlink, and Sage
* Overlay with FSE
* All the above in Sage with 100% scaling

Sadly, VMR9+FSE is the only way I have gotten DXVA to work.
Any suggestions on how to get rid of the jaggies?

Edit: Interesting thing. When a .TS is running in FSE+VMR9, I can alt-tab, forcing sage to a windowed mode, then I can continue to play the .TS with VMR9, without FSE. But as soon as I switch to another .TS, I get a blue screen.

Using nvidia driver 182.08 WHQL