Various fixes / improvements to the old FX9 decompiler

[lanyizi]

• Now every test of DX9's AsmMatchesFxc passes! (Failing disassembly tests #9)
• The decompiler can now (almost) always produce ready-to-compile DirectX 9 shaders. In the 1008 test cases of DX9Tests, 992 cases have succeeded, only 16 cases have failed (mainly because there are still some unimplemented instructions).
  The HlslAst has been broken by me (mainly because I don't know how to let its nodes to support relative addressing), so HlslWriter's doAstAnalysis option will no longer work, but in my opinion that won't be a big deal since you're working on a new common IR design anyway ;)
• Fxlc shaders (expressions and preshaders) now are correctly decompiled using human-understandable constant variable names instead of stuffs like c0 c1 c2
• Main shader can now correctly reference the outputs of preshader by looking at RegisterState.CtabOverride (since preshader would overwrite constant buffer)
• Tried to ensure decompilation is correct when regarding the write masks of destination registers, but I'm still a bit unsure about them (I feel they are counterintuitive, write masks are not how things work in HLSL)

[Joschuka]

Thanks a lot for this PR, it's extremely useful for a project I'm currently working on to decompile some DX9 shader bytecode. The output is the best I've seen so far and makes it much easier to understand what's going on.
I was wondering if you were planning to add support for the last few unsupported instructions soon or if I should try to fix it myself, I have a few pixel shaders that have unsupported ones (they disassemble without problems though).

[lanyizi]

Thanks a lot for this PR, it's extremely useful for a project I'm currently working on to decompile some DX9 shader bytecode. The output is the best I've seen so far and makes it much easier to understand what's going on.

You're welcome, and thanks for the compliment, I feel so happy when reading them xD

I was wondering if you were planning to add support for the last few unsupported instructions soon or if I should try to fix it myself, I have a few pixel shaders that have unsupported ones (they disassemble without problems though).

You can try to fix them yourself, just remember to run the unit tests after the changes (currently my version passes 992 out of 1008 test cases in DX9Tests). Since spacehamster seems to be busy at the moment, you can consider creating pull request on my forked repository instead.

[Joschuka]

Got it thanks ! I'm not super familiar with shaders but hopefully I should be able to study your code and add the missing instructions (I think it's the saturate one that wasn't implemented in my case). I'll make a PR when I have the time, thanks again for your great work !

[Albeoris]

@lanyizi , can I disassemble a shader in this format?
CubeBlend.shader.txt

[lanyizi]

@Albeoris from what I see, that file is already disassembled, isn't it?
If you mean decompiling, then it's a bit harder,

1. If you want DXDecompiler to read your txt contaning assembly, and expect it to produce HLSL source code, then currently it's not possible because DXDecompiler doesn't parse plain-text assembly (instead it produces plain-text assembly by reading a binary file).
2. That seems to be Unity's own ShaderLab format, I don't know much about it, but it looks different from standard HLSL shader / HLSL effect.

However, there are existing projects like AssetRipper which should be able to extract many types of Unity Assets, including shaders IIRC. I suggest you to take a look at those projects, they also have a Discord channel.

[Albeoris]

@lanyizi , yeah. As far as I understand, this is the format of precompiled shaders that used older versions of Unity.
Accordingly, inside the resources of the game they are in the same format.
And I don't know how to make them work on a modern version of Unity. I need to either disassemble them or compile them. And I have not found any tool that is capable of this. :С

Judging by the instructions, this is HLSL:
https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx9-graphics-reference-asm-vs-3-0

[OktoberfestTheEnthusiast]

Hello! @lanyizi I noticed that you made a pull request for a DirectX9 Shader Decompiler at GitHub, and I was wondering if you could maybe help with this? I'm currently attempting to Decompile a Shader for Guitar Hero: World Tour on PC but I get this traceback whenever I try to decompile it, I would appreciate the help if you can since it would be really great to have this Shader fixed, thank you and have a great day!

System.NotImplementedException: Saturate, PartialPrecision
at DXDecompiler.DX9Shader.HlslWriter.g__WriteAssignment|14_0(String sourceFormat, SourceOperand[] args, <>c__DisplayClass14_0& ) in C:\Users\Diego\Documents\DXDecompiler (Correct)\src\DXDecompiler\DX9Shader\Decompiler\HlslWriter.cs:line 188
at DXDecompiler.DX9Shader.HlslWriter.WriteInstruction(InstructionToken instruction) in C:\Users\Diego\Documents\DXDecompiler (Correct)\src\DXDecompiler\DX9Shader\Decompiler\HlslWriter.cs:line 297
at DXDecompiler.DX9Shader.HlslWriter.WriteInstructionList() in C:\Users\Diego\Documents\DXDecompiler (Correct)\src\DXDecompiler\DX9Shader\Decompiler\HlslWriter.cs:line 764
at DXDecompiler.DX9Shader.HlslWriter.Write() in C:\Users\Diego\Documents\DXDecompiler (Correct)\src\DXDecompiler\DX9Shader\Decompiler\HlslWriter.cs:line 654
at DXDecompiler.DX9Shader.DecompileWriter.Decompile() in C:\Users\Diego\Documents\DXDecompiler (Correct)\src\DXDecompiler\DX9Shader\DecompileWriter.cs:line 44
at DXDecompiler.DX9Shader.HlslWriter.Decompile(ShaderModel shaderModel, String entryPoint, EffectHLSLWriter effect) in C:\Users\Diego\Documents\DXDecompiler (Correct)\src\DXDecompiler\DX9Shader\Decompiler\HlslWriter.cs:line 79
at DXDecompiler.DX9Shader.HlslWriter.Decompile(Byte[] bytecode, String entryPoint) in C:\Users\Diego\Documents\DXDecompiler (Correct)\src\DXDecompiler\DX9Shader\Decompiler\HlslWriter.cs:line 67
at DXDecompilerCmd.Program.Main(String[] args) in C:\Users\Diego\Documents\DXDecompiler (Correct)\src\DXDecompilerCmd\Program.cs:line 160

HairShader.zip

[OktoberfestTheEnthusiast]

Update:

I was looking at the shader's assembly and I noticed that it had an odd assembly line that's most likely not supported by your decompiler....

from HairShader_0_ps0.fxc

'dp3_sat_pp r1.w, r1, r2'
'mul_sat_pp r0.w, r0.w, r2.w'
'mul_sat_pp r1.xyz, r1.w, c14.xyyw'

These three assembly lines uses '_sat_pp' in their file, and it's mostly likely a combination of PartialPrecision and Saturate.