Add UD0_COMPAT decoder mode

[flobernd]

Closes #436

[jpidancet]

Hi @flobernd, thanks a lot for working on this.
I verified your changes give the expected results using this simple program:

#include <Zydis.h>
#include <assert.h>

int main(void)
{
	ZydisDecoder decoder;
	const char data[4] = "\x0f\xff\x00\x00";
	ZydisDecodedInstruction instruction;
	ZydisDecodedOperand operands[ZYDIS_MAX_OPERAND_COUNT];
	ZyanStatus status;

	ZydisDecoderInit(&decoder, ZYDIS_MACHINE_MODE_LONG_64,
			 ZYDIS_STACK_WIDTH_64);
	ZydisDecoderEnableMode(&decoder, ZYDIS_DECODER_MODE_UD0_COMPAT, ZYAN_FALSE);
	status = ZydisDecoderDecodeFull(&decoder,
					data, sizeof(data),
					&instruction,
					operands);
	assert(ZYAN_SUCCESS(status));
	assert(instruction.mnemonic == ZYDIS_MNEMONIC_UD0);
	assert(instruction.length == 3);

	ZydisDecoderEnableMode(&decoder, ZYDIS_DECODER_MODE_UD0_COMPAT, ZYAN_TRUE);
	status = ZydisDecoderDecodeFull(&decoder,
					data, sizeof(data),
					&instruction,
					operands);

	assert(ZYAN_SUCCESS(status));
	assert(instruction.mnemonic == ZYDIS_MNEMONIC_UD0);
	assert(instruction.length == 2);

	return 0;	
}

[flobernd]

I think the encoder does not like this new mode. Maybe @mappzor can have a look 🙂

[mappzor]

Encoder itself is fine, it can encode everything based on mnemonics, signatures and encodings. ZydisFuzzEncoder starts with a random encoder request, so it needs to figure out how to configure decoder for verification purposes.

I would just add this piece of code here:

    if (request.mnemonic == ZYDIS_MNEMONIC_UD0 && request.operand_count == 0)
    {
        status = ZydisDecoderEnableMode(&decoder, ZYDIS_DECODER_MODE_UD0_COMPAT, ZYAN_TRUE);
        if (!ZYAN_SUCCESS(status))
        {
            fputs("Failed to enable UD0_COMPAT mode\n", ZYAN_STDERR);
            abort();
        }
    }