Please read the disclaimer

#vv_max: Hey, at least its not subleq.

Analyses

We are given a 64bit Windows executable, the binary is expecting 2 string arguments.

arguments

after that we have 3 important functions.

3_important_functions

1) set_funcs_in_array:

Populates an array with 24 different function addresses.

funcs_array

2) run_vm:

Runs the vm engine.

vm_engine

3) check_result_vm:

1- Compares 2 strings generated by the vm. 2- Checks if the first argument is equal to FLARE2019 3- Xor a string generated by the vm with a hardcoded string.

Tracing the vm

The simplest approach to understand how a virtual machine protection works is to understand the functionality of each custom opcode of the vm, then dynamically tracing it’s arguments and it’s output (if required).

To do so I added breakpoints before and after the execution of each custom opcode to retrieve the arguments and the result using IDAPython.

for example:

example_bps

the resulting log file was something as follow:

script_output

after a deep analysis and a backward tracing (form the result to the original data) the hash generated from the second argument, I came up with the following algorithm

input_alteration

Note: It is not needed to understand the algorithm that uses the first argument, as it has to be "FLARE2019"

Knowing the resulting hash from “FLARE2019” after executing the vm is: 7070b2ac01d25e610aa72aa8081c861ae845c829b2f3a11e0000000000000000, and that both hashes generated hashes must be equal, we can just reverse the second algorithm and give it the known hash to retrieve the right input for the second argument.

script here

flags_output

With some fails and tries I was able to retrieve the flag which is: AVX5_VM_M4K3S_BASE64_C0MPL1C4T3D@flare-on.com