This is the 2nd challenge from FireEye's 2015 "FLARE On" challenge (http://flare-on.com/)
The first thing to notice about this one is how similar it looks to the last challenge.
Although this time, instead of a simple XOR loop, it looks like we're calling sub_401084 to decide if "You are success" or if "You are failure":
Let's take a closer look at that subroutine:
While it looks like there are some weird instructions used in there, it does look rather small and hopefully we can replicate the logic in a separate Python script.
After writing this out by hand on a piece of paper, reducing & rewriting it a few times, I came up with something along the lines of this:
def rol(byte, count):
byte = (byte << count | byte >> (8 - count)) & 0xFF
data = '\xAF\xAA\xAD\xEB\xAE\xAA\xEC\xA4\xBA\xAF\xAE\xAA\x8A\xC0\xA7\xB0\xBC\x9A\xBA\xA5\xA5\xBA\xAF\xB8\x9D\xB8\xF9\xAE\x9D\xAB\xB4\xBC\xB6\xB3\x90\x9A\xA8'[::-1]
AH = AL = AX = BX = DX = 0
result = ''
for c in data:
AH = rol(1, DX)
AL = (ord(c) - AH - 1) ^ 0xC7
BX += ord(c)
DX = BX & 3
result += chr(AL)
Running this across the data stored in memory gives you something like this: