[KalmarCTF] [REV] Oracle

TL;DR Oracle turned out to be a nice little reversing chain with just enough nonsense to stay fun. The core idea was simple once I found the real entrypoint: the binary decrypts a blob with the first 7 input bytes, jumps through a Heaven’s Gate transition into 64 bit code, then uses a polynomial decoder and a final FNV-1a casing check to validate the flag.

The actual entrypoint

I started in Lazarus land and just searched for TWindow strings in IDA. That exposed the public methods right below it, which gave me a decent place to start poking around:

FormResize 0x004264E0
inputFieldKeyPress 0x004268A0
inputFieldKeyPress2 0x00426760
inputFieldKeyPress3 0x00426690

FormResize ended up being the useful entrypoint for static analysis. It had an anti debug check that would bounce execution into the other keypress handlers if a debugger was present. If not, it eventually flowed into inputFieldKeyPress, which also had the Enter key check, so that matched the visible behavior in the UI.

From there I followed a call to 0x00426430, and that is where the first real logic showed up.

First stage: blob decryption with XOR 0x57

That function decrypted a bunch of blobs with 0x57. Those became the strings shown in the message boxes. For example, the first decrypted one was:

Speak your truth and i shall convene with the gods on your behalf

Then it loaded 0x4B7 bytes from 0x00556260 and called into it. That was the part that actually mattered.

The first stage of that blob decrypted itself using the input:

for (i = 0; i < size; i++) {
blob[i] = blob[i] ^ input[i % 7] ^ 0x57;
}

That input[i % 7] is the important bit. It only uses the first 7 input bytes, over and over, as a rolling key.

So the problem immediately becomes: recover those first 7 bytes, and the rest of the blob opens up.

Recovering the first 7 bytes

After the self decryption, the code checked whether the first 7 decrypted bytes were all 0x90. That gives the equation directly:

encrypted[i] ^ input[i] ^ 0x57 = 0x90

Solve for the input byte:

input[i] = encrypted[i] ^ 0x57 ^ 0x90
input[i] = encrypted[i] ^ 0xC7

So yeah, the challenge more or less makes you recover the first seven bytes from the encrypted blob. In practice this just confirms the expected flag prefix. A lot of work for basically rediscovering kalmar{, which was funny, but it also gives the correct rolling XOR key for the next stage.

Once I applied that statically to the bytes starting at 0x005562C2, the decrypted body started to make sense. NOPs showed up first, then actual instructions.

The bitness switch that makes IDA look cursed

At 0x00556326 the code pulled a Heaven’s Gate trick, which is why IDA looked pretty confused at first.

The important chunk was this:

.data:0000000000556330 push 33h
.data:0000000000556332 call $+5
.data:0000000000556337 mov eax, 9
.data:000000000055633C add [rsp+10h], eax
.data:000000000055633F retf

Pushing 0x33 puts the 64 bit code segment on the stack on WoW64, so the retf switches execution into 64 bit mode. Once I recognized that, the fix in IDA was straightforward: treat the relevant region as 64 bit and continue from there.

Below that, I found:

.data:0000000000556348 jmp loc_5565EA

Which led here:

.data:00000000005565EA mov rdi, r9
.data:00000000005565ED mov rsi, rsp
.data:00000000005565F0 sub rsi, 1000h
.data:00000000005565F7 call length
.data:00000000005565FC cmp rax, 28h
.data:0000000000556600 jnz loc_5566F6
.data:0000000000556606 mov r10, rax
.data:0000000000556609 mov r11, rsi
.data:000000000055660C jmp loc_5563AD

There is a length check for 0x28, which is 40. I did not spend much time on it. At that point it was obvious enough that the flag length was 40, so I just kept going.

The weird call/pop trick

The next part was classic stack nonsense.

Execution landed here:

.data:00000000005563AD call mr_cool
.data:00000000005563B2 db 55h, 55h, 0A5h, 0C0h, 00h

And mr_cool looked like this:

.data:0000000000556611 pop rsi
.data:0000000000556612 mov rdi, r10
.data:0000000000556615 call helper_algo
.data:000000000055661A lea rbx, [r11+r10]
.data:000000000055661E mov bl, [rbx]
.data:0000000000556620 cmp al, bl
.data:0000000000556622 jnz loc_5566F6

That pop rsi grabs the return address from the call mr_cool, so rsi now points at 0x5563B2. In other words, the bytes directly after the call are not random garbage. They are data.

That data continues for a while and ends with two NOPs:

.data:0000000000556460 db 0BCh
.data:0000000000556461 db 49h
.data:0000000000556462 nop
.data:0000000000556463 nop

At that point I was pretty convinced the bytes after the call were the real payload, and helper_algo was the thing used to interpret them.

That was correct.

The polynomial decoder

After cleaning it up, helper_algo boiled down to this:

unsigned char helper_algo(int index, float table) {
int chunk = index / 4;
float coef = (float )((char )table + chunk * 16);
float a = coef[0];
float b = coef[1];
float c = coef[2];
float d = coef[3];
float x = (float)index;
float y = d + cx + bxx + axxx;
int v = (int)y;
float frac = y - (float)v;
return (unsigned char)v;
}

So for every character index, it reads four floats, treats them as cubic polynomial coefficients, evaluates the polynomial, converts the result to a byte, and compares that against the corresponding character from the flag candidate.

That sounds more dramatic than it really is. In practice I just dumped the bytes from IDA and wrote a quick Python script.

import struct

data = bytes.fromhex(
"55 55 A5 C0 00 00 D0 41 AB AA F6 C1 00 00 96 42 "
"00 00 94 C1 00 C0 90 43 00 80 B6 C4 00 90 19 45 "
"AB AA EA C0 00 00 53 43 55 F5 F8 C4 00 C8 C2 45 "
"55 55 45 41 00 C0 F2 C3 55 29 C6 45 00 24 D6 C6 "
"55 55 3D C1 00 E0 1D 44 AB 16 2F C6 80 41 81 47 "
"55 55 CD 41 00 70 CF C4 D5 83 0B 47 40 CD 79 C8 "
"55 55 BD 41 00 50 E2 C4 D5 25 34 47 00 F5 BE C8 "
"55 55 95 C0 00 00 D2 43 55 99 44 C6 00 38 F5 47 "
"55 55 45 41 00 30 99 C4 2B 71 1E 47 80 4F DA C8 "
"AB AA BA C1 00 D8 23 45 95 A9 BF C7 38 69 95 49 "
"AB AA A6 C1 00 10 24 45 95 4A D7 C7 08 4F BC 49"
)

def f32(b):
return struct.unpack("<f", b)[0]

def helper(index, data):
chunk = index // 4
off = chunk * 16

a = f32(data[off:off+4])
b = f32(data[off+4:off+8])
c = f32(data[off+8:off+12])
d = f32(data[off+12:off+16])

x = float(index)
y = d + c*x + b*(x*x) + a*(x*x*x)

v = int(y)
frac = y - float(v)

if frac > 0.5:
    v += 1

return v & 0xff

flag = ""
for i in range(40):
ch = chr(helper(i, data))
print(i, ch)
flag += ch

print(flag)

That printed:

KALMAR{M15S_TH3_S1GN5_4ND_3NTER_TH3_M4Z3}

So at that point I had basically the whole flag. Except the casing was still wrong.

Of course it was not over yet.

The final casing checks

There was still a block of code after the polynomial comparison loop that clearly did something important:

.data:0000000000556628 dec r10
.data:000000000055662B cmp r10, 0
.data:000000000055662F jge short loc_556612
.data:0000000000556631 mov rdi, r9
.data:0000000000556634 add rdi, 7
.data:0000000000556638 mov rsi, 4
.data:000000000055663F call magic_last_casing_part
.data:0000000000556644 cmp eax, 1FDB82EBh
.data:0000000000556649 jnz loc_5566F6
.data:000000000055664F mov rdi, r9
.data:0000000000556652 add rdi, 0Ch
.data:0000000000556656 mov rsi, 3
.data:000000000055665D call magic_last_casing_part
.data:0000000000556662 cmp eax, 0C498C8A6h
.data:0000000000556667 jnz loc_5566F6
.data:000000000055666D mov rdi, r9
.data:0000000000556670 add rdi, 10h
.data:0000000000556674 mov rsi, 5
.data:000000000055667B call magic_last_casing_part
.data:0000000000556680 cmp eax, 0D451383Bh
.data:0000000000556685 jnz short loc_5566F6
.data:0000000000556687 mov rdi, r9
.data:000000000055668A add rdi, 16h
.data:000000000055668E mov rsi, 3
.data:0000000000556695 call magic_last_casing_part
.data:000000000055669A cmp eax, 0F1D1BDC9h
.data:000000000055669F jnz short loc_5566F6
.data:00000000005566A1 mov rdi, r9
.data:00000000005566A4 add rdi, 1Ah
.data:00000000005566A8 mov rsi, 5
.data:00000000005566AF call magic_last_casing_part
.data:00000000005566B4 cmp eax, 5768CCA7h
.data:00000000005566B9 jnz short loc_5566F6
.data:00000000005566BB mov rdi, r9
.data:00000000005566BE add rdi, 20h
.data:00000000005566C2 mov rsi, 3
.data:00000000005566C9 call magic_last_casing_part
.data:00000000005566CE cmp eax, 0E25D1966h
.data:00000000005566D3 jnz short loc_5566F6
.data:00000000005566D5 mov rdi, r9
.data:00000000005566D8 add rdi, 24h
.data:00000000005566DC mov rsi, 4
.data:00000000005566E3 call magic_last_casing_part
.data:00000000005566E8 cmp eax, 45B2772Bh
.data:00000000005566ED jnz short loc_5566F6

What this does is hash a few short slices of the input and compare them against constants.

For example:

mov rdi, r9
add rdi, 7
mov rsi, 4
call magic_last_casing_part
cmp eax, 1FDB82EBh

That means:

hash(flag[7:11]) == 0x1FDB82EB

And the same pattern repeats for these ranges:

flag[7:11]
flag[12:15]
flag[16:21]
flag[22:25]
flag[26:31]
flag[32:35]
flag[36:40]

So the remaining job was just to identify that hash function and brute force the case of each chunk.

The function turned out to be FNV-1a:

def fnv1a(bs):
h = 0x811C9DC5
for b in bs:
h ^= b
h = (h * 0x01000193) & 0xFFFFFFFF
return h

Once I had that, the casing recovery was easy. Each chunk is only a few bytes long, so brute forcing upper and lower case combinations is basically instant.

Here is the full solver I used:

import struct
from itertools import product

data = bytes.fromhex(
"55 55 A5 C0 00 00 D0 41 AB AA F6 C1 00 00 96 42 "
"00 00 94 C1 00 C0 90 43 00 80 B6 C4 00 90 19 45 "
"AB AA EA C0 00 00 53 43 55 F5 F8 C4 00 C8 C2 45 "
"55 55 45 41 00 C0 F2 C3 55 29 C6 45 00 24 D6 C6 "
"55 55 3D C1 00 E0 1D 44 AB 16 2F C6 80 41 81 47 "
"55 55 CD 41 00 70 CF C4 D5 83 0B 47 40 CD 79 C8 "
"55 55 BD 41 00 50 E2 C4 D5 25 34 47 00 F5 BE C8 "
"55 55 95 C0 00 00 D2 43 55 99 44 C6 00 38 F5 47 "
"55 55 45 41 00 30 99 C4 2B 71 1E 47 80 4F DA C8 "
"AB AA BA C1 00 D8 23 45 95 A9 BF C7 38 69 95 49 "
"AB AA A6 C1 00 10 24 45 95 4A D7 C7 08 4F BC 49"
)

checks = [
(7, 4, 0x1FDB82EB),
(12, 3, 0xC498C8A6),
(16, 5, 0xD451383B),
(22, 3, 0xF1D1BDC9),
(26, 5, 0x5768CCA7),
(32, 3, 0xE25D1966),
(36, 4, 0x45B2772B),
]

def f32(b):
return struct.unpack("<f", b)[0]

def helper(i):
off = (i // 4) * 16
a = f32(data[off:off + 4])
b = f32(data[off + 4:off + 8])
c = f32(data[off + 8:off + 12])
d = f32(data[off + 12:off + 16])
x = float(i)
y = d + c * x + b * x * x + a * x * x * x
v = int(y)
if y - float(v) > 0.5:
v += 1
return chr(v & 0xFF)

def fnv1a(bs):
h = 0x811C9DC5
for b in bs:
h ^= b
h = (h * 0x01000193) & 0xFFFFFFFF
return h

def recover_case(chunk, target):
choices = []
for ch in chunk:
if ch.isalpha():
choices.append((ch.lower(), ch.upper()))
else:
choices.append((ch,))
for cand in product(*choices):
s = "".join(cand)
if fnv1a(s.encode()) == target:
return s
raise ValueError((chunk, hex(target)))

flag = list("".join(helper(i) for i in range(40)))

for start, size, target in checks:
solved = recover_case("".join(flag[start:start + size]), target)
flag[start:start + size] = solved

flag = "".join(flag)
print(flag)

That printed:

KALMAR{M15S_Th3_S1gN5_4Nd_3NteR_tH3_M4Z3}

Then I just fixed the standard prefix casing.

Flag

kalmar{M15S_Th3_S1gN5_4Nd_3NteR_tH3_M4Z3}

Wrap up

I liked this one.

The initial UI and anti debug bits try to waste your time, then the challenge pivots into a self decrypting blob, a WoW64 Heaven’s Gate transition, a polynomial based flag decoder, and one last casing check hidden behind FNV-1a chunk hashes. That is a pretty decent amount of trickery without turning into complete garbage.

The nicest part is that once you stop treating it like one giant cursed blob and instead split it into stages, each piece is actually manageable. Recover the 7 byte key, decrypt the body, understand the 64 bit handoff, dump the float table, then brute force casing.

Messy on first look. Pretty clean once the structure shows itself.