Custom Encryption CTF Challenge Walkthrough – Step-by-Step Guide

Custom Encryption Challenge - CTF Writeup

Challenge Overview

This was an enjoyable crypto challenge that integrated several encryption methods and a single proprietary algorithm. We had a Python script (`custom_encryption.py`) that contained the encryption logic, and an encrypted flag file (`enc_flag`) with the ciphertext we needed to decrypt.

Initial Analysis

When I first reviewed the code, I could see that it wasn't your typical single-layer encryption. The `test()` function was doing something interesting - it was chaining multiple encryption methods together:

1. Some kind of key exchange mechanism (looked like Diffie-Hellman)

2. A custom XOR operation 

3. A multiplication-based cipher

The encrypted flag file gave us these values:

a = 94

b = 21

cipher is: [131553, 993956, 964722, 1359381, 43851, 1169360, 950105, 321574, 1081658, 613914, 0, 1213211, 306957, 73085, 993956, 0, 321574, 1257062, 14617, 906254, 350808, 394659, 87702, 87702, 248489, 87702, 380042, 745467, 467744, 716233, 380042, 102319, 175404, 248489]

Breaking Down the Encryption

Step 1: Diffie-Hellman Key Exchange

The code starts with a classic Diffie-Hellman setup:

- p = 97 (prime modulus)

- g = 31 (generator)

- a = 94 and b = 21 (private keys)

The shared key calculation follows the standard DH formula:

u = g^a mod p = 31^94 mod 97

v = g^b mod p = 31^21 mod 97

shared_key = v^a mod p = u^b mod p

This gives us our multiplication key for the final encryption step.

Step 2: Dynamic XOR with String Reversal

The `dynamic_xor_encrypt()` function was particularly tricky. It does two things:

1.Reverses the input string using `plaintext[::-1]`

2. XORs each character with the repeating key "trudeau"

The reversal caught me off guard initially - it's not something you see in typical XOR implementations.

Step 3: Multiplication Cipher

The final `encrypt()` function multiplies each character's ASCII value by (shared_key * 311). The constant 311 seemed arbitrary, but was consistent throughout.

Building the Decryption

To reverse this, I had to work backwards through each step:

Reverse Step 3: Division

def decrypt_multiplication(cipher_list, key):

    plaintext_chars = []

    for value in cipher_list:

        if value == 0:

            plaintext_chars.append(chr(0))  # Handle null chars

        else:

            original_ord = value // (key * 311)

            plaintext_chars.append(chr(original_ord))

    return ''.join(plaintext_chars)

Reverse Step 2: XOR and Un-reverse

def dynamic_xor_decrypt(cipher_text, text_key):

    plaintext = ""

    key_length = len(text_key)

    

    # XOR is its own inverse

    for i, char in enumerate(cipher_text):

        key_char = text_key[i % key_length]

        decrypted_char = chr(ord(char) ^ ord(key_char))

        plaintext += decrypted_char

    

    # Reverse the string back to original order

    return plaintext[::-1]

 Reverse Step 1: Reproduce the Key

I had to calculate the same shared key using the given a and b values:

u = generator(g, a, p)  # 31^94 mod 97

v = generator(g, b, p)  # 31^21 mod 97

shared_key = generator(v, a, p)  # (31^21)^94 mod 97

The Gotchas

A few things that made this challenge interesting:

1. The string reversal in the XOR function - easy to miss if you're just skimming the code

2. Zero handling - some cipher values were 0, representing null characters

3. Order of operations - had to decrypt in exactly the reverse order of encryption

4. The magic number 311 - just had to accept it was part of the algorithm

Solution

Putting it all together, the decryption process:

1. Calculate the Diffie-Hellman shared key using the provided `a=94` and `b=21`

2. Divide each cipher value by `(shared_key * 311)` to get ASCII values

3. Convert back to characters and XOR with trudeau

4. Reverse the resulting string to get the original message

Lessons Learned

This challenge was a great reminder that:

- Read the code carefully - small details like string reversal can trip you up

- Work backwards systematically- don't try to jump straight to the answer

- Test your assumptions - I initially missed the reversal and got garbage output

- Custom crypto is often just known techniques chained together - once you identify the individual pieces, it becomes much more manageable

The combination of Diffie-Hellman, XOR, and simple multiplication created a deceptively complex-looking cipher, but breaking it down step by step made it totally solvable. Pretty clever challenge design!

Python script-

def generator(g, x, p):

    """Generate g^x mod p"""

    return pow(g, x) % p

def decrypt_multiplication(cipher_list, key):

    """Reverse the multiplication encryption: divide each value by (key * 311)"""

    plaintext_chars = []

    for value in cipher_list:

        if value == 0:

            # Handle the case where original char was 0 (null character)

            plaintext_chars.append(chr(0))

        else:

            # Reverse: original_char = cipher_value / (key * 311)

            original_ord = value // (key * 311)

            plaintext_chars.append(chr(original_ord))

    return ''.join(plaintext_chars)

def dynamic_xor_decrypt(cipher_text, text_key):

    """Reverse the dynamic XOR encryption"""

    # The original function reversed the plaintext, so we need to account for that

    plaintext = ""

    key_length = len(text_key)

    

    # XOR decryption (XOR is its own inverse)

    for i, char in enumerate(cipher_text):

        key_char = text_key[i % key_length]

        decrypted_char = chr(ord(char) ^ ord(key_char))

        plaintext += decrypted_char

    

    # Reverse the string back to original order (since original was reversed)

    return plaintext[::-1]

def decrypt_flag():

    """Decrypt the flag using the provided values"""

    # Given values from enc_flag

    a = 94

    b = 21

    cipher = [131553, 993956, 964722, 1359381, 43851, 1169360, 950105, 321574, 1081658, 613914, 0, 1213211, 306957, 73085, 993956, 0, 321574, 1257062, 14617, 906254, 350808, 394659, 87702, 87702, 248489, 87702, 380042, 745467, 467744, 716233, 380042, 102319, 175404, 248489]

    

    # Constants from the original code

    p = 97

    g = 31

    text_key = "trudeau"

    

    # Reproduce the Diffie-Hellman key exchange

    u = generator(g, a, p)  # g^a mod p

    v = generator(g, b, p)  # g^b mod p

    

    # Calculate shared key

    shared_key = generator(v, a, p)  # (g^b)^a mod p = g^(ab) mod p

    # Verify: generator(u, b, p) should give the same result

    

    print(f"u = {u}")

    print(f"v = {v}")

    print(f"shared_key = {shared_key}")

    

    # Step 1: Reverse multiplication encryption

    semi_cipher_chars = decrypt_multiplication(cipher, shared_key)

    print(f"After reversing multiplication: {repr(semi_cipher_chars)}")

    

    # Step 2: Reverse dynamic XOR encryption

    original_message = dynamic_xor_decrypt(semi_cipher_chars, text_key)

    print(f"Decrypted message: {original_message}")

    

    return original_message

# Let's test the decryption

flag = decrypt_flag()

print(f"\nFinal flag: {flag}")

Final Flag-picoCTF{custom_d2cr0pt6d_8b41f976}

Final Thoughts

My favorite thing about this challenge was that it combined a number of cryptographic principles without becoming too complex. It was a matter of solving a puzzle where each piece had to fit with the others perfectly. The code was tidy enough to make sense, but intelligent enough to keep you guessing.

If you're facing similar challenges, my suggestion is to always step through the encryption process in detail, then construct your decryption by reversing each operation. And be sure to watch out for those subtle string manipulations!