Malware Analysis

Malware Reverse Engineering — The Art of Unpacking the Beast

---

Malware reverse engineering (MRE) is the forensic dissection of malicious binaries to uncover what they do, how they do it, and how to stop (or repurpose) them. From ransomware unpacking to APT loader tracing, this is where binary analysis meets threat intelligence.

This guide covers static and dynamic analysis, behavior mapping, anti-analysis evasion, and memory forensics — built for both CTF reverse challenges and real-world malware teardown.

---

I. 🧩 Core Malware Concepts

Concept	Description
Dropper	Initial loader delivering payloads.
Payload	Actual malicious code (stealer, RAT, ransomware).
Persistence	Mechanisms ensuring malware re-executes after reboot.
C2 (Command & Control)	Remote infrastructure controlling malware.
Packing / Obfuscation	Techniques hiding or encrypting code.
Injection	Injecting payloads into legitimate processes (e.g., explorer.exe).
Sandbox Evasion	Detecting or escaping automated analysis.

---

II. ⚙️ Analysis Environment Setup

🧠 Golden Rule: Never analyze on your main OS.

⚙️ 1. Safe VM Setup

• Host OS: Linux or Windows with isolation.

• VM: Windows 10/11 + Snapshots.

• Disable networking, or use INetSim for fake internet simulation.

⚙️ 2. Essential Tools

Category	Tools
Static	Ghidra, IDA, DIE, PEiD
Dynamic	x64dbg, Process Monitor, Procmon, Process Hacker
Memory	Volatility3, Rekall
Network	Wireshark, Fakenet-NG, INetSim
Strings/Entropy	FLOSS, binwalk, Detect-It-Easy
Behavior	Any.Run, CAPEv2, Cuckoo Sandbox

---

III. ⚙️ Static Analysis — Pre-Execution Dissection

🧠 1. Identify File Type

file sample.exe
pefile sample.exe

Check if PE, ELF, or Mach-O.

⚙️ 2. Check Packing / Encryption

Detect-It-Easy sample.exe

or

upx -t sample.exe

If UPX packed:

upx -d sample.exe

🧩 3. Extract Strings

floss sample.exe > strings.txt
grep -E "http|cmd|powershell|key|flag" strings.txt

---

⚙️ 4. Check Imports / APIs

peframe sample.exe

or manually:

objdump -p sample.exe | grep -i import

Look for suspicious APIs:

CreateRemoteThread
VirtualAllocEx
InternetConnect
WriteProcessMemory

---

🧠 5. Analyze Sections

readelf -S malware

Suspicious indicators:

• Unusual entropy in .text

• Code in .data or .rdata

• Overwritten headers

---

⚙️ 6. Use Ghidra / IDA

Follow logic flow:

• WinMain(), _start, or main()

• API imports: LoadLibraryA, GetProcAddress

• Hardcoded strings: URLs, registry keys, file paths

• Look for decryption or unpacking routines (loops or XORs)

---

IV. ⚙️ Dynamic Analysis — Behavior in Action

🧩 1. Run Under Monitor

procmon.exe

Track:

• File system modifications

• Registry writes

• Process creation

• Network connections

⚙️ 2. Check Network Behavior

fakenet-ng
wireshark

Monitor:

• HTTP POST to unknown IPs

• DNS requests to suspicious domains

• Encrypted outbound traffic

---

🧠 3. API Call Tracing

API Monitor
Process Hacker
x64dbg

Watch for:

• CreateProcess (spawning payload)

• RegSetValueEx (persistence)

• OpenProcess + WriteProcessMemory (injection)

---

⚙️ 4. Debug Step-by-Step

x64dbg sample.exe

Breakpoints:

• CreateProcessA

• WinExec

• InternetConnect

• WriteFile

Step through to watch decryption and unpacking routines.

---

V. 💣 Unpacking Malware

🧩 1. Detect the Packer

• UPX → simple unpack

• Themida / VMProtect → advanced

• Custom XOR → manual reversing

⚙️ 2. Dump Unpacked Memory

x64dbg → Dump memory → Rebuild PE with Scylla

💣 3. Automate with UnpacMe / CAPE

Upload → auto-unpack → retrieve clean binary.

---

VI. ⚙️ Malware Behavior Mapping

🧠 1. File System Indicators

• %AppData%\Microsoft\Windows\Start Menu\Programs\Startup\

• %TEMP%\random.exe

• C:\Users\<user>\AppData\Roaming\

⚙️ 2. Registry Persistence

reg query HKCU\Software\Microsoft\Windows\CurrentVersion\Run

🧩 3. Scheduled Tasks

schtasks /query /fo LIST /v

---

VII. ⚙️ Code Injection & Process Hollowing

Technique	Description	API Indicators
Classic Injection	Inject shellcode into remote process	OpenProcess, WriteProcessMemory, CreateRemoteThread
Process Hollowing	Replace memory of a legitimate process	NtUnmapViewOfSection, ZwWriteVirtualMemory
Reflective Loading	Load DLL from memory	LoadLibrary, VirtualAlloc
APC Injection	Queue APCs in target thread	QueueUserAPC

Use x64dbg or Procmon to visualize injected threads and memory regions.

---

VIII. 🧠 Network & C2 Analysis

⚙️ 1. Capture Communication

wireshark -i 1 -f "tcp port 80 or 443"

Look for:

• POST requests with encoded data

• Domains resembling legit names (e.g. micros0ft-update.net)

⚙️ 2. Fake C2 Interaction

Use INetSim to simulate network services:

inetsim --service http --service dns

Observe malware’s connection attempts.

---

IX. ⚙️ Memory Forensics

🧠 1. Capture RAM

winpmem.exe --output mem.raw

⚙️ 2. Analyze with Volatility

volatility3 -f mem.raw windows.pslist
volatility3 -f mem.raw windows.malfind
volatility3 -f mem.raw windows.cmdline

Use malfind to detect injected code and hidden modules.

---

X. ⚙️ Deobfuscation & Decryption

🧠 1. Identify XOR / RC4 Loops

Look for patterns like:

xor eax, 0x13
add edi, 1

or

for (i=0;i<len;i++) buf[i]^=key;

⚙️ 2. Recreate Algorithm

Use Python to decrypt:

data = [0x45, 0x47, 0x50]
key = 0x13
print(''.join([chr(b ^ key) for b in data]))

💣 3. Dynamic Dump Decrypted Data

Hook memory using:

frida-trace -i "VirtualAlloc" -i "WriteFile" sample.exe

---

XI. ⚙️ Common Malware Techniques

Technique	Description
Process Hollowing	Inject payload into legitimate process
Persistence via Registry	Run malware on boot
Keylogging	Capture keystrokes via GetAsyncKeyState
Credential Dumping	Access lsass.exe memory
Screen Capture	Using BitBlt and CreateCompatibleBitmap
Exfiltration	Sending stolen data via HTTP/DNS
C2 Obfuscation	Encrypting communications (base64, XOR)

---

XII. ⚙️ Evasion & Anti-Analysis Tricks

Behavior	Indicator	Bypass
Anti-VM	Checks for VMware registry keys	Patch or emulate registry
Anti-Debug	Calls to IsDebuggerPresent	Patch to return false
Sleep Delays	Long Sleep() calls	Hook and skip
Environment Checks	Looking for process names like procmon.exe	Rename tools
Encrypted APIs	Dynamic import resolution	Break at LoadLibraryA

---

XIII. ⚙️ Behavioral Sandboxing

🧠 1. CAPEv2 Sandbox

Automatically:

• Unpacks

• Monitors API calls

• Generates YARA signatures

• Produces network PCAPs

⚙️ 2. Any.Run (Interactive)

Upload sample → run → interact → export behavior trace.

---

XIV. ⚙️ YARA & Signature Creation

Create custom detection rules:

rule EvilMalware {
    strings:
        $s1 = "VirtualAllocEx"
        $s2 = "cmd.exe /c"
        $s3 = "POST /data"
    condition:
        all of them
}

Scan:

yara -r EvilMalware.yar samples/

---

XV. ⚙️ Incident Response Integration

Artifact	Description
Memory Dump	Extract injected payloads
Disk Image	Locate persistence
PCAP	Analyze exfiltration
Registry Hive	Startup entries
Logs	Process creation timestamps

Integrate with ELK/Splunk or MISP for threat intel correlation.

---

XVI. ⚔️ Pro Tips & Operator Tricks

✅ Run in Layers: Start static → trace behavior → dump memory → rebuild PE → analyze logic.

✅ Snapshots Are Life: Revert VM after each execution to reset artifacts.

✅ Frida = Power: Hook live API calls to dump decrypted payloads mid-runtime.

✅ Network Sim = Gold: Fake all internet services with INetSim — you’ll see what domains it tries to reach.

✅ Look for Sleep Skips: Replace Sleep(600000) with Sleep(0) to save time.

✅ Automate Your Analysis: Use CAPEv2 to generate reports + YARA rules automatically.

✅ Check For Hidden DLLs: Injected or reflective DLLs won’t show up in imports — check malfind.

---

XVII. ⚙️ Quick Reference Table

Goal	Tool / Command	Description
Identify Malware	Detect-It-Easy	Detect packers, compiler
Extract Strings	floss sample.exe	Decode obfuscated strings
Monitor Behavior	procmon, fakenet-ng	File/network trace
Disassemble	ghidra, ida	Inspect logic flow
Dump Memory	x64dbg + Scylla	Dump unpacked code
Analyze RAM	volatility3	Extract hidden processes
Detect C2	wireshark	Inspect network traffic
Sandbox	cape, any.run	Automated behavioral analysis
Create YARA	yara EvilMalware.yar samples/	Detection rule matching

---