The Antivirus Hacker's Handbook

The Antivirus Hacker's Handbook guides you through the process of reverse engineering antivirus software. You explore how to detect and exploit vulnerabilities that can be leveraged to improve future software design, protect your network, and anticipate attacks that may sneak through your antivirus' line of defense. You'll begin building your knowledge by diving into the reverse engineering process, which details how to start from a finished antivirus software program and work your way back through its development using the functions and other key elements of the software. Next, you leverage your new knowledge about software development to evade, attack, and exploit antivirus software—all of which can help you strengthen your network and protect your data.

While not all viruses are damaging, understanding how to better protect your computer against them can help you maintain the integrity of your network.

• Discover how to reverse engineer your antivirus software
• Explore methods of antivirus software evasion
• Consider different ways to attack and exploit antivirus software
• Understand the current state of the antivirus software market, and get recommendations for users and vendors who are leveraging this software

The Antivirus Hacker's Handbook is the essential reference for software reverse engineers, penetration testers, security researchers, exploit writers, antivirus vendors, and software engineers who want to understand how to leverage current antivirus software to improve future applications.

Introduction xix

Part I Antivirus Basics 1

Chapter 1 Introduction to Antivirus Software 3

What Is Antivirus Software? 3

Antivirus Software: Past and Present 4

Antivirus Scanners, Kernels, and Products 5

Typical Misconceptions about Antivirus Software 6

Antivirus Features 7

Basic Features 7

Making Use of Native Languages 7

Scanners 8

Signatures 8

Compressors and Archives 9

Unpackers 10

Emulators 10

Miscellaneous File Formats 11

Advanced Features 11

Packet Filters and Firewalls 11

Self-Protection 12

Anti-Exploiting 12

Summary 13

Chapter 2 Reverse-Engineering the Core 15

Reverse-Engineering Tools 15

Command-Line Tools versus GUI Tools 16

Debugging Symbols 17

Tricks for Retrieving Debugging Symbols 17

Debugging Tricks 20

Backdoors and Configuration Settings 21

Kernel Debugging 23

Debugging User-Mode Processes with a Kernel-Mode Debugger 25

Analyzing AV Software with Command-Line Tools 27

Porting the Core 28

A Practical Example: Writing Basic Python Bindings for Avast for Linux 29

A Brief Look at Avast for Linux 29

Writing Simple Python Bindings for Avast for Linux 32

The Final Version of the Python Bindings 37

A Practical Example: Writing Native C/C++ Tools for Comodo Antivirus for Linux 37

Other Components Loaded by the Kernel 55

Summary 56

Chapter 3 The Plug-ins System 57

Understanding How Plug-ins Are Loaded 58

A Full-Featured Linker in Antivirus Software 58

Understanding Dynamic Loading 59

Advantages and Disadvantages of the Approaches for Packaging Plug-ins 60

Types of Plug-ins 62

Scanners and Generic Routines 63

File Format and Protocol Support 64

Heuristics 65

Bayesian Networks 66

Bloom Filters 67

Weights-Based Heuristics 68

Some Advanced Plug-ins 69

Memory Scanners 69

Non-native Code 70

Scripting Languages 72

Emulators 73

Summary 74

Chapter 4 Understanding Antivirus Signatures 77

Typical Signatures 77

Byte-Streams 78

Checksums 78

Custom Checksums 79

Cryptographic Hashes 80

Advanced Signatures 80

Fuzzy Hashing 81

Graph-Based Hashes for Executable Files 83

Summary 85

Chapter 5 The Update System 87

Understanding the Update Protocols 88

Support for SSL/TLS 89

Verifying the Update Files 91

Dissecting an Update Protocol 92

When Protection Is Done Wrong 100

Summary 101

Part II Antivirus Software Evasion 103

Chapter 6 Antivirus Software Evasion 105

Who Uses Antivirus Evasion Techniques? 106

Discovering Where and How Malware Is Detected 107

Old Tricks for Determining Where Malware Is Detected: Divide and Conquer 107

Evading a Simple Signature-Based Detection with the Divide and Conquer Trick 108

Binary Instrumentation and Taint Analysis 113

Summary 114

Chapter 7 Evading Signatures 117

File Formats: Corner Cases and Undocumented Cases 118

Evading a Real Signature 118

Evasion Tips and Tricks for Specific File Formats 124

PE Files 124

JavaScript 126

String Encoding 127

Executing Code on the Fly 128

Hiding the Logic: Opaque Predicates and Junk Code 128

PDF 129

Summary 131

Chapter 8 Evading Scanners 133

Generic Evasion Tips and Tricks 133

Fingerprinting Emulators 134

Advanced Evasion Tricks 136

Taking Advantage of File Format Weaknesses 136

Using Anti-emulation Techniques 137

Using Anti-disassembling Techniques 142

Disrupting Code Analyzers through Anti-analysis 144

More Anti-Anti-Anti… 147

Causing File Format Confusion 148

Automating Evasion of Scanners 148

Initial Steps 149

Installing ClamAV 150

Installing Avast 150

Installing AVG 151

Installing F-Prot 152

Installing Comodo 153

Installing Zoner Antivirus 154

MultiAV Configuration 154

peCloak 158

Writing the Final Tool 160

Summary 162

Chapter 9 Evading Heuristic Engines 165

Heuristic Engine Types 165

Static Heuristic Engines 166

Bypassing a Simplistic Static Heuristic Engine 166

Dynamic Heuristic Engines 173

Userland Hooks 173

Bypassing a Userland HIPS 176

Kernel-Land Hooks 178

Summary 180

Chapter 10 Identifying the Attack Surface 183

Understanding the Local Attack Surface 185

Finding Weaknesses in File and Directory Privileges 185

Escalation of Privileges 186

Incorrect Privileges in Files and Folders 186

Incorrect Access Control Lists 187

Kernel-Level Vulnerabilities 187

Exotic Bugs 188

Exploiting SUID and SGID Binaries on Unix-Based Platforms 189

ASLR and DEP Status for Programs and Binaries 190

Exploiting Incorrect Privileges on Windows Objects 193

Exploiting Logical Flaws 196

Understanding the Remote Attack Surface 197

File Parsers 198

Generic Detection and File Disinfection Code 199

Network Services, Administration Panels, and Consoles 199

Firewalls, Intrusion Detection Systems, and Their Parsers 200

Update Services 201

Browser Plug-ins 201

Security Enhanced Software 202

Summary 203

Chapter 11 Denial of Service 207

Local Denial-of-Service Attacks 208

Compression Bombs 208

Creating a Simple Compression Bomb 209

Bugs in File Format Parsers 212

Attacks against Kernel Drivers 213

Remote Denial-of-Service Attacks 214

Compression Bombs 214

Bugs in File Format Parsers 215

Summary 215

Part III Analysis and Exploitation 217

Chapter 12 Static Analysis 219

Performing a Manual Binary Audit 219

File Format Parsers 220

Remote Services 228

Summary 233

Chapter 13 Dynamic Analysis 235

Fuzzing 235

What Is a Fuzzer? 236

Simple Fuzzing 237

Automating Fuzzing of Antivirus Products 239

Using Command-Line Tools 240

Porting Antivirus Kernels to Unix 243

Fuzzing with Wine 244

Problems, Problems, and More Problems 247

Finding Good Templates 248

Finding Template Files 250

Maximizing Code Coverage 252

Blind Code Coverage Fuzzer 253

Using Blind Code Coverage Fuzzer 254

Nightmare, the Fuzzing Suite 259

Configuring Nightmare 260

Finding Samples 262

Configuring and Running the Fuzzer 262

Summary 266

Chapter 14 Local Exploitation 269

Exploiting Backdoors and Hidden Features 270

Finding Invalid Privileges, Permissions, and ACLs 274

Searching Kernel-Land for Hidden Features 279

More Logical Kernel Vulnerabilities 285

Summary 295

Chapter 15 Remote Exploitation 297

Implementing Client-Side Exploitation 297

Exploiting Weakness in Sandboxing 297

Exploiting ASLR, DEP, and RWX Pages at Fixed Addresses 298

Writing Complex Payloads 300

Taking Advantage of Emulators 301

Exploiting Archive Files 302

Finding Weaknesses in Intel x86, AMD x86_64, and ARM Emulators 303

Using JavaScript, VBScript, or ActionScript 303

Determining What an Antivirus Supports 304

Launching the Final Payload 306

Exploiting the Update Services 307

Writing an Exploit for an Update Service 308

Server-Side Exploitation 317

Differences between Client-Side and Server-Side Exploitation 317

Exploiting ASLR, DEP, and RWX Pages at Fixed Addresses 318

Summary 318

Part IV Current Trends and Recommendations 321

Chapter 16 Current Trends in Antivirus Protection 323

Matching the Attack Technique with the Target 324

The Diversity of Antivirus Products 324

Zero-Day Bugs 324

Patched Bugs 325

Targeting Home Users 325

Targeting Small to Medium-Sized Companies 326

Targeting Governments and Big Companies 326

The Targets of Governments 327

Summary 328

Chapter 17 Recommendations and the Possible Future 331

Recommendations for Users of Antivirus Products 331

Blind Trust Is a Mistake 332

Isolating Machines Improves Protection 337

Auditing Security Products 338

Recommendations for Antivirus Vendors 338

Engineering Is Different from Security 339

Exploiting Antivirus Software Is Trivial 339

Perform Audits 340

Fuzzing 340

Use Privileges Safely 341

Reduce Dangerous Code in Parsers 342

Improve the Safety of Update Services and Protocols 342

Remove or Disable Old Code 343

Summary 344

Index 347

JOXEAN KORET is a security researcher at Coseinc, a Singapore-based security services firm. He is an in-demand speaker at international security conferences, and a well-regarded specialist in reverse-engineering, vulnerability research, and malware analysis.

ELIAS BACHAALANY is a computer programmer, reverse engineer, and technical writer employed by Microsoft. An authority in multiple programming languages, database design, and web development, he played an instrumental role in the IDAPython project and IDA Pro's scripting facilities development.