Binary memory protection measures on Windows OS

Why is binary memory protection important?

You may remember when the Blaster worm struck the internet, or more recently when WannaCry caused global havoc using a leaked EternalBlue Windows OS exploit. Both are examples of malware that used buffer overflow memory corruption vulnerabilities, causing remote code execution and infecting millions of machines worldwide.

Most operating systems, written in C or C++, have limited memory protection, allowing these attacks to occur. Malware like Blaster and WannaCry manipulate the environment, instructions, and memory layout of a program or operating system to gain control over it.

Security professionals have implemented mechanisms to prevent software exploitation and minimize damage caused by memory corruption bugs. A "silver bullet" solution would be a mechanism that makes it challenging and unreliable for attackers to exploit vulnerabilities, allowing developers to leave buggy code in place while they work on fixing or rewriting it in memory-safe languages.

Let’s review some of the most common mechanisms and protection measures provided inside Windows OS from Windows XP to Windows 11.

ASLR

Address space layout randomization (ASLR) is a computer security technique that prevents an attacker from reliably jumping to, for example, a particular exploited function in a program’s memory. ASLR randomly arranges the address space positions of a process’s key data areas, including the base of the executable and the positions of the stack, heap, and libraries. The effectiveness of ASLR depends on the entropy of the process’s address space (simply put, the probability of finding a random local variable).

Because of this protection, exploit payloads must be uniquely tailored to a specific process address space.

Vista and Windows Server 2008 were the first operating systems in the Windows family to provide ASLR natively, though this system was first developed back in 2001. Prior to these releases, there were several third-party solutions like WehnTrust available that provided ASLR functionality to varying degrees.

When Symantec conducted research on ASLR in Windows Vista, they found that ASLR had a significant effect when implemented in Windows 8 (or Windows 8.1). It provided higher entropy for address space layouts. The larger address space for 64-bit processes also increased the entropy of the ASLR for any given process.

Windows 8 added randomization for all BottomUp and TopDown memory allocations, increasing the effectiveness of ASLR, which was not available in Windows 7.

In Windows 8, Microsoft introduced operating system support to force EXEs/DLLs to be rebased at runtime if they did not opt-in to ASLR. This mitigation can be enabled system-wide or on a per-process basis. You can modify the settings of mandatory ASLR through the Windows Security app.

ASLR, like any other security technique, has its weaknesses and attack vectors (heap spray, offset2libc, Jump Over ASLR, and others). Even one memory disclosure can completely defeat ASLR and provide an attacker with a significant opportunity. In addition to this, ASLR is only efficient when all executables and shared libraries loaded in the address space of a process are randomized. For example, research by Trend Micro researchers showed that Microsoft Edge browser exploit mitigations, including ASLR, could be bypassed. You can watch a video from the BlackHat conference to learn more.

DEP

Data Execution Prevention (DEP) is a protection mechanism that blocks the execution of code in memory pages marked non-executable. The NX (No-Execute) bit is a protection feature on CPUs used by DEP to prevent attackers from executing shellcode (instructions injected and executed by attackers) on the stack, heap, or in data sections. If DEP is enabled and a program attempts to execute code on a non-executable page, an access violation exception will be triggered.

Starting with Windows XP Service Pack 2 (2004) and Windows Server 2003 Service Pack 1 (2005), the DEP was implemented for the first time on x86 architecture.

An application can be compiled with the /NXCOMPAT flag to enable DEP for that application. You can also use editbin.exe /NXCOMPAT over a .exe file to enable it on a previously compiled file.

On 64-bit versions of Windows, DEP is always turned on for 64-bit processes and cannot be disabled. Windows also implemented software DEP (without the use of the NX bit) through Microsoft’s "Safe Structured Exception Handling" (SafeSEH), which I will talk about a bit later.

Despite being a useful protection measure, the NX bit can be bypassed. This leaves us unable to execute instructions placed on the stack, but still able to control the execution flow of the application. This is where the ROP (Return Oriented Programming) technique becomes relevant.

GS (Stack Canaries)

Stack canaries are a security feature that helps protect against binary exploits. They are random values that are generated every time a program is run. When placed in certain locations, they can be used to detect stack corruption. The /GS compiler option, when specified, causes the compiler to store a random value on the stack between the local variables and the return address of a function. According to Microsoft, these application elements will be protected:

• Any array (regardless of length or element size)
• Structs (regardless of their contents)

In a typical buffer overflow attack, the attacker’s data is used to try to overwrite the saved EIP (Extended Instruction Pointer) on the stack. However, before this can happen, the cookie is also overwritten, rendering the exploit ineffective (though it may still cause a denial of service). If the function epilogue detects the altered cookie and the application terminates.

The second important protection mechanism of /GS is variable reordering. To prevent attackers from overwriting local variables or arguments used by the function, the compiler will rearrange the layout of the stack frame and will put string buffers at a higher address than all other variables. So when a string buffer overflow occurs, it cannot overwrite any other local variables.

It was introduced with the release of Visual Studio 2003. Two years later, they enabled it by default with the release of Visual Studio 2005.

However, this protection measure is also not bullet-proof, since the attacker can either try to read the canary value from the memory or brute force the value. By using these two techniques, attackers can acquire the canary value, place it into the payload, and successfully redirect program flow or corrupt important program data.

CFG/XFG

Control Flow Guard (CFG) is a highly-optimized platform security feature that was created to combat memory corruption vulnerabilities. Placing tight restrictions on where an application can execute code makes it much harder for exploits to execute arbitrary code through vulnerabilities such as buffer overflows.

CFG creates a per-process bitmap, where a set bit indicates that the address is a valid destination. Before performing each indirect function call, the application checks if the destination address is in the bitmap. If the destination address is not in the bitmap, the program terminates.

Microsoft has enabled a new mechanism by default in Windows 11, Windows 10, and in Windows 8.1 Update 3. Developers can now add CFG to their programs by adding the /guard:cf linker flag before program linking in Visual Studio 2015 or newer. As of the Windows 11 and 10 Creators Update (Windows 11 and 10 version 1703), the Windows kernel is compiled with CFG.

To enhance CFG (Control Flow Guard), Microsoft introduced Xtended Control Flow Guard (XFG). By design, CFG only checks if functions are included in the CFG bitmap, which means that technically if a function pointer is overwritten with another function that exists in the bitmap, it would be considered a valid target.

XFG addresses this issue by creating a ~55-bit hash of the function prototype (consisting of the return value and function arguments) and placing it 8 bytes above the function itself when the dispatch function is called. This hash is used as an additional verification before transferring the control flow.

Getting back to the CFG, there are multiple techniques to bypass it. For example, you can set the destination to code located in a non-CFG module loaded in the same process, or find an indirect call that was not protected by CFG.

SafeSEH

SafeSEH is an exception handler. An exception handler is a programming construct used to provide a structured way of handling both system and application-level error conditions. Commonly they will look something like the code sample below:

Windows supplies a default exception handler when an application has no exception handlers applicable to the associated error condition. When the Windows exception handler is called, the application will be terminated.

Exception handlers are stored in the format of a linked list with the final element being the Windows default exception handler. This is represented by a pointer with the value 0xFFFFFFFF. Elements in the SEH chain before the Windows default exception handler are the exception handlers defined by the application.

If an attacker can overwrite a pointer to a handler and then cause an exception, they might be able to get control of the program.

SafeSEH is a security mechanism introduced with Visual Studio 2003. It works by adding a static list of good exception handlers in the PE file at the timing of compiling. Before executing an exception handler, it is checked against the table. Execution is passed to the handler only if it matches an entry in the table. SafeSEH only exists in 32-bit applications because 64-bit exception handlers are not stored on the stack. By default, they build a list of valid exception handlers and store it in the file’s PE header.

Preventing SEH exploits in most applications can be achieved by specifying the /SAFESEH compiler switch. When /SAFESEH is specified, the linker will also produce a table of the image’s safe exception handlers. This table specifies for the operating system which exception handlers are valid for the image, removing the ability to overwrite them with arbitrary values. If you want to see how this mitigation technique can be bypassed in real-life, this blog post from RCE Security offers more useful information.

Conclusion

Memory corruption vulnerabilities have plagued software for decades. As mentioned in the beginning, there are multiple mitigation techniques to prevent software exploitation and minimize damage caused by memory corruption bugs. However, those protections are definitely not a “silver bullet” solution for all memory corruption vulnerabilities.

For the developer, this means that no one should not blindly rely on the OS-provided protections. Instead, try to propagate secure coding practices and integrate security toolings like fuzzers and static code analyzers.

Lastly, move to memory-safe languages like Rust, if possible. For the attackers, even if the target application has all available mitigation measures, there may still be ways to bypass those protections.