Implement stack guard pages

[Martin Decky]

Implement stack guard pages to detect stack overflows as they happen.

Details

Stack guard pages are special unmapped pages in the virtual address space which trigger an exception whenever the code tries to access the memory within the guard page. This serves two main purposes:

• A guard page above and below thread's stack serves as a safety mechanism to detect stack overflows and underflows. Stack underflows usually happen when the code control flow is somehow corrupted, stack overflows usually happen when the size of the stack is not sufficient for the current level of function nesting and local variables allocation (many times this is a sign of unbounded recursion).
• A guard page can be also used to implement a stack which grows automatically as needed (which allows the initial stack size to be very small). However, to fully utilize automatically growing stacks in a multithreaded environment, the compiler and run-time environment are required to support discontinuous stacks. GCC has support for this on several platforms.

The guard pages mechanism can be generalized to also protect other (even all) memory areas within the address space.

What Gains and Benefits will this bring?

Undetected stack overflows and underflows can lead to silent memory corruption (usually corrupting the data on the stacks of other threads). This usually leads to application/kernel crash eventually, but the root cause of the crash is usually not obvious and debugging such cases is very tedious. Guard pages serve as an early detection mechanism which points the programmer directly to the source of the problem when it happens.

Guard pages in general are not a fail-safe security measure and they cannot detect complex stack corruption conditions (stack corruption within the dedicated stack area), but they are still a very helpful enhancement with almost no run-time overhead.

Difficulty

Medium

Required skills

A successful applicant will need to have very good skills in programming in the C language and the ability to learn and use the memory management subsystem of HelenOS.

Documentation

• DeveloperDocs
• ​Split Stacks in GCC

Possible mentors

HelenOS Core Team, Martin Decky