Introduction: The Foundation of OS Memory Management

Building an operating system from scratch on x86 architecture is an intricate dance with hardware, and one of the most critical steps after initial boot is establishing a robust memory management scheme. The rudimentary 16-bit real mode environment offered by the BIOS is severely limiting, confining us to the first megabyte of memory. To unlock the full potential of modern systems, access gigabytes of RAM, and implement features like virtual memory and protection, we must transition to 32-bit (or 64-bit) protected mode and set up foundational structures like the Global Descriptor Table (GDT) and Paging. This article delves into these essential steps, providing a detailed guide for minimal x86 bootloader development.

Transitioning to Protected Mode: The GDT’s Role

Real Mode Constraints

Upon power-on, an x86 processor initializes in 16-bit real mode. In this mode, memory is addressed using a segment:offset scheme (e.g., `0x07C0:0x0000`). This legacy design restricts physical memory access to 1MB (0x00000 to 0xFFFFF), effectively locking out the vast majority of available RAM on modern machines. Furthermore, real mode lacks memory protection, making it unsuitable for robust multitasking operating systems.

The Global Descriptor Table (GDT)

The GDT is the cornerstone for entering protected mode. It is an array of 8-byte segment descriptors that define the characteristics of various memory segments in our system. Each descriptor specifies a segment’s base address, its limit (size), and access rights (e.g., read/write, execute, privilege level). When the CPU is in protected mode, all memory accesses are performed relative to a segment descriptor loaded from the GDT.

GDT Descriptor Structure

An 8-byte GDT descriptor is structured as follows:

• Limit (0-15): Lower 16 bits of segment limit.
• Base (0-15): Lower 16 bits of segment base address.
• Base (16-23): Middle 8 bits of segment base address.
• Type: 4 bits, specifies segment type (e.g., code, data, stack).
• S (Descriptor Type): 1 bit, 0 for system, 1 for code/data.
• DPL (Descriptor Privilege Level): 2 bits, 0=highest, 3=lowest.
• P (Present): 1 bit, 1 if segment is present in memory.
• Limit (16-19): Upper 4 bits of segment limit.
• AVL (Available): 1 bit, available for use by software.
• L (64-bit Code Segment): 1 bit, 1 for 64-bit code, 0 otherwise.
• D/B (Operand Size/Big): 1 bit, 1 for 32-bit operands/stack, 0 for 16-bit.
• G (Granularity): 1 bit, 0 for byte, 1 for 4KB page granularity.
• Base (24-31): Upper 8 bits of segment base address.

Setting up the GDT

Typically, we’ll define at least three descriptors: a null descriptor (required by convention), a code segment, and a data segment. These segments will often span the entire 4GB address space with a base of 0, allowing for a flat memory model initially. The `LGDT` instruction loads the GDT register (`GDTR`) with the 48-bit address and size of our GDT.

; GDT definitions in assembly (NASM syntax)GDT_START:  NULL_DESCRIPTOR:  dd 0x0                  ; Null Descriptor  dd 0x0CODE_DESCRIPTOR:  dw 0xFFFF                 ; Segment Limit (0-15)  dw 0x0                    ; Base Address (0-15)  db 0x0                    ; Base Address (16-23)  db 10011010b            ; Access Byte (Present, Priv 0, Code, Read/Exec)  db 11001111b            ; Flags (Granularity 4KB, 32-bit, Limit 16-19)  db 0x0                    ; Base Address (24-31)DATA_DESCRIPTOR:  dw 0xFFFF                 ; Segment Limit (0-15)  dw 0x0                    ; Base Address (0-15)  db 0x0                    ; Base Address (16-23)  db 10010010b            ; Access Byte (Present, Priv 0, Data, Read/Write)  db 11001111b            ; Flags (Granularity 4KB, 32-bit, Limit 16-19)  db 0x0                    ; Base Address (24-31)GDT_END:GDT_POINTER:  dw GDT_END - GDT_START - 1 ; GDT Size (limit)  dd GDT_START               ; GDT Base Address; In a bootloader section (after GDT definition):; Load the GDTlgdt [GDT_POINTER]

Entering Protected Mode

Enabling A20 Line

Before entering protected mode, the A20 line must be enabled. This allows the CPU to access memory above 1MB. Historically, this was managed by the keyboard controller. Modern systems typically handle this automatically or provide faster methods (e.g., using fast A20 gate register at port 0x92).

; A20 Gate Enable (via keyboard controller, common legacy method); Wait for input buffer to be emptymov dx, 0x64test al, 0x2; wait for keyboard controller to be ready.in al, dxjz $-2; Send 'write output port' command to keyboard controller (0x64)mov al, 0xD1out dx, al; Wait for input buffer to be emptytest al, 0x2in al, dxjz $-2; Send 'enable A20' command (0xDF) to keyboard controller (0x60)mov al, 0xDFout 0x60, al

Setting the PE Bit

The protected mode enable (PE) bit is the least significant bit of the CR0 control register. Setting this bit flips the CPU from real mode to protected mode.

; Set PE bit in CR0mov eax, cr0or eax, 0x1mov cr0, eax

Far Jump into 32-bit Code

After enabling protected mode, a far jump is necessary to flush the instruction pipeline and load new segment selectors into the segment registers. This also ensures the CPU begins executing code with the new GDT-based segment definitions.

; Perform a far jump to clear the prefetch queue and load new segment registers; CODE_SEG is the selector for our code descriptor (e.g., 0x8 if null is 0x0)jmp 0x8:protected_mode_start

protected_mode_start:  ; Now in 32-bit protected mode  mov ax, 0x10         ; DATA_SEG selector  mov ds, ax  mov es, ax  mov fs, ax  mov gs, ax  mov ss, ax  ; Setup stack pointer  mov esp, 0x90000     ; Example stack address

Detecting and Initializing RAM

BIOS Memory Map (INT 15h, EAX=E820h)

Once in protected mode, we can use BIOS interrupt 15h, function EAX=E820h, to obtain a detailed memory map from the BIOS. This function returns an array of memory descriptors, each specifying a base address, length, and type (e.g., usable RAM, reserved, ACPI reclaimable). We need to iterate through this BIOS call until it indicates no more entries are available.

; Pseudo-code for E820 memory map detection; This operation is typically done in real mode before entering protected mode,; or directly in protected mode if BIOS allows calls via specific mechanisms.; For simplicity, assuming a real-mode execution context for the BIOS call.; Initialize a buffer to store the memory mapentries_count = 0pointer_to_e820_buffer = 0x8000 ; A buffer in low memorye820_loop:  mov eax, 0xE820  mov ebx, 0        ; Continuation value, 0 for first call  mov ecx, 20       ; Size of descriptor structure  mov edx, 0x534D4150 ; 'SMAP' magic number  mov edi, pointer_to_e820_buffer ; Buffer to store result  int 0x15          ; Call BIOS service  jc e820_error     ; Carry flag set on error  cmp eax, 0x534D4150 ; Check 'SMAP' magic again  jne e820_error  test ebx, ebx   ; Check EBX for continuation. If 0, no more entries.  jz e820_done  inc entries_count  add pointer_to_e820_buffer, 20 ; Move to next entry slot  jmp e820_loope820_done:  ; E820 memory map is now stored in pointer_to_e820_buffer

Basic RAM Initialization

After detecting usable RAM regions, it’s good practice to initialize them, typically by zeroing out the memory. This ensures a clean slate, preventing unexpected data from influencing later kernel operations or introducing security vulnerabilities.

; Example: Zeroing out a memory region (e.g., 4MB from 1MB to 5MB); In 32-bit protected mode, assume general purpose registers are availablemov edi, 0x100000 ; Start address (1MB)mov ecx, 0x400000 ; Size to clear (4MB)xor eax, eax      ; Value to write (0)rep stosd         ; Fill memory with EAX (0)

Paging: Enabling Virtual Memory

Why Paging?

Paging is a memory management scheme that allows the operating system to map virtual memory addresses to physical memory addresses. This offers several crucial benefits:

• Virtual Memory: Programs see a contiguous, isolated address space, simplifying development.
• Memory Protection: Prevents programs from accessing each other’s memory or kernel space.
• Demand Paging: Allows loading only necessary parts of a program into physical RAM.
• Swapping: Enables storing less-used memory pages on disk.

Page Directory and Page Table Structure

On x86, paging uses a two-level hierarchy (for 32-bit systems) to translate virtual addresses to physical addresses, involving a Page Directory and multiple Page Tables. Both are typically 4KB aligned.

• Page Directory (PD): A 4KB table containing 1024 4-byte Page Directory Entries (PDEs). Each PDE points to a Page Table.
• Page Table (PT): A 4KB table containing 1024 4-byte Page Table Entries (PTEs). Each PTE points to a 4KB physical page of RAM.

A 32-bit virtual address is split: 10 bits for Page Directory index, 10 bits for Page Table index, and 12 bits for the offset within the 4KB page.

Identity Paging

For initial bootloader stages, a common approach is identity paging, where virtual addresses directly map to their corresponding physical addresses (e.g., virtual address `0xC0000000` maps to physical address `0xC0000000`). This simplifies the transition and allows the kernel to operate with familiar physical addresses before setting up more complex virtual memory schemes.

Setting up Page Tables

The process involves creating a Page Directory, then creating Page Tables for the regions we want to map. For identity mapping, the PTEs will simply contain the physical address of the corresponding 4KB page with appropriate flags.

; Pseudo-code for setting up identity paging for the first 4MB; Assume 'page_directory' and 'page_table_0' are 4KB-aligned physical addresses; Clear the Page Directory and Page Table for first 4MBmemset(page_directory, 0, 4096);memset(page_table_0, 0, 4096);; Set the first PDE in the Page Directory to point to our first Page Tablepage_directory[0] = page_table_0_physical_address | PRESENT_BIT | READ_WRITE_BIT | USER_SUPERVISOR_BIT;; Populate the first Page Table (mapping first 4MB identity)for (i = 0; i < 1024; i++) {  page_table_0[i] = (i * 4096) | PRESENT_BIT | READ_WRITE_BIT | USER_SUPERVISOR_BIT;}

Enabling Paging

To enable paging, the physical address of the Page Directory must be loaded into the CR3 register. Then, the PG bit (bit 31) in CR0 is set.

; Load the Page Directory's physical address into CR3mov eax, page_directory_physical_addressmov cr3, eax; Enable paging by setting the PG bit (bit 31) in CR0mov eax, cr0or eax, 0x80000000mov cr0, eax; At this point, paging is active. Subsequent memory accesses are translated.

Putting It All Together: The Boot Sequence

The sequence of operations for establishing memory management in a minimal x86 bootloader typically flows as follows:

1. Real Mode Setup: Initial bootloader code (usually from `0x7C00`).
2. A20 Line Enable: Allows access to memory above 1MB.
3. GDT Definition and Load: Set up the GDT in memory and load its address into `GDTR` using `LGDT`.
4. Protected Mode Entry: Set the PE bit in `CR0`, then perform a far jump to flush the pipeline and load new segment selectors.
5. Memory Detection: Use BIOS INT 15h, EAX=E820h (or equivalent) to map available physical RAM.
6. RAM Initialization: Zero out detected usable memory regions.
7. Page Directory and Page Tables Creation: Allocate and populate the Page Directory and necessary Page Tables (e.g., for identity mapping).
8. Paging Enable: Load the Page Directory physical address into `CR3` and set the PG bit in `CR0`.
9. Higher-Half Kernel Jump: (Future Step) With paging enabled, the bootloader can then jump to a higher-half virtual address where the main kernel code resides, giving it a consistent and protected address space from the start.