Introduction: Why Protected Mode?

When crafting a custom operating system or bootloader for x86 architectures, one of the most critical early steps is transitioning the CPU from Real Mode to Protected Mode. Upon reset, an x86 processor starts in Real Mode, a compatibility mode that emulates the behavior of the original Intel 8086 processor. This means it’s limited to 16-bit registers, can only address 1MB of memory, and lacks crucial features like memory protection and virtual memory. These limitations make Real Mode unsuitable for modern operating systems.

Protected Mode, introduced with the Intel 80286 and significantly enhanced with the 80386, unlocks the full power of the x86 architecture. It provides access to a 4GB (or more, with PAE) linear address space, 32-bit (and later 64-bit) registers, sophisticated memory management (segmentation and paging), and hardware-enforced protection mechanisms vital for multitasking and system stability. Our goal in this article is to meticulously guide you through the assembly-level steps required to make this fundamental switch.

Prerequisites for Protected Mode

1. The A20 Line

Before entering Protected Mode, you must enable the A20 line. The A20 line is the 21st address line of the CPU (A0-A19 are the first 20). In Real Mode, addresses greater than 1MB would ‘wrap around’ due to the 8086’s 20-bit address bus. For example, 0x100000 (1MB) would be treated as 0x00000. Enabling A20 ensures that all 32 address lines (up to 4GB) are active, preventing this address wrapping and allowing access to memory above 1MB.

The most common method to enable A20 is through the keyboard controller (8042 chip) at I/O ports 0x60 and 0x64. This involves sending specific commands to the controller.

; A20 enabling routine via Keyboard Controller (Port 0x64/0x60)
enable_a20:
    ; Wait for input buffer to be empty
    in  al, 0x64        ; Read status port
    test al, 0x02       ; Test bit 1 (input buffer status)
    jnz enable_a20      ; If not empty, loop

    ; Send 'write output port' command
    mov al, 0xD1
    out 0x64, al

    ; Wait for input buffer to be empty again
    in  al, 0x64
    test al, 0x02
    jnz enable_a20

    ; Send data to output port (enable A20, bit 1)
    mov al, 0xDF        ; 0xDF = 11011111b (set bit 1)
    out 0x60, al

    ; Wait for output buffer to be empty
    in  al, 0x64
    test al, 0x01
    jz enable_a20

    ; Read output buffer (dummy read to clear it)
    in  al, 0x60
    ret

2. The Global Descriptor Table (GDT)

In Protected Mode, memory access is controlled by segment descriptors stored in a Global Descriptor Table (GDT). Instead of directly using segment registers as base addresses (like in Real Mode), they now hold selectors, which are 16-bit indices into the GDT. Each GDT entry (descriptor) defines a memory segment, specifying its base address, limit (size), and access rights (e.g., readable, writable, executable, privilege level).

For a minimal bootloader, we typically need at least three descriptors:

1. Null Descriptor: Always the first entry, required by hardware.
2. Code Segment Descriptor: Defines the segment where our 32-bit code resides. This usually covers the entire 4GB address space for simplicity in early boot.
3. Data Segment Descriptor: Defines the segment for data, stack, etc. This also typically spans 4GB.

Each descriptor is 8 bytes long. Here’s how you might define a minimal GDT in assembly:

; GDT structure definitions
; Format: Base(31:24), Flags(23:20), Limit(19:16), Type(15:8), Base(7:0)

gdt_start:
    ; 0x00: Null Descriptor (required)
    dq 0x0

    ; 0x08: Code Segment Descriptor (selector 0x08)
    ; Base = 0, Limit = 0xFFFFF (4GB), R/W, Executable, Present, 32-bit, Granularity 4KB
    dw 0xFFFF         ; Limit (bits 15-0)
    dw 0x0000         ; Base (bits 15-0)
    db 0x00           ; Base (bits 23-16)
    db 10011010b      ; Access Byte (P=1, DPL=00, S=1, Type=1010b = R/X)
    db 11001111b      ; Flags (G=1, D=1, L=0, AVL=0) + Limit (bits 19-16)
    db 0x00           ; Base (bits 31-24)

    ; 0x10: Data Segment Descriptor (selector 0x10)
    ; Base = 0, Limit = 0xFFFFF (4GB), R/W, Present, 32-bit, Granularity 4KB
    dw 0xFFFF         ; Limit (bits 15-0)
    dw 0x0000         ; Base (bits 15-0)
    db 0x00           ; Base (bits 23-16)
    db 10010010b      ; Access Byte (P=1, DPL=00, S=1, Type=0010b = R/W)
    db 11001111b      ; Flags (G=1, D=1, L=0, AVL=0) + Limit (bits 19-16)
    db 0x00           ; Base (bits 31-24)

gdt_end:

; GDTR (GDT Register) structure: Limit (2 bytes) | Base Address (4 bytes)
gdt_descriptor:
    dw gdt_end - gdt_start - 1 ; GDT Limit (size - 1)
    dd gdt_start               ; GDT Base Address

The GDT is loaded into the CPU’s GDTR (Global Descriptor Table Register) using the `LGDT` instruction. The `GDTR` holds the 32-bit linear base address and the 16-bit limit (size – 1) of the GDT.

The Transition Steps

1. Disable Interrupts

Before making any drastic changes to the CPU’s operating mode, it’s wise to disable interrupts using the `CLI` (Clear Interrupt Flag) instruction. This prevents any unexpected hardware interrupts (like a timer tick) from occurring while the CPU is in an inconsistent state during the mode switch.

2. Enable A20 Line

Execute the `enable_a20` routine we defined earlier. This ensures that memory above 1MB is addressable once in Protected Mode.

3. Load the GDT

Use the `LGDT` instruction with the address of your `gdt_descriptor` to inform the CPU about the location and size of your GDT.

    cli                     ; Disable interrupts
    call enable_a20         ; Enable A20
    lgdt [gdt_descriptor]   ; Load GDT

4. Set the Protection Enable (PE) Bit in CR0

This is the moment of truth. Setting the least significant bit (bit 0) of the CR0 (Control Register 0) register to 1 is what officially switches the CPU into Protected Mode.

    mov eax, cr0            ; Move CR0 contents to EAX
    or eax, 0x1             ; Set the Protection Enable (PE) bit (bit 0)
    mov cr0, eax            ; Move EAX back to CR0, enabling Protected Mode

5. Flush the Instruction Pre-fetch Queue

After enabling Protected Mode, the CPU’s instruction pipeline might still contain 16-bit instructions fetched while in Real Mode. To ensure that all subsequent instructions are interpreted in 32-bit Protected Mode, a far jump (a jump that specifies both a segment and an offset) is required. This effectively flushes the pipeline and causes the CPU to fetch new instructions using Protected Mode’s segmentation rules.

The `JMP` instruction below uses `0x08` as the code segment selector (which points to our 32-bit code segment in the GDT) and `protected_mode_entry` as the 32-bit offset.

    jmp 0x08:protected_mode_entry ; Far jump to the 32-bit code segment

align 4
protected_mode_entry:
    ; We are now in 32-bit Protected Mode!

6. Reload Segment Registers

Immediately after the far jump, only the `CS` (Code Segment) register is updated with a Protected Mode selector. The other segment registers (`DS`, `ES`, `SS`, `FS`, `GS`) still hold their old Real Mode values, which are now meaningless or incorrect in Protected Mode. They must be explicitly reloaded with a Protected Mode data segment selector (in our case, `0x10`).

    mov ax, 0x10    ; Load data segment selector
    mov ds, ax      ; Set Data Segment
    mov es, ax      ; Set Extra Segment
    mov fs, ax      ; Set F Segment
    mov gs, ax      ; Set G Segment
    mov ss, ax      ; Set Stack Segment

Post-Transition: What’s Next?

Congratulations! Your CPU is now operating in 32-bit Protected Mode. From this point forward, you can use 32-bit instructions and access the full 4GB linear address space. You’re ready to set up your stack, initialize memory, potentially enable paging, and finally jump to a higher-level language kernel (e.g., C or C++).

    ; Example: A simple 32-bit instruction
    mov ebx, 0xB8000 ; Video memory base address
    mov ecx, 0x0741  ; White 'A' character
    mov [ebx], ecx   ; Write 'A' to screen at (0,0)

    ; Your kernel entry point would typically follow here
    ; jmp kernel_main

Conclusion

Transitioning to Protected Mode is a foundational step in writing a custom x86 bootloader and operating system. It unlocks the modern capabilities of the CPU, moving beyond the severe limitations of Real Mode. By carefully managing the A20 line, setting up a Global Descriptor Table, and executing the precise sequence of instructions to manipulate the CR0 register and segment selectors, you lay the groundwork for a robust and powerful operating system environment. This expert-level guide has provided the necessary code and explanations to master this critical phase of bootloader development.