Writing a legacy PXE bootloader
This is my first post in this blog (hopefully not the last). Here I will try to describe my journey in building a legacy PXE bootloader. Specifically in this instalment I will describe the initial work I'v done in setting up the test environment.
Back in the first half of the 2000s, I was developing my own toy operating system as a hobby, I did that for a few years, but never really got anywhere because of multitude of reasons (mainly I kept rewritting it over and over because it was "not perfect"), and eventually I gave up having only written a part of a bootloader. Now that I'm about 20+ years older, and hopefully wiser, I kind of got an inkling of trying it again. However I kind of don't want to go the 64bit UEFI way as most people do today. Mainly because I kinda feel nostalgic for old machines (I horded have a bunch of them in the last few years), and I don't want to relearn everything from scratch, I decided to stick with 32bit and legacy boot so I can almost jump in where I left of, and anyway, this hobby was never really meant to be practical anyway.
Now back when I started, the cutting edge way to boot my hobby os was building a floppy image, writting it to an actual physical floppy disk, then inserting it into my test machine and cycling power (my test machine was a 486 with a single floppy drive). Anything that could boot a USB or CD was beyond my financial grasp. And this was really slow and annoying (albeit even CD or USB wouldn't have been much better as it still has the same problem). Today I own a bunch of old machines that are equipped with the Intel Boot Agent (PXE 2.1) So I got to thinking, why not give this PXE thing a shake. That would give me the ability to skip physical media altogether and get closer to the ability to simply push a button in my development machine and have my freshly built binary boot on my test machine. Not having to mess with physical media makes my development cycle somewhat faster.
Setting up and testing the environment
So I got to reading the PXE specification, and it was promising. The one nice thing right of the bat is that the NBP (network boot program, which is the initial piece of code that PXE downloads to initiate the boot process) can be atleast 32KiB in size (altough it can probably be larger), which is much better then the 512 bytes normally afforded to the developer when doing normal disk booting on a legacy boot system. Everyone who has attempted to write anything in 512 bytes can attest to what a pain it is, and forget using anything else but naked assembly code in that case. But with a budget of 32KiB I can fit a C program in, with no problem at all, with some effort. But first, I needed to setup an environment for testing, and for that a bootp/dhcp+tftp server is needed.
Now I looked around for software that could do the job, and since I am a windows user, most of what I could find were gui based servers, and some were payed to boot (pun intended), none of which were very convenient or automatable. And trying to setup linux to do this was just too much hassle, as it's sort of an ancient, long forgotten, dark art, involving writting runic incantations in text files, and I wanted something that I could just start from command line in my windows terminal, pass it a few arguments and off to the races. So I read the DHCP, BOOTP and TFTP rfcs and figured, well that's pretty simple, and sat down to write my own.
Another little thing that I needed to address is DHCP, since every network already comes equipped with a DHCP server in the form of your router, hence adding another DHCP server into it might cause confusion, or atleast would require messing with some configuration on the router (if it even supports such a split configuration). Thankfully it's really easy to fix, by just getting another network adapter (for example an USB one), connecting your test machine to the adapter, and then binding the bootp server to that interface exclusively. This makes the machine effectively isolated in it's own network and no comflicts with other DHCP servers will occure.
So now It was time to test the setup by writting a small hello world binary and trying to boot it over the network. Thankfully this is very simple, the network boot program (or NBP) still obeys most of the same rules as a regular legacy bootsector: it still starts in 16bit, still is loaded into 0x7c00, and it starts via a jump to 0x0:0x7c00, we can still call the BIOS to do all the things that a normal bootsector needs, like printing a character to the screen. So I whip out my text editor and assembler and come up with this. Notice I put the string at the end, so it's the very last 16 bytes of the binary, that makes it possible to verify all 32KiB were transfered. It's possible to make it even larger to see how far PXE can be stretched in terms of NBP size, but it's not guaranteed all implementations will beheave the same, even though I never seen any other implementation then the Intel Boot Agent and the opensource iPXE.
format binary
use16
org 0x7c00
start:
mov ax, cs
mov ds, ax
mov si, hello
cld
print:
lodsb
or al, al
jz done
mov bx, 0x07
mov ah, 0x0e
int 0x10
jmp print
done:
cli
hlt
times 32768 - 16 - ($-$$) db 0xf4
hello:
db "Hello World!",13,10,0,0
Building it ...
>fasm hello.asm hello.bin
flat assembler version 1.73.32 (16384 kilobytes memory)
2 passes, 32768 bytes.
The binary is 32KiB, as expected. Also just to make sure everything is in place, verify with a hex dump.
> xxd -s 0x7f00 hello.bin
00007f00: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007f10: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007f20: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007f30: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007f40: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007f50: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007f60: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007f70: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007f80: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007f90: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007fa0: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007fb0: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007fc0: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007fd0: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007fe0: f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 f4f4 ................
00007ff0: 4865 6c6c 6f20 576f 726c 6421 0d0a 0000 Hello World!....
String as at the end as expected.
By the way, a pro tip for anyone working this low level, learn to wield a hex editor and other reverse engineering tools like ghidra, IDA, various disassemblers, etc. and very your binaries, you never know when you can make a mistake that costs you hours or days of debugging, that can be sometimes spotted by a simple look under a hex editor. Trust me, when you get good at it, there's so many things you can spot by just having a squiz inside the binary.
Now back to testing. Here are settings for bootpd.
v4_bind_address = 10.0.0.1 ; Bind to this adapter idenfitied by the IP address
syslog_listen_port = 514 ; Not yet implemented, just a placeholder
tftp_listen_port = 69 ; The port to listen on for TFTP requests
dhcp_listen_port = 67 ; The port to listen on for DHCP requests
tftp_base_dir = ./ ; Root directory for TFTP requests
[00-1c-7e-35-ed-20] ; MAC address of the computer these settings apply to
; Most of this information is needed for the DHCP response
boot_file_name = hello.bin ; The file to boot from (the NBP)
v4_client_address = 0.0.0.0 ; Placeholder for the client IP address, not used
v4_your_address = 10.0.0.2 ; The IP address to assign to the client
v4_server_address = 10.0.0.1 ; The IP address of the BOOTP/DHCP/TFTP server (this computer)
v4_gateway_address = 0.0.0.0 ; The IP address of the gateway, not used
v4_subnet_mask = 255.0.0.0 ; The subnet mask of this adhoc network
v4_router_address = 10.0.0.1 ; The IP address of the router, not used
v4_dhcp_server_address = 10.0.0.1 ; The IP address of the DHCP server (this computer)
v4_log_server_address = 10.0.0.1 ; The IP address of the log server, not used
server_host_name = home.2bits.in ; The hostname of the server, not used
domain_name = home.2bits.in ; The domain name of the server, not used
address_lease_time = 172800 ; The time in seconds to lease the IP address
address_renewal_time = 86400 ; The time in seconds to renew the IP address
address_rebinding_time = 138240 ; The time in seconds to rebind the IP address
Start it up with command bootpd.exe -C config.ini -O v4_bind_address=10.0.0.1
> bootpd.exe -C config.ini -O v4_bind_address=10.0.0.1
2024-07-26 23:10:02 DEBUG Overriding config ...
2024-07-26 23:10:02 DEBUG * Adding line: 'v4_bind_address=10.0.0.1'
2024-07-26 23:10:02 INFO Starting DHCP server, listening on '10.0.0.1:67' ...
2024-07-26 23:10:02 INFO * Receiver thread started.
2024-07-26 23:10:02 INFO * Responder thread started.
2024-07-26 23:10:02 INFO Starting TFTP server on '10.0.0.1:69', with root at 'C:\Wherever\I\put\this\thing\helloasm\' ...
2024-07-26 23:10:02 INFO * Receiver thread started.
2024-07-26 23:10:02 INFO * Responder thread started.
Now bootpd is waiting for a connection from the test machine, press the on button, it thinks a bit, and both bootpd and the machine spring into action.
2024-07-26 23:11:25 INFO Received 548 bytes from '0.0.0.0:68'.
2024-07-26 23:11:25 INFO Responding to '0.0.0.0:68' (transaction 0x8035ed20) DHCP.DISCOVER packet with DHCP.OFFER packet.
2024-07-26 23:11:27 INFO Received 548 bytes from '0.0.0.0:68'.
2024-07-26 23:11:27 INFO Responding to '0.0.0.0:68' (transaction 0x8035ed20) DHCP.REQUEST packet with DHCP.ACK packet.
2024-07-26 23:11:27 INFO Received 26 byte TFTP packet from '10.0.0.2:2070' ...
2024-07-26 23:11:27 INFO From '10.0.0.2:2070' received : RRQ(file="hello.bin", mode="octet", tsize="0")
2024-07-26 23:11:27 INFO Received 31 byte TFTP packet from '10.0.0.2:2071' ...
2024-07-26 23:11:27 ERROR Connection terminated : ERROR(code=Not defined, message="TFTP Aborted")
2024-07-26 23:11:27 INFO From '10.0.0.2:2071' received : RRQ(file="hello.bin", mode="octet", blksize="1456")
2024-07-26 23:11:27 DEBUG Killing session 0x206c6b96680 ...
2024-07-26 23:11:27 INFO Starting transfer of hello.bin to '10.0.0.2:2071' (file_size = 32768 bytes, blksize = 1456 bytes, timeout = 1 sec) ...
2024-07-26 23:11:27 INFO Finished sending hello.bin to '10.0.0.2:2071' ...
2024-07-26 23:11:27 DEBUG Killing session 0x206c6b96c20 ...
2024-07-26 23:12:18 INFO Received 548 bytes from '0.0.0.0:68'.
2024-07-26 23:12:18 INFO Responding to '0.0.0.0:68' (transaction 0x8035ed20) DHCP.DISCOVER packet with DHCP.OFFER packet.
2024-07-26 23:12:20 INFO Received 548 bytes from '0.0.0.0:68'.
2024-07-26 23:12:20 INFO Responding to '0.0.0.0:68' (transaction 0x8035ed20) DHCP.REQUEST packet with DHCP.ACK packet.
2024-07-26 23:12:20 INFO Received 26 byte TFTP packet from '10.0.0.2:2070' ...
2024-07-26 23:12:20 INFO From '10.0.0.2:2070' received : RRQ(file="hello.bin", mode="octet", tsize="0")
2024-07-26 23:12:20 INFO Received 31 byte TFTP packet from '10.0.0.2:2071' ...
2024-07-26 23:12:20 ERROR Connection terminated : ERROR(code=Not defined, message="TFTP Aborted")
2024-07-26 23:12:20 INFO From '10.0.0.2:2071' received : RRQ(file="hello.bin", mode="octet", blksize="1456")
2024-07-26 23:12:20 DEBUG Killing session 0x206c6b967a0 ...
2024-07-26 23:12:20 INFO Starting transfer of hello.bin to '10.0.0.2:2071' (file_size = 32768 bytes, blksize = 1456 bytes, timeout = 1 sec) ...
2024-07-26 23:12:20 INFO Finished sending hello.bin to '10.0.0.2:2071' ...
2024-07-26 23:12:20 DEBUG Killing session 0x206c6b96b00 ...
Preparation for developing the bootloader
Now what I described above wasn't quite the quick and simple, I rather skipped alot of back and forth while developing bootpd. In actuallity it took me a while to get it operating mostly bug free, even though the protocol is dead simple, it takes quite a bit of effort to make a useful piece of software. One thing that I really can't live without when developing is a debugger. In this setup of booting directly on hardware I am SoL when it comes to debugging step by step. So I needed to address this situation.
Luckily there is a way, and it's using an emulator (or virtual machine). Any OS developer woun't leave home without it. Specifically for debugging purposes something like VMWare or VirtualBox are kind of a nonstarter, unless you want to go the old fashioned way (spamming prints everywhere and hope your print actually works). Again, luckily there are emulators that have pretty decent debugging capabilities, namely Bochs (with it's builtin debugger) and Qemu (via GDB). I'm not a big fan of using GDB, it never quite made me scream with joy, nor is it very good with 16bit debugging. But it's good to have a backup plan, as well as testing on multiple emulators gives you a better chance of finding obscure bugs, so I setup both.
Setting up bochs
Bochs is my go to for debugging 16bit bit code, and alot of the time 32bit as well, I like it's simplicity, it also has a very handy feature called magic break, when you enable it, every time it finds a xchg bx,bx
(which is a nop basically) it will break letting you single step from that point. Bochs also has integrated PXE mocking, all it lacks is a PXE rom, but that's no problem, as it turns out just copying the rom from Qemu works absolutely fine, so that's what I did. I chose to emulate a ne2k, so I copied pxe-ne2k_pci.rom from Qemu into my bochs configuration directory, and added the line ne2k: ioaddr=0x300, irq=3, type=pci, mac=52:54:00:12:34:57, ethdev="paths/to/your/binaries", ethmod=vnet, script=vnet.ini, bootrom=pxe-ne2k_pci.rom
to my bochs.dbg.ini file. Also had to add a vnet.ini file with the following settings:
net = 192.168.10.0
host = 192.168.10.1
hostname = home.2bits.in
dhcpstart = 192.168.10.15
dns = 192.168.10.2
ftp = 192.168.10.3
bootfile = hello.bin
I will not go into detail what the settings mean, there is an example of bochs.ini and vnet.ini inside the bochs installation directory, it's pretty straight forward. Then all I need to do is add a mgic break point (xchg bx, bx
) to my hello.asm and launch bochs like so cd your_bochs_settings_dir && C:\Path\To\Bochs\bochsdbg.exe -f bochs.dbg.ini -q
, wait for the UI to appear and click continue, wait a bit, and we're in line flin (you can tell I'm a fan of Dave Jones) .
Now I can single step trough the assembly, place break points etc. Ofcourse source level debugging this is not and once I get to the C program stage It'll be somewhat more difficult to use this, but with a decent enough intuition of how C maps to assembly and some creative use of the magic break, alot can be done.
Moving on to qemu
Ethernet adapter NordVPNTapAdapter:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : TAP-NordVPN Windows Adapter V9
Physical Address. . . . . . . . . : 00-FF-AD-34-08-24
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
The important bit is the name "NordVPNTapAdapter" which will be used in the Qemu configuration. Also a MAC address is needed which will be used in the bootpd configuration, but Qemu will assign it's own MAC address, that will be displayed by Qemu when it starts and initializes it's virtual network adapter. This makes the updated bootpd config look something like this:
v4_bind_address = 10.0.1.1
syslog_listen_port = 514
tftp_listen_port = 69
dhcp_listen_port = 67
tftp_base_dir = ./
[52-54-00-12-34-56] ; Given by Qemu upon startup of it's PXE rom
boot_file_name = hello.bin
v4_client_address = 0.0.0.0
v4_your_address = 10.0.1.2
v4_server_address = 10.0.1.1
v4_gateway_address = 0.0.0.0
v4_subnet_mask = 255.0.0.0
v4_router_address = 10.0.1.1
v4_dhcp_server_address = 10.0.1.1
v4_log_server_address = 10.0.1.1
server_host_name = home.2bits.in
domain_name = home.2bits.in
address_lease_time = 172800
address_renewal_time = 86400
address_rebinding_time = 138240
And then start Qemu like so "C:\Path\to\qemu\qemu-system-x86_64.exe" -machine pc -m 2048 -vga qxl -netdev tap,id=n0,ifname=NordVPNTapAdapter -device ne2k_pci,netdev=n0 -option-rom "C:\Path\to\qemu\share\pxe-ne2k_pci.rom"
. Well actually both bootpd and qemu need to be started sort of simultaneusly as the tap adapter isn't brought up until qemu uses it, and qemu can't boot until bootpd is started, what I did instead is make a batch script that looks like this:
start C:\Path\to\bootpd\bootpd.exe -C config.ini -O v4_bind_address=10.0.1.1
start C:\Path\to\qemu\qemu-system-x86_64.exe -machine pc -m 2048 -vga qxl -netdev tap,id=n0,ifname=NordVPNTapAdapter -device ne2k_pci,netdev=n0 -option-rom C:\Path\to\qemu\share\pxe-ne2k_pci.rom
Run it, in like flin, bob's your uncle, etc.
There's one little detail still missing from this, and that is GDB. As I mentioned already, GDB really doesn't like 16bit code, It will not be able to currectly disassembly or debug it unless you add a bit of magic fairy dust to it (helpfully described in this SO answer). So the actual qemu startup script looks more like this:
start C:\Path\to\bootpd\bootpd.exe -C config.ini -O v4_bind_address=10.0.1.1
start C:\Path\to\qemu\qemu-system-x86_64.exe -S -gdb tcp::41234 -machine pc -m 2048 -vga qxl -netdev tap,id=n0,ifname=NordVPNTapAdapter -device ne2k_pci,netdev=n0 -option-rom C:\Path\to\qemu\share\pxe-ne2k_pci.rom
gdb -ix path/to/init_real_mode.txt \
-ex "target remote localhost:1234" \
-ex "set tdesc filename gdb/target.xml" \
-ex "set architecture i8086" \
-ex "set disassembly-flavor intel" \
-ex "break *0x7c01" \
-ex "continue" \
-ex "x /5i 0x7c00"
Once more, from the top ...
Beutiful, now I can use both bochs and qemu+gdb for debugging. PS, I'm using GDB from Ubuntu on WSL here, just incase seomone tries to download the windows port and discovers that it doesn't work or smth, no angry letters please.
Next time on 2bits.in ...
Not wanting this article to be a mile long I will cut it here for now, but another post is comming, as I still want to describe the process of getting a C program to run as a NBP, and all the trials and turbulations of bootstrapping OpenWatcom to be my main toolchain for this endevour, and perhaps some justification why I chose said toolchain.