Kernel Sanders: Building a Unix Kernel from Scratch

As part of my undergarduate degree at the University of Illinois at Urbana-Champaign, one of my core classes, Computer Systems Engineering, also known as ECE391, was a hands-on computer engineering course that covered many of the core components behind OS-level engineering. The final project, known as Machine Problem (MP) 3, put our knowledge to the test in a multi-stage endeavor that involved building a Unix kernel in x86 assembly language and native C in groups of four. When our group came together, we promptly chose the name "Kernel Sanders" to represent our project.

As this course was a prerequisite for later computer security courses, I opted to take it as soon as possible. This was the first of the "Big Four" ECE courses I opted to take, and from what I gathered from student reviews, it was a difficult place to start. I was able to experience this for myself from a team leadership and project management perspective; my team and I spent many evenings in the 2nd floor of the ECE building attempting to piece together bits of code to make our kernel come to life.

When I took this course in 2022, MP3 was split into five checkpoints with deadlines placed over the course of two months. The checkpoints were roughly organized as such:

• Checkpoint 1: Bootstrapping the Kernel and initializing the drivers
• Checkpoint 2: Terminal drivers and filesystem
• Checkpoint 3 and 4: System calls
• Checkpoint 5: Signal handler

I will admit that this project was challenging, and this story is one about failure. To write about my experience may seem counterintuitive, but these are my highlights from this experience:

• To me, failure presents more opportunities to learn than success.
• Time management is an essential part of any project, and knowing where your boundaries are may be even more helpful.
• It's important to know how to reallocate tasks when your team is impacted, either from losing a team member or when targets are adjusted.

At the beginning of the project, our first task was to bootstrap the kernel. The project repository, hosted on GitHub and accessible to all team members, contained some placeholder files that were designed to interface with QEmu, an emulator service that hosted the environment for our kernel. This step required writing some basic assembly linkage in x86 that tethered QEmu to our kernel. This step was critical as this is the only way we can interface with our kernel outside of the Visual Studio development environment, and it was the primary platform that would be used for testing by TAs and professors at each checkpoint's deadline.

For the second checkpoint, the first challenge our team faced was implementing the terminal driver. The terminal driver is a critical part of our application; without the terminal, we wouldn't be able to interface keyboard input or run programs from the filesystem, two main components of any kernel. As such, our testing scheme for the terminal driver included reading input from the keyboard and writing strings to the terminal.

In our write test, we encountered issues with writing long strings to our terminal. With help from some TAs, we discovered that this issue had to do with how our terminal handled wrapping text between lines, which tied back to some of our driver code and monitor interface from the first checkpoint. This was a true exercise in finding and squashing bugs as they appear in development, a valuable skill which I use often and will likely continue to use as my career progresses.

The next challenge for our team was implementing a read-only filesystem. The filesystem is another critical element of our Unix kernel; without it, users would not be able to store and retrieve information through the terminal driver. This enables users to run programs and browse the filesystem directory, among other critical tasks.

Testing for the filesystem involved checking if a file exists (test), printing the content of different sized files (cat), and printing a list of all files in the filesystem (ls). Coupled with the issues we encountered in our terminal driver, testing the filesystem was a more difficult endeavor than we anticipated, but by working together as a team we were able to complete all of our objectives by the second deadline.

The fourth checkpoint took place shortly after Thanksgiving break, which the school observed as a holiday. The requirements were lower than the previous checkpoints, but still included some important items that were necessary for the creation of our kernel.

Perhaps the most important part of this checkpoint was the integration of video memory (vidmem) into our kernel. Given the troubleshooting we had already conducted regarding assembly linkage and the filesystem, the incorporation of vidmem into our project is where we became adept at using GDB to dissect our code. Much like the third checkpoint, the test programs we used to verify our progress were launching multiple shells (shell > shell), incorporating arguments into commands (cat <filename>), and a quick two-frame animation test to ensure vidmem is mapped correctly (fish).

This checkpoint was where our pacing started to suffer; not only were our other classes starting to pick up and finals began to loom near, but we had lost a member of our team and we weren't quite sure how to allocate the work between the three of us that remained. This resulted in more hours in the lab, more research on the internet, and many trips to the Taco Bell on University Street to keep us fueled through it all.

At the final checkpoint, teams can incorporate additional features for extra credit. Our team found this checkpoint to be the most challenging part of the project, not necessarily for what we wanted to do but how we needed to bring everything together. With final exams and other projects impending, our team found it difficult to truly come together and deliver the final product. This turned into a great learning experience for the three of us; we learned how to efficiently manage our time when we needed to deliver MVPs.

We were able to implement a quick round-robin scheduling algorithm for our kernel but fell short when it came to implementing multiple terminals. The efficiency of the schedulign algorithm came in the form of a slightly more reponsive terminal, which mattered when it came to programs like fish and pingpong. For the multiple terminal setup, the idea is that you can toggle between three terminals by using the function keys (terminal 1 on F1, terminal 2 on F2, etc.) Looking back on this, it might have been a memory or paging error, but we were not able to retain context on the terminals when we switched between them. For example, the output of counter would disappear if we tabbed in and out of the terminal.

This project was my first true endeavor into operating systems and kernel creation, and there's no better way to do this than from scratch. Working as a team helped split the load, but by the time we lost our fourth team member we ultimately struggled with how to keep up the pace and reach the finish line. Regardless, this was a learning lesson for all of us in time management, project management, and sifting through code figuring out what went wrong.

The lessons learned through this project have definitely helped me with future classes, future work projects, and future personal projects. The ability to really dig into how the kernel works, especially when I helped write the kernel, really aided in my understanding of how to use GDB (and how to debug in general). I find that I am able to perform some of my best work under pressure, so the added stress of completing Checkpoint 5 while finals loomed near helped me a little bit at the end. And finally, working as a team became very important with this project; it became critical for later projects, both in and out of school.

Overall, this project was a great way to develop skills in OS programming and architecture. This project served as one of the main reasons why I wanted to get into software engineering and low-level architecture, and later courses helped solidify my sentiment.

OpenAI: header image

EEEGuide.com: i8259 pinout