Chroot and rootfs

Folder structure and rootfs

Folder structure is an important part of Unix and Unix-like operating systems (like Linux). The folder structure starts at a single root / and when you check the content of that root (ls /), you may realise that different distributions often have similar or overlapping folders. The folder structure of the root is actually defined by the Filesystem Hierarchy Standard. Basically some folders are always expected by the Linux kernel to be present.

In a Linux distribution, we sometimes talk about a “rootfs”, short for “root file system”. This is basically the folder and file structure starting at /. It’s possible to mount images, folders, and storage devices to specific places in the folder hierarchy. Generally the rootfs is the structure starting at / without these extra mount points.

A nice way to start playing and understanding this rootfs thing more, is the so-called chroot command (short for “change root”, so the “ch” can be pronounced as the “ch” in “change”). With it you can basically say “pretend this folder is the root and then run this comand”. Let’s try it!

Chroot jail

For the following I installed Alpine 3.16 on some old hardware. You should be able to do similar things with other Linux distros as well. Note that you need elevated privileges, so make sure you run the following commands as the root user (not to be confused with the root folder, they are two different things).

Let’s say we are in our home folder ~. Let’s make a folder to play in

mkdir my_chroot

Now let’s try to start a shell who believes this folder is actually the root.

chroot my_chroot sh

What we see is that we get an error chroot: can't execute 'sh': No such file or directory, so what happened?

We started a chroot in the folder my_chroot and asked to execute the command sh. When you ask to execute a command without specifying it’s location, your shell will try to find the command in the so called PATH. The PATH is nothing more than a variable holding a colon separated list of folders where we can store the executables we want to be able to call without needing to specify the specific location. To see the list, you can run echo $PATH. For me it looks like /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

So what happened now is that we try to locate sh in these folders, but these folders don’t exist. After all, when we try to locate /bin inside the chroot, what we really look at is the ~/my_chroot/bin folder when looked at from the host system. So even if the command exists on our host, we can’t access it from inside the chroot. To make sure we can run a command, we need to make it accesable inside the chroot folder structure. To do that we can mount the folder containing the command, or we can place the command there manually. Let’s do the seconds one!

First we’ll check where the correct sh binary is. Then we can copy it over to the location we need it. There’s a good chance this binary relies on some libraries to be present, so we check for that with the ldd command and copy those over as well.

which sh
# I see the binary is `/bin/sh`
mkdir my_chroot/bin
cp /bin/sh my_chroot/bin/sh

ldd /bin/sh
# I see there's one lib `/lib/ld-musl-x86_64.so.1`
mkdir my_chroot/lib
cp /lib/ld-musl-x86_64.so.1 my_chroot/lib/ld-musl-x86_64.so.1

Now we can try to run this shell in the root container again chroot my_chroot sh, and if it works, you should now have a shell! If you still get the chroot: failed to run command ‘sh’: No such file or directory error, you should double check that both the binary and the libs are indeed there and with the right permissions. The difference between the shell you had and the one you have now may not be immediately visible, but you should be at root now and a lot of commands wont work who otherwise did. Some may work because they are part of the shell, but others may not be and you need the binaries for them. In my case ls didn’t work, pwd did. To exit the shell, you can do exit. If you want to add more commands, you can do that in a similar way. Btw, if you ever hear talk about a “chroot jail”, this is basically what they are talking about; A piece of software that’s run with chroot. The chroot environment (i.e. the environment that the software runs in after starting it with the chroot command) is the “chroot jail”.

It’s also possible that some commands need certain configuration files. When I copy apk in the same way and then do apk --help from inside the chroot, it works. But when I do apk update it gives errors about not being able to lock and open a database. After a lot of searching and trial and error I eventually did

# First move apk and the libraries
which apk
mkdir my_chroot/sbin
cp /sbin/apk my_chroot/sbin/apk

ldd /sbin/apk my_chroot/lib/
cp /lib/libcrypto.so.1.1 my_chroot/lib/libcrypto.so.1.1
cp /lib/libz.so.1 my_chroot/lib/libz.so.1
cp /lib/libapk.so.3.12.0 my_chroot/lib/libapk.so.3.12.0
cp /lib/ld-musl-x86_64.so.1 my_chroot/lib/ld-musl-x86_64.so.1
cp /lib/libssl.so.1.1  my_chroot/lib/libssl.so.1.1

# Now the following already works
chroot my_chroot apk --help

# But we need a bunch of other stuff
# if we want to `chroot my_chroot apk update`
mkdir my_chroot/lib/apk/
cp -r /lib/apk/db my_chroot/lib/apk/db
mkdir -p my_chroot/etc/apk
cp /etc/apk/repositories my_chroot/etc/apk/repositories
touch my_chroot/etc/apk/world
cp -r /lib/apk my_chroot/lib/apk
mkdir my_chroot/var
cp /etc/resolv.conf my_chroot/etc/resolv.conf
cp /bin/busybox my_chroot/bin/busybox

# Now update almost works
# It still gives an error about an untrusted key
# We can probably fix it by copying the correct files
# Or we can work around it by using the ` --allow-untrusted` parameter
chroot my_chroot apk update --allow-untrusted

Now we can actually install packages inside our chroot environment. That’s pretty sweet huh!

# With this we can install all basic tools you get in Alpine
chroot my_chroot sh
apk add alpine-base --allow-untrusted

Everything is a file

When we now check the content of the chroot folder ls my_chroot, you may notice a new folder proc. This was added when we did apk update. You may have heard that in Unix “everything is a file”. What people mean by this is that you can access everything, even processes and devices, from the filesystem. The dev, proc and sys folders are originally empty, but when you start up your operating system, the Linux kernel will populate them. The dev folder will show devices while proc will show processes. Note that while we see a proc folder here, it’s not actually mounted from the kernel. I assume apk needed it for some reason and then created it because it wasn’t there.

When you need these folders in the chroot, one way to do it, is to mount them. When looking online, it seems that the following is often used for this.

mount -t proc none /mnt/proc
mount --rbind /sys /mnt/sys
mount --rbind /dev /mnt/dev

Limitations

With chroot we can limit some access a program has to the system. Not only can’t it access certain parts of the file system, it doesn’t even know about those parts. And that’s pretty neat I think. But there’s more than just the file system. It’s possible that an attacker may still get access to the host system through other means. First of all there’s the fact that we can mount or link things into the chroot, meaning that those parts of our system are now accessable. If the sh binary isn’t a separate binary, but linked to the sh of the host system, then making changes to the binary means we make the changes to the actual sh binary on our host system! But even besides that, the file system isn’t everything. The proc folder already shows us that there’s a thing like processes, and there’s other things as well. If you want to go a step further than only chroot, there’s containers. Containers use chroot, but also similar techniques to limit other parts of the system. You can check out LXC containers for example who can be easily managed using LXD. What you end up with is a system that uses the same kernel as the host, but can start it’s own operating system from the rootfs in the containerised environment. One step further than this is virtual machines, where you even run your own separate boot process and kernel.

Round-up

Even with all the caveats, chroot already gives us a first insight of what’s needed for some software to work and gives us a playground to get some first insights into a rootfs a bit. Try it!