Posts

Compiling a Kernel Module for BeagleBone

Preface

This post was written as I attempted to create a kernel module for BeagleBone. The contents includes my troubleshooting efforts and a final resolution as to the method I found which worked, found at the end of the post.

Troubleshooting Phase

Here’s what I started with:
BeagleBone A6 (beaglebone:~$ uname -a
Linux beaglebone 3.2.28 #1 Sun Oct 21 15:51:05 CEST 2012 armv7l GNU/Linux
)
Ubuntu 11.04 via VMWare Player with the following installed:

  • git
  • gawk
  • subversion
  • texinfo (stand-in for makeinfo)
  • texi2html
  • chrpath

To get the Angstrom kernel source, I followed the instructions from Angstrom, plus some hints from the beaglebone specific page.

git clone git://github.com/Angstrom-distribution/setup-scripts.git
cd setup-scripts
MACHINE=beaglebone ./oebb.sh config beaglebone
MACHINE=beaglebone ./oebb.sh update
MACHINE=beaglebone ./oebb.sh bitbake virtual/kernel

On that last step, building the kernel, I ran into an issue:

NOTE: Error expanding variable buildhistory_get_imageinfo       | ETA:  00:04:56
NOTE: Error during finalise of /home/user/setup-scripts/sources/meta-openembedded/meta-initramfs/recipes-bsp/images/initramfs-kexecboot-klibc-image.bb
ERROR: Failure expanding variable buildhistory_get_imageinfo, expression was 	if [ "${@base_contains('BUILDHISTORY_FEATURES', 'image', '1', '0', d)}" = "0" ] ; then
		return
	fi

...
which triggered exception OSError: [Errno 12] Cannot allocate memory
ERROR: Command execution failed: Exited with 1
NOTE: Error expanding variable buildhistory_get_imageinfo
NOTE: Error during finalise of /home/user/setup-scripts/sources/meta-openembedded/meta-initramfs/recipes-bsp/images/initramfs-kexecboot-image.bb

I had started out with only 512MB RAM, being that I only have 2GB on the host machine. I tried upping it to 1GB for the virtual machine, but no dice. Last RAM modification attempt, 2 GB specified for the virtual machine, and yet the failure remained. In my searches, I came across someone who’d been working on a similar setup and found they’d used some additional tools I had not installed:

  • build-essential (turned out to already be installed, not of my doing)
  • python-psyco (did not resolve the problem)

One of the messages that came up during the config stage had said it preferred bash to dash, but that it was going to access it via /bin/sh, so if I really wanted it to be happy, I should remap the symbolic link, which I did. Another message that comes up during the bitbaking suggests to do ‘. /home/user/.oe/environment-angstromv2012.05’ and then run bitbake something (I chose nano) without using ./oebb.sh. So I gave that a try. It failed almost identically.

bitbake nano
Pseudo is not present but is required, building this first before the main build
NOTE: angstrom DOES NOT support libiconv because the eglibc provided iconv library is used
NOTE: angstrom DOES NOT support libiconv because the eglibc provided iconv library is used
NOTE: Error expanding variable do_populate_sdk#########         | ETA:  00:01:58
NOTE: Error during finalise of /home/user/setup-scripts/sources/openembedded-core/meta/recipes-core/meta/external-python-tarball.bb
ERROR: Failure expanding variable METADATA_REVISION, expression was ${@base_detect_revision(d)} which triggered exception OSError: [Errno 12] Cannot allocate memory
ERROR: Command execution failed: Exited with 1
NOTE: Error expanding variable toolchain_create_sdk_version
NOTE: Error during finalise of /home/user/setup-scripts/sources/openembedded-core/meta/recipes-core/meta/meta-toolchain-gmae.bb

Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

So, I got a borrowed Ubuntu 11.04 machine, as the memory note really seemed important. I started with cloning the setup-scripts repo, configuring for beaglebone, and running the update. Then I stopped, and did the . /home/user/.oe/environment-angstromv2012.05 and then run bitbake nano steps as previously suggested. This time was much more positive! Note that I disregarded the request to change /bin/sh to point to bash as the computer is a loaner. However, quite some time later, nano finished building. I hadn’t imagined it would take so long, so I wasn’t watching the clock, so I can only say it took several hours.

Once nano built, I felt confident that I would finally be able to bitbake kernel/virtual, so I started it and let it run, which it did within a few hours. I went ahead and bitbaked systemd-image as well, before finally bitbaking virtual/kernel -c compile -f, as suggested in the similar setup post I mentioned earlier. Finally this created files where the aforementioned post suggests, which look to be what is needed (for me the location was: /home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git).

Now we should have what is needed to cross compile a kernel module for BeagleBone on Ubuntu. However, things aren’t just going to start being simple, are they? I set up my extremely basic kernel module:

#include <linux/module.h>
#include <linux/kernel.h>

static int __init enable_usermode(void)
{
        printk(KERN_INFO "Usermode enabled.\n");
        return 0;
}

static void __exit disable_usermode(void)
{
        printk(KERN_INFO "Usermode disabled.\n");
}

module_init(enable_usermode);
module_exit(disable_usermode);

And my Makefile:

obj-m += enable_usermode.o

CROSS = /home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/sysroots/x86_64-linux/usr/bin/armv7a-angstrom-linux-gnueabi/arm-angstrom-linux-gnueabi-
KDIR = /home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git

PWD := $(shell pwd)

all:
        make -C $(KDIR) M=$(PWD) CROSS_COMPILE=$(CROSS) modules
clean:
        make -C $(KDIR) M=$(PWD) CROSS_COMPILE=$(CROSS) clean

And ran make, which resulted in this output:

make -C /home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git M=/home/user/LKM CROSS_COMPILE=/home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/sysroots/x86_64-linux/usr/bin/armv7a-angstrom-linux-gnueabi/arm-angstrom-linux-gnueabi- modules
make[1]: Entering directory `/home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git'
  CC [M]  /home/user/LKM/enable_usermode.o
cc1: error: unrecognized command line option "-m64"
cc1: error: unrecognized command line option "-mno-red-zone"
cc1: error: unrecognized command line option "-mcmodel=kernel"
cc1: error: unrecognized command line option "-maccumulate-outgoing-args"
make[2]: *** [/home/user/LKM/enable_usermode.o] Error 1
make[1]: *** [_module_/home/user/LKM] Error 2
make[1]: Leaving directory `/home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git'
make: *** [all] Error 2

This is disappointing. I figure that there may be some aspect to the way the bitbake setup is done that I’m causing problems with from my side, so I figure it’s time to pursue moving to the BeagleBone. (Note, there was a resolution to the above, found below related to using make ARCH=arm.)

Compiling kernel modules directly on BeagleBone

I started with this in parallel to the effort on Ubuntu, by trying to get the kernel headers using opkg install kernel-headers, however this gave errors:

$ opkg install kernel-headers
Installing kernel-headers (3.2.28-r16a+gitr720e07b4c1f687b61b147b31c698cb6816d72f01) to root...
Downloading http://feeds.angstrom-distribution.org/feeds/v2012.05/ipk/eglibc/armv7a/machine/beaglebone/kernel-headers_3.2.28-r16a+gitr720e07b4c1f687b61b147b31c698cb6816d72f01_beaglebone.ipk.
wget: server returned error: HTTP/1.1 404 Not Found
Collected errors:
* opkg_download: Failed to download http://feeds.angstrom-distribution.org/feeds/v2012.05/ipk/eglibc/armv7a/machine/beaglebone/kernel-headers_3.2.28-r16a+gitr720e07b4c1f687b61b147b31c698cb6816d72f01_beaglebone.ipk, wget returned 1.
* opkg_install_pkg: Failed to download kernel-headers. Perhaps you need to run 'opkg update'?
* opkg_install_cmd: Cannot install package kernel-headers.

So, I checked out http://feeds.angstrom-distribution.org/feeds/v2012.05/ipk/eglibc/armv7a/machine/beaglebone/, where sure enough the kernel-headers had been updated recently, so I ran opkg update as suggested. Unfortunately, using the same kernel module from earlier and the same Makefile with the KDIR changed to KDIR := /lib/modules/$(shell uname -r)/build (and no CROSS_COMPILE), the result of running make was a complaint:
make: *** /lib/modules/3.2.28/build: No such file or directory.  Stop.
Looking into the directory, of course there is no build folder. The only thing I had installed was kernel-headers, which apparently is insufficient for the task of building kernel modules directly on BeagleBone. More disappointingly, I noticed that the only kernel-headers version now available is 3.2.30, while my BeagleBone has Angstrom 3.2.28.

Next, I decided to get a better idea of why I was getting conflicting information about what to use for KDIR in the Makefile. It seems that the standard is to have a symlink /lib/modules/$(uname -r)/build which points to a folder named for your linux kernel in /usr/src. What’s key is that there should be a Makefile here, which determines how kernel modules are built. So, I began a painfully slow scp of the output of the build on Ubuntu, first compressing setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git to hopefully make the ordeal a bit faster.

With the copy ongoing, I decided to give another shot at the cross compiling. I was looking in my Embedded Linux Primer book and realized that I’d forgotten about passing ARCH=arm when calling make. So I went back to my cross compile environment, and navigated to the folder with the Makefile and the basic kernel module code, and ran make ARCH=arm. It built! In a flurry of excitement, I copied it to my BeagleBone and tried to insmod, only to be greeted with: insmod: error inserting 'enable_usermode.ko': -1 Invalid module format

Finally the tar.gz finished copying, so I extracted it to ~/angstrom-build. As root, I symlinked this to a couple places:

ln -s /home/beaglebone/angstrom-build /usr/src/linux-3.2.28
ln -s /usr/src/linux-3.2.28 /lib/modules/3.2.28/build

Then I gave a quick try to building my kernel module, knowing it was likely I’d need to make modules and scripts, but wanting to see what might happen. The attempt yielded this error:
/bin/sh: scripts/basic/fixdep: cannot execute binary file
So I went back to ~/angstrom-build and ran

make modules
make scripts

This took a couple hours and make scripts didn’t seem to do anything. Once this was done, I was able to build the kernel module, however, trying to insert it gave the same issue as when I had tried to insert the cross compiled kernel module. So, I decided to check out what dmesg had to say:
enable_usermode: disagrees about version of symbol module_layout
Uh oh. It seems that even though the numerical version of the kernel matches, something else within the pre-built kernel distributed on the BeagleBone doesn’t match what I’ve built.

So I have two options; the long way being to try to setup the kernel I built on an SD card, or the shorter way, upgrading the kernel on my BeagleBone to see if maybe the kernel-dev package I installed previously can be used once the kernel is upgraded. I chose to try to upgrade the kernel using opkg upgrade. With this selection, I went ahead and did make scripts in /usr/src/kernel before changing the Makefile to use KDIR = /usr/src/kernel, and voila! This time the kernel module built and was successfully inserted!

Suggested course of action to compile a kernel module on BeagleBone

Note that I include directives to switch user (su) as you’ll need to be root in some cases. You may replace /home/user with ~, I typed it out because I noticed that the font made it look like a hyphen sometimes. The command exit will take you back to the previous user, or if you are in the original user who signed in, will disconnect you (so don’t use exit if you only have one user, root). Finally, dmesg will show the kernel prints, so you can look for the text from the example kernel module to be printed when you insert and remove the module.

opkg update
opkg --tmp-dir /home/user upgrade
*reboot if kernel upgraded using: shutdown -r now*
opkg install kernel-headers
opkg install kernel-dev
cd /usr/src/kernel
su
make scripts
exit
cd /home/user
mkdir mykernelmod
cd mykernelmod
*create the below described c file*
*create the below described Makefile*
make ARCH=arm
su
insmod mykernelmod.ko
dmesg
rmmod mykernelmod.ko
dmesg
exit

mykernelmod.c

#include <linux/module.h>
#include <linux/kernel.h>

static int __init enable_mod(void)
{
        printk(KERN_INFO "My Kernel Module is enabled.\n");
        return 0;
}

static void __exit disable_mod(void)
{
        printk(KERN_INFO "My Kernel Module is disabled.\n");
}

module_init(enable_mod);
module_exit(disable_mod);

Makefile

obj-m += mykernelmod.o
KDIR = /usr/src/kernel
PWD := $(shell pwd)

all:
        make -C $(KDIR) M=$(PWD) modules
clean:
        make -C $(KDIR) M=$(PWD) clean

Some useful links