Compiling a Kernel Module for BeagleBone

Preface

This post was written as I attempted to create a kernel module for BeagleBone. The contents includes my troubleshooting efforts and a final resolution as to the method I found which worked, found at the end of the post.

Troubleshooting Phase

Here’s what I started with:
BeagleBone A6 (beaglebone:~$ uname -a
Linux beaglebone 3.2.28 #1 Sun Oct 21 15:51:05 CEST 2012 armv7l GNU/Linux
)
Ubuntu 11.04 via VMWare Player with the following installed:

  • git
  • gawk
  • subversion
  • texinfo (stand-in for makeinfo)
  • texi2html
  • chrpath

To get the Angstrom kernel source, I followed the instructions from Angstrom, plus some hints from the beaglebone specific page.

git clone git://github.com/Angstrom-distribution/setup-scripts.git
cd setup-scripts
MACHINE=beaglebone ./oebb.sh config beaglebone
MACHINE=beaglebone ./oebb.sh update
MACHINE=beaglebone ./oebb.sh bitbake virtual/kernel

On that last step, building the kernel, I ran into an issue:

NOTE: Error expanding variable buildhistory_get_imageinfo       | ETA:  00:04:56
NOTE: Error during finalise of /home/user/setup-scripts/sources/meta-openembedded/meta-initramfs/recipes-bsp/images/initramfs-kexecboot-klibc-image.bb
ERROR: Failure expanding variable buildhistory_get_imageinfo, expression was 	if [ "${@base_contains('BUILDHISTORY_FEATURES', 'image', '1', '0', d)}" = "0" ] ; then
		return
	fi

...
which triggered exception OSError: [Errno 12] Cannot allocate memory
ERROR: Command execution failed: Exited with 1
NOTE: Error expanding variable buildhistory_get_imageinfo
NOTE: Error during finalise of /home/user/setup-scripts/sources/meta-openembedded/meta-initramfs/recipes-bsp/images/initramfs-kexecboot-image.bb

I had started out with only 512MB RAM, being that I only have 2GB on the host machine. I tried upping it to 1GB for the virtual machine, but no dice. Last RAM modification attempt, 2 GB specified for the virtual machine, and yet the failure remained. In my searches, I came across someone who’d been working on a similar setup and found they’d used some additional tools I had not installed:

  • build-essential (turned out to already be installed, not of my doing)
  • python-psyco (did not resolve the problem)

One of the messages that came up during the config stage had said it preferred bash to dash, but that it was going to access it via /bin/sh, so if I really wanted it to be happy, I should remap the symbolic link, which I did. Another message that comes up during the bitbaking suggests to do ‘. /home/user/.oe/environment-angstromv2012.05’ and then run bitbake something (I chose nano) without using ./oebb.sh. So I gave that a try. It failed almost identically.

bitbake nano
Pseudo is not present but is required, building this first before the main build
NOTE: angstrom DOES NOT support libiconv because the eglibc provided iconv library is used
NOTE: angstrom DOES NOT support libiconv because the eglibc provided iconv library is used
NOTE: Error expanding variable do_populate_sdk#########         | ETA:  00:01:58
NOTE: Error during finalise of /home/user/setup-scripts/sources/openembedded-core/meta/recipes-core/meta/external-python-tarball.bb
ERROR: Failure expanding variable METADATA_REVISION, expression was ${@base_detect_revision(d)} which triggered exception OSError: [Errno 12] Cannot allocate memory
ERROR: Command execution failed: Exited with 1
NOTE: Error expanding variable toolchain_create_sdk_version
NOTE: Error during finalise of /home/user/setup-scripts/sources/openembedded-core/meta/recipes-core/meta/meta-toolchain-gmae.bb

Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

So, I got a borrowed Ubuntu 11.04 machine, as the memory note really seemed important. I started with cloning the setup-scripts repo, configuring for beaglebone, and running the update. Then I stopped, and did the . /home/user/.oe/environment-angstromv2012.05 and then run bitbake nano steps as previously suggested. This time was much more positive! Note that I disregarded the request to change /bin/sh to point to bash as the computer is a loaner. However, quite some time later, nano finished building. I hadn’t imagined it would take so long, so I wasn’t watching the clock, so I can only say it took several hours.

Once nano built, I felt confident that I would finally be able to bitbake kernel/virtual, so I started it and let it run, which it did within a few hours. I went ahead and bitbaked systemd-image as well, before finally bitbaking virtual/kernel -c compile -f, as suggested in the similar setup post I mentioned earlier. Finally this created files where the aforementioned post suggests, which look to be what is needed (for me the location was: /home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git).

Now we should have what is needed to cross compile a kernel module for BeagleBone on Ubuntu. However, things aren’t just going to start being simple, are they? I set up my extremely basic kernel module:

#include <linux/module.h>
#include <linux/kernel.h>

static int __init enable_usermode(void)
{
        printk(KERN_INFO "Usermode enabled.\n");
        return 0;
}

static void __exit disable_usermode(void)
{
        printk(KERN_INFO "Usermode disabled.\n");
}

module_init(enable_usermode);
module_exit(disable_usermode);

And my Makefile:

obj-m += enable_usermode.o

CROSS = /home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/sysroots/x86_64-linux/usr/bin/armv7a-angstrom-linux-gnueabi/arm-angstrom-linux-gnueabi-
KDIR = /home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git

PWD := $(shell pwd)

all:
        make -C $(KDIR) M=$(PWD) CROSS_COMPILE=$(CROSS) modules
clean:
        make -C $(KDIR) M=$(PWD) CROSS_COMPILE=$(CROSS) clean

And ran make, which resulted in this output:

make -C /home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git M=/home/user/LKM CROSS_COMPILE=/home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/sysroots/x86_64-linux/usr/bin/armv7a-angstrom-linux-gnueabi/arm-angstrom-linux-gnueabi- modules
make[1]: Entering directory `/home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git'
  CC [M]  /home/user/LKM/enable_usermode.o
cc1: error: unrecognized command line option "-m64"
cc1: error: unrecognized command line option "-mno-red-zone"
cc1: error: unrecognized command line option "-mcmodel=kernel"
cc1: error: unrecognized command line option "-maccumulate-outgoing-args"
make[2]: *** [/home/user/LKM/enable_usermode.o] Error 1
make[1]: *** [_module_/home/user/LKM] Error 2
make[1]: Leaving directory `/home/user/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git'
make: *** [all] Error 2

This is disappointing. I figure that there may be some aspect to the way the bitbake setup is done that I’m causing problems with from my side, so I figure it’s time to pursue moving to the BeagleBone. (Note, there was a resolution to the above, found below related to using make ARCH=arm.)

Compiling kernel modules directly on BeagleBone

I started with this in parallel to the effort on Ubuntu, by trying to get the kernel headers using opkg install kernel-headers, however this gave errors:

$ opkg install kernel-headers
Installing kernel-headers (3.2.28-r16a+gitr720e07b4c1f687b61b147b31c698cb6816d72f01) to root...
Downloading http://feeds.angstrom-distribution.org/feeds/v2012.05/ipk/eglibc/armv7a/machine/beaglebone/kernel-headers_3.2.28-r16a+gitr720e07b4c1f687b61b147b31c698cb6816d72f01_beaglebone.ipk.
wget: server returned error: HTTP/1.1 404 Not Found
Collected errors:
* opkg_download: Failed to download http://feeds.angstrom-distribution.org/feeds/v2012.05/ipk/eglibc/armv7a/machine/beaglebone/kernel-headers_3.2.28-r16a+gitr720e07b4c1f687b61b147b31c698cb6816d72f01_beaglebone.ipk, wget returned 1.
* opkg_install_pkg: Failed to download kernel-headers. Perhaps you need to run 'opkg update'?
* opkg_install_cmd: Cannot install package kernel-headers.

So, I checked out http://feeds.angstrom-distribution.org/feeds/v2012.05/ipk/eglibc/armv7a/machine/beaglebone/, where sure enough the kernel-headers had been updated recently, so I ran opkg update as suggested. Unfortunately, using the same kernel module from earlier and the same Makefile with the KDIR changed to KDIR := /lib/modules/$(shell uname -r)/build (and no CROSS_COMPILE), the result of running make was a complaint:
make: *** /lib/modules/3.2.28/build: No such file or directory.  Stop.
Looking into the directory, of course there is no build folder. The only thing I had installed was kernel-headers, which apparently is insufficient for the task of building kernel modules directly on BeagleBone. More disappointingly, I noticed that the only kernel-headers version now available is 3.2.30, while my BeagleBone has Angstrom 3.2.28.

Next, I decided to get a better idea of why I was getting conflicting information about what to use for KDIR in the Makefile. It seems that the standard is to have a symlink /lib/modules/$(uname -r)/build which points to a folder named for your linux kernel in /usr/src. What’s key is that there should be a Makefile here, which determines how kernel modules are built. So, I began a painfully slow scp of the output of the build on Ubuntu, first compressing setup-scripts/build/tmp-angstrom_v2012_05-eglibc/work/beaglebone-angstrom-linux-gnueabi/linux-ti33x-psp-3.2.28-r16b+gitr720e07b4c1f687b61b147b31c698cb6816d72f01/git to hopefully make the ordeal a bit faster.

With the copy ongoing, I decided to give another shot at the cross compiling. I was looking in my Embedded Linux Primer book and realized that I’d forgotten about passing ARCH=arm when calling make. So I went back to my cross compile environment, and navigated to the folder with the Makefile and the basic kernel module code, and ran make ARCH=arm. It built! In a flurry of excitement, I copied it to my BeagleBone and tried to insmod, only to be greeted with: insmod: error inserting 'enable_usermode.ko': -1 Invalid module format

Finally the tar.gz finished copying, so I extracted it to ~/angstrom-build. As root, I symlinked this to a couple places:

ln -s /home/beaglebone/angstrom-build /usr/src/linux-3.2.28
ln -s /usr/src/linux-3.2.28 /lib/modules/3.2.28/build

Then I gave a quick try to building my kernel module, knowing it was likely I’d need to make modules and scripts, but wanting to see what might happen. The attempt yielded this error:
/bin/sh: scripts/basic/fixdep: cannot execute binary file
So I went back to ~/angstrom-build and ran

make modules
make scripts

This took a couple hours and make scripts didn’t seem to do anything. Once this was done, I was able to build the kernel module, however, trying to insert it gave the same issue as when I had tried to insert the cross compiled kernel module. So, I decided to check out what dmesg had to say:
enable_usermode: disagrees about version of symbol module_layout
Uh oh. It seems that even though the numerical version of the kernel matches, something else within the pre-built kernel distributed on the BeagleBone doesn’t match what I’ve built.

So I have two options; the long way being to try to setup the kernel I built on an SD card, or the shorter way, upgrading the kernel on my BeagleBone to see if maybe the kernel-dev package I installed previously can be used once the kernel is upgraded. I chose to try to upgrade the kernel using opkg upgrade. With this selection, I went ahead and did make scripts in /usr/src/kernel before changing the Makefile to use KDIR = /usr/src/kernel, and voila! This time the kernel module built and was successfully inserted!

Suggested course of action to compile a kernel module on BeagleBone

Note that I include directives to switch user (su) as you’ll need to be root in some cases. You may replace /home/user with ~, I typed it out because I noticed that the font made it look like a hyphen sometimes. The command exit will take you back to the previous user, or if you are in the original user who signed in, will disconnect you (so don’t use exit if you only have one user, root). Finally, dmesg will show the kernel prints, so you can look for the text from the example kernel module to be printed when you insert and remove the module.

opkg update
opkg --tmp-dir /home/user upgrade
*reboot if kernel upgraded using: shutdown -r now*
opkg install kernel-headers
opkg install kernel-dev
cd /usr/src/kernel
su
make scripts
exit
cd /home/user
mkdir mykernelmod
cd mykernelmod
*create the below described c file*
*create the below described Makefile*
make ARCH=arm
su
insmod mykernelmod.ko
dmesg
rmmod mykernelmod.ko
dmesg
exit

mykernelmod.c

#include <linux/module.h>
#include <linux/kernel.h>

static int __init enable_mod(void)
{
        printk(KERN_INFO "My Kernel Module is enabled.\n");
        return 0;
}

static void __exit disable_mod(void)
{
        printk(KERN_INFO "My Kernel Module is disabled.\n");
}

module_init(enable_mod);
module_exit(disable_mod);

Makefile

obj-m += mykernelmod.o
KDIR = /usr/src/kernel
PWD := $(shell pwd)

all:
        make -C $(KDIR) M=$(PWD) modules
clean:
        make -C $(KDIR) M=$(PWD) clean

Some useful links

BeagleBone first troubles & resolutions

So, for Fall semester 2012, I’m taking a class focusing on embedded Linux systems using the BeagleBone. I thought it would be worthwhile to jot down the issues I come across and the solutions I find.
BeagleBone
I have an A6 revision BeagleBone. I purchased the “kit” sold at Adafruit because I’m always excited to have extras for the projects I imagine I’ll get around to at some point. :) Also, the kit comes with a power supply, which I wanted to have simply because I didn’t want to leave a computer running simply to power the BeagleBone (seems kind of silly to power a low power device with a high power one, especially when I’d like to have it accessible all the time).

First things first, I perused the Adafruit tutorial and the GigaMegaBlog setup post.

First trouble: Installing BeagleBone Drivers on Windows fails!

I wanted to install the drivers on both a Windows Vista Enterprise 32 bit computer and a Windows 7 64 bit computer. Neither worked. After some searching, the best hint I had was to do with the PID (Product ID), but it took me a while longer to sort out a solution. Here’s what I came up with (as copied from a post to the class messageboard):

The easiest place to get the complete set of driver files is from the mass storage the BeagleBone presents itself as when first connected. Make a copy of the BB:\Drivers\Windows folder on your host machine, wherever you like to organize such things.
Next, open the two .inf files found in Windows\src\files\FTDI\ and do a find and replace: A6D0 -> 6010. A6D0 is the PID for BeagleBone revisions A4 and older, 6010 is the PID for BeagleBone revisions A5 and newer.

Next, make sure to uninstall any drivers you tried already to install. I would also suggest disconnecting the BeagleBone at this point.

The most sure fire way from here, I think, to get the drivers to connect correctly, is to explicitly use the “have disk” option to install. Once you’ve reconnected the BeagleBone, ydevice and select “Update Driver”, then “Browse”, then “Let me pick”, then just skip the selection, then “I have a disk”, then select the modified .inf files in Windows\src\files\FTDI\. I’m not completely sure if there’s any requirement that you select them in specific, but I used ftdiport.inf for USB Serial port, and ftdibus.inf for XDS100v2 compatible USB Serial Converter A & B.

Hopefully this will work for you if you’re trying to get the drivers working with Windows!

Second trouble: Cannot SSH into BeagleBone after restarting!

Having played around, setting up dynamic DNS on my router and checking things out via SSH several times, there was some reason I decided I needed to restart (shutdown -r now). Later, I decided to check in on my BeagleBone and couldn’t reach it! I hooked up the trusty USB cable and poked around, coming up with nothing. A thought occurred to me, and I decided to try to restart DropBear (the SSH service). Lo and behold, it said it found no instances to restart (but claimed to restart anyway). This did not start DropBear, somewhat to my surprise, so I took a guess and tried starting DropBear via:

/etc/init.d/dropbear start

It worked! I haven’t restarted since, so I haven’t sorted out whether it was a fluke that it didn’t start, and I haven’t found anything specific in my searching about why this might have happened. Perhaps an update to this post will be forthcoming.