Migrating to Raspberry Pi

Prelude

This is my first blog post. Oh my, what a responsibility. But more importantly—there is still a lot to be done before this post renders itself available to a larger audience. From the top of my head,

• Choosing which format to use for the source file of this very post. I am comfortable with LaTeX, something akin Markdown or just plain XML. Right now, this sentence is being written surrounded by XML and, I am coming to conclude, this is too much boilerplate code for blogging. Switching to LaTeX.

• Registering a domain name. Too bad .dude top-level is not available. Then again, how would conflicts be resolved? Who would judge who is a dude and who is not...

• Getting a VPS with Arch Linux, installing Apache. I do sincerely agree this sounds an overkill for just a blog post, but what geek would not opt for a near ultimate control over his website?

• Setting up a website, which can be as easy as borrowing the Wikipedia engine, or as easy as writing some PHP, Python or Ruby code myself. Fair enough, I never wrote anything in Ruby, so that will be a bit more difficult. But who wants easy? No, no, let us have some fun and do it in Lisp. (A remark from the days to come: learning Lisp by writing a custom .tex parser is not an easy undertaking.)

• Actually writing the post itself, which is not about Apache servers and Lisp but which is about a more mandane topic of disposing of one very old PC (the local domain name is “dinosaur”) and one old notebook, and switching to a Raspberry Pi setup instead.

What is Wrong With The Old Notebook?

Well, my old notebook is running Linux, Arch flavour, but besides, if I am completely honest, there is nothing wrong with it. The HDD is making some funny noises but S.M.A.R.T. says the HDD is healthy and there is a recent backup (times two). In any event, the cheapest and the easiest course of action is to replace the HDD, maybe even go for an SSD for a quieter operation.

The situation is not this simple, however. There is also competition on my desk—a newly acquired Mac Mini is shinier and I tend to use it more often as a result. For once, I am writing this post in Aquamacs. But, I do need my shot of Linux hacking every now and then. So, as it stands now, there are two keyboards, two mice, and two monitors. That is two items too many :)

One brilliant idea that I sympathise with is to put my notebook in a drawer and simply access it from my Mac via terminal. (As far as I have picked up on forums, X forwarding is slow and VNC is not ideal either, so graphical interfacing will have to wait till better times.) And if the notebook is going to sit in a drawer, it might as well become a Raspberry Pi, right?

Shopping List

The prices are from http://amazon.co.uk.

So, Raspberry Pi—essential. Aluminium case? Men also buy jewellery. I guess a memory card needs fast random read and write times, and EVO has good reviews from the folk using it in smartphones, so I am giving it a try. So far so good, but the last two items stole a day from me.

How difficult can it be to buy a usb powered hub? Just select the latest version, 3.whatever, estimate the required number of ports and find the cheapest one on amazon with not too many 1 star reviews. Eh, not so simple it turns out to be.

USB 3.0 is backward compatible with 2.0 but not 100%: low speed and full speed devices do not work with Raspberry Pi via a USB 3.0 hub. Ok, ditching 3.0, opting for 2.0, reading on. USB peripherals can be power hungry and Raspberry Pi cannot supply a lot of power. Fine, choosing a hub with external power. By the way, do I buy a power cable for the Pi separately? Ah, it’s powered by USB, but not all USB hubs are powerful enough for the job. And one should pay attention to avoid backfeeding, while other people seem to use backfeeding to power the Pi. Argh! Half a day has passed by now, it is high time to choose something. I am choosing the fanciest usb hub possible as a result, with some prominent marketing that it is Raspberry Pi compatible.

Moving on to external HDD. Or is it going to be external SSD? No, still a tad too expensive considering all the other components. Western Digital, Samsung, Toshiba, Hitachi, Transcend? All seem popular on Amazon. Linux folk complain about not being able to reformat WD due to smart firmware, about Samsung not spinning down properly, etc., but mostly there seem to be solutions. I don’t care, let’s just try. Wait, what are all those one star reviews? Let’s check them out. Hard drive failure, hard drive failure, hard drive failure, ... Hmm... Here is a table with a proportion of one star reviews for some of the more popular external HDDs.

So, hard drives fail occasionally. And the earth goes around the sun. I should not be surprised, I guess. Still, having read through all those reviews of drive failures and of priceless “movies” forever gone I have decided to go for a RAID 6.

With Raspberry Pi, everything sits on a single usb 2.0 bus. If I access external usb storage via Pi’s ethernet, I should be able to get min(35/2, 12) = 12 MB/s read speed and min(35/7, 12) = 5 MB/s write speed (3 reads and 3 writes penalty for RAID 6 plus 1 transfer). I have about 100GB of files on my old notebook, so we are looking at 6 hours of copy time! I recall it took me less time to copy Doom from a friend, back then when I was young. Welcome to the stone age. But then, 12 MB/s should be fine for streaming video and 5 MB/s is faster than my internet. And where else do we download stuff from but internet these days?

To achieve the above speeds with a RAID 6 the media should be capable of 3 MB/s read speed (assuming 4 drives) and of 5 MB/s read plus write (e.g. 10 MB/s each). What does it mean? Cheap flash drives are off the list. After comparing various options, I am gravitating towards buying three SanDisk Extreme 64GB, which together with a microSD of the same breed will give me a RAID 6 installation. Hopefully, it might be even a tad faster given that the microSD sits on a different bus. We will see.

The Goodies

The goodies have been delivered

The goodies have been assembled

First FLOPS

Does it work or does it not? This is the question for this evening. Getting Arch Linux ARM, partitioning the microSD card, copying the filesystem, inserting the card into the Raspberry Pi, switching on the power... The lights are not blinking. It does not work. Getting a cup of tea.

After some head scratching, it turns out that I needed to mark the boot partition as W95 FAT32 (it was marked as Linux). This was the only glitch and the Pi is working now. Checking DHCP leases, logging in via SSH.

Benchmarks

I have run random and sequential read and write tests on both the SanDisk Extreme 64GB microSD card and one of the SanDisk Extreme 64GB flash drives. I have used fio for this purpose with the following parameters.

The results are summarised in the table below. The table also lists the results with a RAID 6 running on the microSD and three flash drives (more on this later), as well as transferring 2,894 photos of various sizes (2.7 GB in total) via SMB to a dm-encrypted volume that seats on top the RAID array.

The numbers look like 10 years old hardware and the actual usage scenario—the last row—gives even slower read and write speeds than fio tests. This is partially due to the 100 Mbit/s ethernet, but dm-encryption, SMB, or the way that fio measured speeds in other tests could also be contributing factors. That being said, the actual speeds are only twice as slow as the theoretical maximum given this setup (5 MB/s write and 12 MB/s read), so I am reasonably satisfied. I’ve also rsynced 70+ GB at some point (all sort of files) and got an average write speed of 3.9 MB/s.

Now, let us take a look at the dark side. While doing random write tests with the microSD, the system froze a few times for a minute or two. I didn’t have such experience with the flash drives. The system also crashed twice completely while I was filling a part of the RAID array with random data. I think these are related accidents. Finally, once the samba server was running, if I listened to music from the Pi and simultaneously wrote a file to it, I would experience delays in music. My suspicion lies with the microSD card. I am planning to drop it from the RAID array one day and check if the performance is any better just with the flash drives. (By the way, besides the accidents with filling 100+GB with random data, the system is stable and I had no further crashes.)

RAID 6

I made some notes for this section but then the life caught up with me and now that I am back half a year has passed. Let us see how much I recall.

The idea was to have the whole Arch distro with the exception of the boot partition on top of a RAID array. So a separate boot environment was required to activate the mdadm array before mounting the root filesystem. Arch makes it very easy with the mkinitcpio scripts. Normally, arch generates separate kernel and initramfs images, which is easy with grub but more difficult with the capricious Raspberry Pi bootloader. In config.txt you need to specify “both the ramfs filename and the memory address to load it at.” Now, this second part might be obvious to the developers and some hard-core system administrators but I was clueless. And, as is common, Google provided conflicting answers.

So, what did work for me? My kernel image size is 5,670,360 bytes and my initramfs size is 9,680,492 bytes. Specifying

did the trick. Also, no “initrd=...” kernel boot option was required. When everything works, I get the following records in my syslog:

There are two ways to prepare a mdadm array on which the root is mounted. Option one, plug all the USB sticks and the SD memory card into another computer, make a RAID 6 array, copy all the necessary files to the array, plug all the pieces back into Raspberry Pi, boot up. I really do not know why I did not follow this easy way. Option two, plug all the USB sticks and the memory card into Raspberry Pi, boot up, create a RAID 5 array with the 3 usb sticks, copy all the system files to the new array, change the root mounting point in cmdline.txt, reboot—the system is now loaded from the RAID 5 array—add the SD card to the array and start reshaping the array from RAID 5 to RAID 6. Drink tea for 146 minutes and contemplate about your silliness and the days wasted on hacking.

Postscriptum

Half a year since I started using my setup, one of the disks—namely the SD card—started to show up as “removed” for no apparent reason. I have simply re-added it to the array and everything works fine again. All in all, I have a feeling that adding the SD card itself to the array was not the brightest of ideas.