cleaning a hard-disk before selling or recycling it

I experienced some drama preparing two old laptops for recycling. If you just want to know my lessons learned about permanent data deletion, skip to the end.

My aim way to remove my data in a permanent way without destroying the value of the hard disks for re-use or re-cycling. Both laptops ran Ubuntu and by a friend I had heard of tools like "shred" and "wipe" which overwrite files before deleting them so that they can't be recovered. Both tools protect against the use of ordinary recovery software as well as advanced forensics. For the former, overwriting once with zeros or anything else is sufficient as anyone with basic knowledge about file-systems knows. For the latter, overwriting many times with zeros, ones, and random patters is considered sufficient according to what the tool makers say.
Since I wanted to delete all data, I started out running "wipe" on the block device (the entire partition underneath the file system) of one data partition of my first laptop. It ran longer than one night (during which I slept with ear plugs, because the old laptop's fan is so loud) and did barely finish the first pass of "wipe". Since I don't really need to protect my data against advanced forensics, I went looking for quicker ways to delete the data. Some sources say that "formatting the drive" is good enough for deleting data, but I personally don't believe so, especially if after formatting, the disk is left blank. (I put "formatting" in quotes because those sources don't distinguish between "formatting a partition" and "repartitioning the disk" which are quite different things to do.)
I don't know of any tool which  easily reverses formatting or repartitioning, but I know how easy it was to recover almost all my data from a broken disk which wouldn't boot the computer any more. Since repartitioning and formatting also usually only changes some sectors of the disk, but leaves most of the disk intact, it is quite similar to a partly broken disk. In practice, installing a fresh OS on the repartitioned disk could disable most of the easy recovery tools, but since I didn't plan on installing anything, I didn't think about how safe this road is.
Thus, I didn't deem it sufficient to just repartition the disk, googled again, and found that the age-old "dd" tool easily does the job. And it's included even in the smallest linux-live-distro. So here's what I did with the first laptop:

1. Boot from external. Nowadays that's USB or even SD. I still had an old live-CD lying around which also did the job. (Won't be an option for my new computers, though, since neither of them has CD drive and I am going to throw away all my CDs when I next move house.)
2. If your external drive is slow or the live-distro you're booting has trouble with drivers, don't even bother to boot into graphical mode. Instead find the option to boot into a terminal and use it.
3. "swapoff -a" just in case your live-distro finds and uses the swap partition of the disk you're going to erase. (Actually happened to me. Could have been a bad surprise.)
4. run "fdisk -l" to see all the disks. With my oldest laptop, there was only "hda". With the other, it was "sda". Also use "mount" to see which device is your external disk, so you don't erase that. ;-P
5. (If you have more than one disk and don't want to erase them all, you might need to mount them to be sure which one is which.)
6. run "dd if=/dev/zero of=/dev/hda conv=noerror" replacing "hda" with your actual device name. I added the "conv=noerror" to ignore write errors so that it continues overwriting everything that still works. It would be too bad if the process stopped because of one bad block in the middle and I had to redo it or --even worse-- wouldn't notice it at all!
7. As usual in the Unix world, there's no fancy message telling you all's done and well. So you have to know yourself, that this is, in fact, the happy end.

My second laptop made much more trouble, since the live-distro I had would boot fine and then just not recognize the disk drive at all. Then I tried to boot from a USB stick which booted into its boot menu, but just after loading the kernel couldn't find the USB device to mount it, so I was left with a "kernel shell" which only supported lsmod, modprobe and the like, so I abandoned that, too. Finally I tried to boot from an SD card, but this wasn't even recognized by the BIOS. (On my newer laptops this works, though.)

So here's how to do it ff you can't boot from external (or don't want to spend the effort to make it work):

1. Boot to a root prompt so that /home won't be needed. (On Ubuntu, I don't even know how to to that, so I took care to delete everything, but leave my empty home directory intact.)
2. Remove all data partitions and the swap partition from /etc/fstab. Run "swapoff -a" and "umount /datapartitions".
3. run dd as above for all the unmounted partitions
4. if you still have data and temporary files on the remaining (OS) partition, delete it with "rm". that's the hardest part, because it requires deciding where there could be private data that has to be deleted and where's the OS files that are still required. 
5. finally, do "dd if=/dev/zero of=/remaining_partition/bigbigfile" for each remaining partition.
6. Again, no happy end message, but that's it.

I found that dd into a file is actually faster than dd into a partition (just my laptops, YMMV), but I find that overwriting the entire partition is safer, because the file-system-internal areas (directories, journals, whatever modern stuff) also contain some private data. That's why I usually take care to have separate data partitions. Also for backups and stuff.

Lesson learned: The "how to securely remove your data" articles on the internet sound so easy with steps one, two, three, yippie, but in practice things get considerably more complicated. And again, as so many times, I am using a lot of technical knowledge about unix just to accomplish simple things. This just indicates to me that technology still has a long way to go...

Notes on suicide

The recent case of Aaron Swartz brought a topic up into public discussion, which I have read and thought about quite a bit since my university times. It's an unfortunate property of public discussions on such topics that they become very emotional and come to quick conclusions. There are those who say: "what a cruel society which pressures bright young people like him into such terrible situations" while others say: "what a stupid guy who plays with fire, but is not willing to take the consequence of his actions". Some blame it on the circumstances, some blame it on the person. But I think that suicide is just a quirk of nature. Some psychological defenses that misfunction. And I think, we can easily prevent suicide by tricking around those defenses. Ready to hear the full story?

As far as I learned from Wikipedia, Aaron had an offer to go to jail for six months in order to avoid a trial for his alleged crime. (And, no matter what you or I think about whether information piracy should be a punishable crime and if yes how hard, this proposed sentence is comparable with previous cases which Aaron or at least his friends probably knew about...) My first thought was that six months of prison doesn't seem like such a bad deal. Many famous (and innocent) people have been in prison (I think especially of Nelson Mandela and Deng Xiaoping) and used the time well for reading and writing and strengthening their personality. But this was probably not the way, Aaron saw it.

So what was going on in the mind of Aaron Swartz? Since I don't know him, other than being a big fan of his study on Wikipedia editors, I will answer a more general question: what goes on in the mind of a person who decides to end their life? Here's the first misfunction of human brains: they don't take things at face value, but often add a ton of symbolic value and interpretation. For example, someone who's bound to go to jail might think about all kinds of awful things that happen in jail which can trigger a hugely exaggerated emotional reaction in that person. But what's worse, if you think you have to go to jail although you're innocent, you might lose faith in the entire system of justice and therefore (since the emotional part of the brain likes to exaggerate and generalize) the entire country and/or society. Following that logic, you might not see any reason to stay living in this country or this world. Trained minds can diffuse such logic by taking an outside view of the situation. To an outside observer, one case of unjust detention wouldn't seem to be a surprising (albeit still regrettable) event given a small percentage of every prison contains people who haven't deserved their sentence. (Just as there are a lot of uncaught criminals, too.) We already know that justice is imperfect. (And looking at cases like Guantanamo, we also know that US "justice" is sometimes genuinely evil.) I believe that a trained mind can notice when its emotions are overboiling and it can give itself a rest to try and take on a different viewpoint.

If it is so easy to work around this particular bug of the human brain, you might wonder why none of Aaron's friends and close ones noticed his precarious state and intervened to bring him back to the bright side? Now comes human brain bug number two: emotional arousal and the feeling to be betrayed makes people distrusting, which sometimes can even affect friends and close ones. How many times have you seen someone giving well-meant advice only to get this exact reply: "You don't understand!". Unfortunately, even the most well-meaning friends often give advice without listening enough and asking the right questions which will permit the affected person to speak their unspeakable fears. And also unfortunately, an affected state of mind will assume that "if you don't ask, you don't want to hear" and maybe even think "if you would hear, you wouldn't believe". It takes a little bit of psychological manipulation to make such an affected person feel at ease and be able to open up. And what's worse, someone trying to help must overcome their natural tendency to be offended by a harsh rebuke of "you don't understand" and instead recognize it as a signal, that something important is going on. I think that most people could -- by practicing meditation, empathy, and compassion -- at least recognize that something is wrong with someone and ask a mentally better trained friend for help. The thing we have to learn and accept is that our natural instincts and reactions sometimes make things worse and everybody who's interested in their own mental well-being (as well as that of their friends and family) should look into practicing self-awareness to better recognize their own mental state. We all have defective brainware -- and we all have the capacity to overcome our deficiencies and strive to become part of Humanity 2.0 where all people are well-intentioned and make the best out of their strengths and weaknesses...

PS: My condolences to all those who knew Aaron. With my post I certainly don't want to judge him or his close ones. All I am trying to achieve is to raise awareness that we might all fall victim to our brain bugs (although for most the results are not as severe) and that we should train and prepare ourselves to survive this life better. And I am sorry for not being able to give more precise advice. I am not an expert at all. Just a bystander who recognizes a certain pattern and tries to offer a more constructive point of view.

The future of inner-city transport is in the sky

Building new transport underground in major cities is prohibitively expensive:

• USD 17bn for 13.7 km of Second Avenue Subway in NYC, although several kms of tunnel have already been contructed in the 70s. http://nymag.com/nymetro/news/features/n_10109/ (March 29, 2004, according to Wikipedia)
• BP 16bn for 22 km central tunnel section of London Crossrail (Leftly, Mark (29 August 2010). "Crossrail delayed to save £1bn". The Independent on Sunday (London). according to Wikipedia)
• EUR 500m for 3 km of Stadtautobahn (only partly under ground) Gericht macht Weg frei für die A 100 – Wowereit begrüßt Urteil. Der Tagesspiegel, 10. Oktober 2012, abgerufen am 10. Oktober 2012.

On the other hand, the flying car has long been assumed as part of science fiction, yet there are no known technologies which make flying and landing over cities economically feasible for many:

• Helicopters require high amounts of energy just to float in the air, without even moving. Besides, they create lots of noise and wind and require too much safety room to fly.
• Planes are less inefficient, but only at high speeds. Most city-trips are shorter than a plane needs to get up to speed. Ultra-light (one-person) planes are very sensitive to wind and still require more space than available to start and land.
• Magnetic levitation also requires vast amounts of energy just to stay afloat. Besides, it can only float some mm or cm high, such that it still needs a full elevated roadway (such as a mono-rail).

Speaking about elevated roadways, those are actually much cheaper than tunnels be it for trains or for cars. However, elevated roadways have the disadvantage of carrying noise very far and creating so much shadow below that the road underneath feels like a tunnel. It is for this reason that elevated roadways (be it for trains or for private cars) are currently avoided in most western cities. Some are even torn down to be replaced by tunnels.
Thinking about the flying car, we have to ask ourselves how it is to be designed to avoid the pitfalls of elevated roadways. Number one: it has to be silent. Number two: it should not cast much shadow. As it turns out, we can grant both wishes economically using current technology. In my next post, I will introduce a means of airborne inner-city transport which is less noisy than the street traffic below and which casts not more shadow than a small multiple of the shadow cast by the people using it.

A crucial difference between good software engineers and bad programmers

When I joined another development team this week to observe them working with our favorite design consultant, I arrived in a moment when they were discussing the name of a certain class. Later on they discussed issues like class responsibilities, where to put certain methods, what to mock in a test. I was amazed not just by the well-informed discussions, but also at the actual questions they were discussing.


In other teams and at other times, the questions would be "why does this not compile"? "how do I get this to work"? "how do I do X with (technology) Y?" (instead of: "is Y a good technology to do X?")


Of course, it's the developer's job to get things done, but if we struggle too much with the low-level questions then they won't be any time for really interesting questions of design and actually doing things well.

Next time, I have to choose a team to work with, I will pay attention to this point.

Giving our architecture away

My team of software developers at work was discussing a book when the question of singletons came up. The format of our discussions was to discuss one chapter in each meeting which everybody had read before. During the meeting we'd try to apply the lessons from the book to examples from our own code and projects. We'd also tried to agree on some coding and design standards to apply to our own team work. So far that's great!

One time, the discussion came to the use of Singletons, but it was a very short discussion, because someone said "we don't use Singletons because we do that with Spring". I insisted that in this case, we should discuss some ground rules and best practices for the use of Spring, but out of the entire team nobody was interested in that thought. I insisted one more time, but the crowd then simply went on with the meeting agenda.

At the time of that meeting (several months ago, but still vivid in my mind), I didn't know much about Spring and all its magic and had hoped I would get a quick lesson out of the meeting. Now, I still haven't managed to read a full book chapter or even blog article about Spring (their dryness makes me fall asleep), but I have seen a lot of the Spring code in our project. And while I have never seen good Spring code, the one in our project just seems wrong to me. Too many levels of abstraction mixed and all intertwined. Too much duplication in Spring contexts for tests and production. And that makes simple refactorings (even renamings) much harder.

This week, the topic came up again, because I talked about it with our development coach Steve Freeman, who shares my suspicion about Spring being misused more than it's actually well used. In a discussion I had with one of coworkers right after, my otherwise so intelligent and reasonable coworker insisted that using Spring is always good, because we use code that has been written by experts and is well-tested in many big systems for a long time. My point was, that we don't actually use much of Spring's functionality, but rather we just use Spring XML as another language to express what could be just as conveniently expressed in Java – even with the advantage of allowing better code-surfing (following references), better refactoring, and a better architecture overall.

In short, my impression is that by doing things "with Spring" we just hand over responsibility over our architecture to "the experts" and give up to think about it ourselves. We all agree that "Singletons are bad", but then again, what's a Spring context if not a big uniform sea of intertwined Singletons? My coworker says: "Spring has important functionality like figuring out the order of bean dependencies for you." But then I totally agree with Steve who says: "Doing the bean creation yourself gives you crucial feedback about the design of your code."

Using Spring dependency injection blindly means giving up you architecture. I am not going to do that, because I am a responsible architect.