make it simpler

I just had the thought that we so often get into heated abstract discussions about design patterns, because in many cases (and especially in most small cases) the actual design doesn't matter: the code will work either way. It's only when a program becomes more complex that some designs make extensions easier than others.
Simpler designs make the least assumptions and are therefore the most flexible. And here's the way to create designs that are as simple as possible:

• First, write all your code in a single method.
• Then, extract repeated code into submethods (your IDE probably has a shortcut for this). The scope of local variables is also a great guide on what code to extract into a submethod. The number of local variables in a method is a great estimate for its complexity and coherence. (For example, if one variable is only used at the top and bottom of a method, but not in the middle, then maybe some of the middle code should be extracted.)
• Once you have several methods which all use the same variables (either in their argument lists or by accessing a subset of the object's instance variables), then that's a good sign to extract all those methods into a new class.
• Finally, use interfaces to capture common behavior and use superclasses to extract shared instance variables from different subclasses. In most cases, this will satisfy all your abstraction needs! Abstract classes, super() calls, all those features and almost never needed!

The Template-Method-Antipattern

I had been skeptical about the GOF pattern book ever since I read it many years ago. Many of the patterns in the book seemed so trivial that I was irritated by how much attention they are given. Others have examples that seem overly simplified and never quite fit what actually happens in practice. Looking back now, I find that the premise of the book seems to be the early OO memes of "avoiding repetition" and "finding the right design for the application domain". The latter relates also to graphical modeling and model-driven-design. While those ideas are doubtlessly important, I think that early OO philosophy over-optimizes in that one direction and forgets about another very important direction: keeping the code as simple as possible. Instead of over-designing and already including space for "later extensions", realize that there are usually unknown unknowns and the later extensions might go into quite a different direction. It is the new agile world where running code and automatic tests are more important than fancy diagrams and great designs.

There are many guidelines which help us to write simpler (and thus more flexible) code. Before criticizing the template method even more, I want to remind you the two most powerful ones:

• Number One law of procedural programming: favor pure functions over mutators. (And if you have mutators, separate them from the pure functions.) Note that this law fully applies to object-oriented programming as well!
• Number One law of OO programming: favor composition over inheritance. (And let most of your inheritance be implementations of pure interfaces.)

I have always had trouble explaining why certain patterns were bad (especially those which over-use inheritance), but it was nonetheless very clear to me. I always found it hard to describe succinctly and precisely what the template does without going into the details of inheritance and the subclasses. Other people just didn't need this kind of clarity it seems. But yet, every time I had a debate with someone over a particular and specific piece of code I could convince them that my simpler variant was better in that particular case. So I am right in all cases, but still couldn't give a generally convincing reasoning why this is so.

Now recently I realized that rigorous unit-testing is a great way to validate a design: if it is hard to test, then that's a big smell and motivation to simplify! In a template method arrangement it is definitely hard (even though it's still possible) to test the template and its instantiations separately.

But instead of going on explaining what I find hard to explain, let's hear what others have to say:

• LosTechies: the Template Method is a lie. Precisely points out how TM favors inheritance over composition, just the opposite of what we should strive for.
• Pure Danger Tech: Patterns I Hate #2: Template Method explains it much better than I could! 
• StackOverflow: Do you use Template Method Pattern in programming languages with closures/delegates/function pointers? – the question already contains the best answer: keep your fingers off template methods!

A crucial difference between good software engineers and bad programmers

When I joined another development team this week to observe them working with our favorite design consultant, I arrived in a moment when they were discussing the name of a certain class. Later on they discussed issues like class responsibilities, where to put certain methods, what to mock in a test. I was amazed not just by the well-informed discussions, but also at the actual questions they were discussing.


In other teams and at other times, the questions would be "why does this not compile"? "how do I get this to work"? "how do I do X with (technology) Y?" (instead of: "is Y a good technology to do X?")


Of course, it's the developer's job to get things done, but if we struggle too much with the low-level questions then they won't be any time for really interesting questions of design and actually doing things well.

Next time, I have to choose a team to work with, I will pay attention to this point.

Giving our architecture away

My team of software developers at work was discussing a book when the question of singletons came up. The format of our discussions was to discuss one chapter in each meeting which everybody had read before. During the meeting we'd try to apply the lessons from the book to examples from our own code and projects. We'd also tried to agree on some coding and design standards to apply to our own team work. So far that's great!

One time, the discussion came to the use of Singletons, but it was a very short discussion, because someone said "we don't use Singletons because we do that with Spring". I insisted that in this case, we should discuss some ground rules and best practices for the use of Spring, but out of the entire team nobody was interested in that thought. I insisted one more time, but the crowd then simply went on with the meeting agenda.

At the time of that meeting (several months ago, but still vivid in my mind), I didn't know much about Spring and all its magic and had hoped I would get a quick lesson out of the meeting. Now, I still haven't managed to read a full book chapter or even blog article about Spring (their dryness makes me fall asleep), but I have seen a lot of the Spring code in our project. And while I have never seen good Spring code, the one in our project just seems wrong to me. Too many levels of abstraction mixed and all intertwined. Too much duplication in Spring contexts for tests and production. And that makes simple refactorings (even renamings) much harder.

This week, the topic came up again, because I talked about it with our development coach Steve Freeman, who shares my suspicion about Spring being misused more than it's actually well used. In a discussion I had with one of coworkers right after, my otherwise so intelligent and reasonable coworker insisted that using Spring is always good, because we use code that has been written by experts and is well-tested in many big systems for a long time. My point was, that we don't actually use much of Spring's functionality, but rather we just use Spring XML as another language to express what could be just as conveniently expressed in Java – even with the advantage of allowing better code-surfing (following references), better refactoring, and a better architecture overall.

In short, my impression is that by doing things "with Spring" we just hand over responsibility over our architecture to "the experts" and give up to think about it ourselves. We all agree that "Singletons are bad", but then again, what's a Spring context if not a big uniform sea of intertwined Singletons? My coworker says: "Spring has important functionality like figuring out the order of bean dependencies for you." But then I totally agree with Steve who says: "Doing the bean creation yourself gives you crucial feedback about the design of your code."

Using Spring dependency injection blindly means giving up you architecture. I am not going to do that, because I am a responsible architect.

Buy, don't build. But if you build, also sell.

Here's an additional thought to my last post. The model from which I inferred this rule is Google. Google famously used a variant of the "buy, don't build" rule when they build their data centers out of many standard computers instead of big customized ones. They also use standard Linux and its tools for many tasks. But Google also very often makes exceptions to this rule. There's the Google Collections (now Guava) libraries replacing of standard Java collections, there's GWT to write JavaScript in Java and Closure to compile JavaScript, and recently they went as far as creating Dart as a new programming language to replace JavaScript and SPDY as a new protocol to replace HTTP. 
In each of those cases, Google must have decided that they could do better than the current state of the art, but that's not a point I want to discuss here.

The noteworthy point that should be model for other companies is that in each of the cases mentioned, Google also polished their homebrew solutions and published them for everybody else to use. They're not selling it in the sense of asking money for it, but they're selling it in the sense of providing it with at least minimal marketing and minimal support (thru documentation and FAQs) and by opening a feedback channel for outsiders to report bugs or request features. At the same time, they allow outsiders to join the project in several ways. 
An easy but important form of outside help is when outsiders blog about it (that's more marketing, but also basic how-to style tech support) or answer other outsider's questions on the support forums. Those forums are often mostly there for outsiders to help each other, so that the Google employees can save their time to answer just those questions which they themselves find interesting. 
And then, of course, there's the more involved styles of outside-help when people beta-test software, package it into distributions or builds for different architectures, write better documentation (or even books) about it, report bugs and minimize bug test cases. And finally, there's the really involved ways of fixing bugs or contributing features.

With Guava, Chromium, and even Android, Google has started pretty foundational projects, which few companies could imitate. But I think that in almost every project that produces a lot of custom code (that is, builds stuff, instead of buying it), there is at least a fraction of that stuff which could be factored out and given to the world as a little product of its own. If you look at the plethora of Java Modules which are available in the global Maven repository, or all of Ruby's Gems, or Python's Eggs, you can imagine that many of them have been factored out of bigger company-specific projects. Many of those libraries initially started out very small, but with time, they grew to a certain completeness that now makes them The Standard Library™ for their domain. Examples that I have used in my current job include JodaTime, Awaitility, Mockito, and WebDriver.

Besides being a great way to give something small back to the community, I also think that treating your modules as products in their own right can be a great tool to drive separation of concerns. It means putting all the technical and general stuff into the public module and all the company- or project-specific stuff out of the module. This creates a decoupled architecture in which success of your module in the public market place means that you might get features and community-support for free. It even let's you profit in the opposite case: if a better product comes along to fulfill the purpose of your module, your decoupled architecture will allow you to adopt the improved solution. Win-win!

So that's my rule: Buy, don't build. But if you build, also sell.

Thanks for this post go to Schlomo Schapiro for teaching by example with YADT and LML.

Three silver bullets that have been forgotten

I recently thought about what my ideal programming language would be. Just for fun, I googled "ideal programming language" and read the first five hits. Man, was I disappointed! It seems that nobody who has any imagination has ever written about the topic in a way that brings them high up in the Google results. Basically all five authors described their ideal programming language simply by listing features from existing languages!
Here's a ruff list of requirements for the ideal programming language that I came up with:

• has only a few built-in features so that it con be customized for different domains
• uses the same core syntax as a basis for all kinds of third-party supplied DSLs 
• allows much better tool-supported factoring than any current tool for any current language provides
• allows IDEs to show the code in different views, rearranging methods, even graphical and tabular representations (see also light table and subtext)
• can scale from being a scripting language for a third-party module up to writing your own DSLs and modules to be used by others
• allows literate programming, support for verification, and also tests. IDEs should understand the verification annotations (such as preconditions, postconditions, invariants) to point out possible errors and better support refactoring.
• is explicit. I think that languages with implicit type conversion, implicit defaults, and other automagical things scale badly, because in a larger program, you'll have a mix of implicit and overwritten stuff and need to keep track of when the defaults are used or what else happens "behind the scences". integrating the language with the IDE means that the defaults can be put into new programs and modules by the IDE thus keeping the advantage of quickly starting a project but also gaining the advantage of staying agile by always seeing where everything comes from and being able to change it just right there.

Interestingly, tool support for different views and verification and flexibility to create DSLs all require that the core language has as few features as possible!

Even more interestingly, as I read Fred Brooks famous essay on "no silver bullet" again, I was easily convinced that new programming languages don't actually make that big of a difference in programmer productivity. Basically, there's been nothing really new since Smalltalk and ML! On the other hand, Brooks convincingly argues that the fastest way to write a software is to not write it at all, but to buy it! Or more generally, reuse existing modules! 

While Brook's essay lists a lot of other technologies and methods from which he doesn't expect any significant improvements in productivity, he also mentions three things which actually do have potential in his mind. In short:

1. Buy, don't build.
2. Develop incrementally from prototypes.
3. Find and develop great designers.

Interesting that he mentions incremental development back then in 1986. Agile ain't that new after all! And defining the minimal product hasn't become easier either. It's still one of the essential complexities of software engineering.

Edit: Added "explicitness" in response to Max' comment.

Don’t test your code – test theirs

Nowadays with many technical problems solved by third-party modules we include in our projects, the code we write ourselves is often so simple that unit tests don’t make any sense any more. Of course, we’ll still need automatic system tests (end-to-end tests) and those tests won’t always be green, but when we drill down to find the problem it is most likely that the problem is not in our code, but in our understanding of one of the modules we’re importing!
So how does that pair with “test first”? It simply means that before writing any code depending on third-party modules you explore the third-party APIs by writing unit tests for them which validate your understanding of just those features that you are going to need. And yes, you “test features, not methods”. (The web doesn’t yet know what that means, so you need to find a book such as GOOS to learn about that.)

two rules about testing and a meta-rule about software engineering

Here are two very important rules to write good automatic tests for software:

1. Don't let the tests parrot the code.
2. Focus tests on areas where you expect failures.

Before I go on with this post, I would like you, Dear Reader, to read those rules again, think about them deeply, and answer the following questions: have you ever seen a test that was useless because it violated rule 1? Have you ever worked on a piece of software that had lots of tests, but still let lots of failures happen? In hindsight, did some or many of those failure concentrate in particular areas?

More generally, due the rules seem true to you? Do they seem helpful? Or are they so obvious that they wouldn't even need to be stated so explicitly? Or might the opposite be true: the rules are too general for an ordinary developer to apply them well without further instruction? For example, what degree of redundancy between test code and production code constitutes parroting? How would an ordinary developer know in which areas to expect the most failures?

When I discovered the above two rules my attitude towards them swung wildly a couple of times. First I thought: “Wow, are those rules great. They're going to be the basis for my next book about software quality.” A bit later, I thought: “What silly rules to brag about. Any half-decent programmer is surely always following them unconsciously, just because they make so much sense.” Currently my thinking is: “Yes, yes, both is true: the good programmers follow the rules, even if they couldn't state them explicitly. But the bad programmers (or even the ordinary average wage-slave programmers of our day) will often create bad tests by ignoring just those two simple rules.”

And when I had that last thought, it occurred to me that the same is probably true about most “rules” or “guidelines” in software engineering or, for that matter, in any practical field. The best and most talented people always act in accordance with some unwritten rules and are often not aware of it. It takes other people to formulate those rules and even more people to reformulate them for specific audiences, discuss examples, do one-on-one coaching, ... until every ordinary practitioner of a particular profession has adopt the guideline into his daily routine.

That's the meta-rule:

Many good engineering (or life) rules sound simple and obvious, yet they require good judgement and/or lots of experience to be applied well.

And now, it's time for your comments!

Variations on a Bowling Scorer

Instead of writing a lot about programming as I usually do, this post is made up mainly of code. The background story is that I was terribly offended by a piece of code I found on the web. So I rewrote it, documenting my process and design decisions on the side. I was very happy with the resulting code I got, but never had the time to rewrite the process and design description for publishing. My code, however, is always written to be readable for anybody, so I am publishing it here without any comments.

You'll probably notice that the code which offended me is very clean on the surface (after all, it's written by Mr. Clean Code himself!), but what bugged me was the design of the program and the tight coupling of methods by shared variables. In fact, the coding style reminded me a lot of programs written in C. (A higher-level machine language from the 70s, for those who don't know it.)

Here's Uncle Bob's solution; copied from his long post where he and a coworker derive it TDD style. (Code-highlighting on Blogger seems really hard to do, so for the moment, I need to feed you black and white code. Sorry for that!) Don't worry about the actual purpose of the code. If you want to learn the rules of bowling scoring, you can read them from the declarative program given below!

//Game.java----------------------------------
public class Game
{
  public int score()
  {
    return scoreForFrame(itsCurrentFrame);
  }

  public void add(int pins)
  {
    itsScorer.addThrow(pins);
    adjustCurrentFrame(pins);
  }

  private void adjustCurrentFrame(int pins)
  {
    if (firstThrowInFrame == true)
    {
      if (adjustFrameForStrike(pins) == false)
        firstThrowInFrame = false;
    }
    else
    {
      firstThrowInFrame=true;
      advanceFrame();
    }
  }

  private boolean adjustFrameForStrike(int pins)
  {
    if (pins == 10)
    {
      advanceFrame();
      return true;
    }
    return false;
  }  

  private void advanceFrame()
  {
    itsCurrentFrame = Math.min(10, itsCurrentFrame + 1);
  }

  public int scoreForFrame(int theFrame)
  {
    return itsScorer.scoreForFrame(theFrame);
  }

  private int itsCurrentFrame = 0;
  private boolean firstThrowInFrame = true;
  private Scorer itsScorer = new Scorer();
}

//Scorer.java-----------------------------------
public class Scorer
{
  public void addThrow(int pins)
  {
    itsThrows[itsCurrentThrow++] = pins;
  }

  public int scoreForFrame(int theFrame)
  {
    ball = 0;
    int score=0;
    for (int currentFrame = 0; 
         currentFrame < theFrame; 
         currentFrame++)
    {
      if (strike())
        score += 10 + nextTwoBalls();
      else if (spare())
        score += 10 + nextBall();
      else
        score += twoBallsInFrame();
    }

    return score;
  }

  private boolean strike()
  {
    if (itsThrows[ball] == 10)
    {
      ball++;
      return true;
    }
    return false;
  }

  private boolean spare()
  {
    if ((itsThrows[ball] + itsThrows[ball+1]) == 10)
    {
      ball += 2;
      return true;
    }
    return false;
  }

  private int nextTwoBalls()
  {
    return itsThrows[ball] + itsThrows[ball+1];
  }

  private int nextBall()
  {
    return itsThrows[ball];
  }

  private int twoBallsInFrame()
  {
    return itsThrows[ball++] + itsThrows[ball++];
  }

  private int ball;
  private int[] itsThrows = new int[21];
  private int itsCurrentThrow = 0;
}

Since I felt that mutable variables shouldn't be necessary in the solution at all, I first produced a solution in a purely functional programming language. Here's my solution in Haskell:

example_throws_incomplete = [1,2,3,4,10,10,1,2,3]

to_frames :: [Int] -> [([Int], [Int])]
to_frames (10:xs)  = ([10], xs) : to_frames xs
to_frames (x:y:xs) = ([x,y], xs) : to_frames xs
to_frames _        = []   -- covers empty lists and trailing bonus throws

test_to_frames = assert_equals (map fst $ to_frames example_throws_incomplete)
          [[1,2],[3,4],[10],[10],[1,2]]

-- how many bonus throws are counted into a frame?
num_bonus [10]               = 2
num_bonus [a, b] | a+b == 10 = 1
num_bonus _                  = 0

-- scoring a frame is really simple:
score_frame (frame_throws, next_throws) = sum frame_throws + sum bonus_throws
     where 
     bonus_throws = take (num_bonus frame) next_throws

-- the straight-forward solution assumes that all the bonus throws are present
score_complete_game throws = scanl1 (+) $ map score_frame $ take 10 $ to_frames throws

assert_equals a b | a == b    = "ok"
                  | otherwise = "expected: " ++ show a ++ "\n"
                             ++ "but got: " ++ show b

Given how easy that was, we now have a clear picture of the solution algorithm and design which we only need to port to Java. It's not exactly I one-to-one port, but you'll recognize the spirit of the algorithm.
Here's the code:

import java.util.List;

public class BowlingScorer {
  static class Frame {
    static final public Frame firstFrame(List throwns) {
      return new Frame(throwns, 1, 0, 0);
    }
    public Frame nextFrame() {
      return new Frame(throwns, number+1, nextPosition(), score());
    }
    public int number() {
      return number;
    }
    public int score() {
      return score;
    }
    public boolean isStrike() {
      return throwns.get(position) == 10;
    }
    public boolean isSpare() {
      return ! isStrike() && throwsScore() == 10;
    }
    public int numThrows() {
      return isStrike() ? 1 : 2;
    }
    public int numBonus() {
      return isStrike() ? 2 : isSpare() ? 1 : 0;
    }
    private int nextPosition() {
      return position + numThrows();
    }
    private int throwsScore() {
      return sumThrowns(position, numThrows());
    }
    private int bonusScore() {
      return sumThrowns(nextPosition(), numBonus());
    }
    private int sumThrowns(int start, int count){
      int result = 0;
      for (int i = start; i < start+count; i++) {
        result += throwns.get(i);
      }
      return result;
    }
    private final List throwns;
    private final int number; // number of frame in game, counting from 1
    private final int position; // this frame's first throw in throwns
    private final int score;
    private Frame(List throwns, int number, int position, int previousScore) {
      this.throwns = throwns;
      this.number = number;      
      this.position = position;
      this.score = previousScore + throwsScore() + bonusScore();
    }
  }
  public int finalScore(List throwns) {
    return getFrame(throwns, 10).score();
  }
  private Frame getFrame(List throwns, int number) {
    // requiresThat(number, atLeast(1));
    // requiresThat(number, atMost(10));
    Frame f = Frame.firstFrame(throwns);
    while (f.number() < number) {
      f = f.nextFrame();
    }
    return f;
  }
  /*
  private static void requiresThat(T thing, Matcher matcher) {
    if (! matcher.matches(thing)) {
      throw new IllegalArgumentException(matcher.reason());
    }
  }  
  */
}

Before start giving a lecture on why my program is better, I'll just let you read and find your own opinion. If you want some more food for thought, try to find the one concern that Uncle Bob's solution treats in two different places in two different ways without explicitly mentioning it as a concern. (Had they found the pattern and named it, they'd probably tried to solve it in one place only.)

If you don't have the time to study the code in detail, at least have a look at that one line in Frame(..) which says:  this.score = previousScore + throwsScore() + bonusScore(); I think that this is as close as we get to formulating a really direct and concise specification of the score of a frame. 


Finally, sorry for improvising some of the Matcher / Assertion stuff as I don't have all the proper tools installed right now.

Misused Mock Objects

During the past year at work, I had trouble with mock objects several times. Often tests would break because the mocks were hard to program or to change. In one case, we missed a critical bug because our mock differed significantly from the real implementation.
Since I didn't have any experience with mocking, I didn't know how to improve any tests towards a best practice solution. I found a lot of code smells, but only for a few of them, did I know how to actually make it better. At one point I thought that mocks were more trouble than they're worth.
Fortunately, I also had some good experience with mocks during this year, which led me to the conclusion that in the bad cases, mocks were just badly used. Thinking about it now, some of the tests that I had to deal with seemed to be written under the motto: “Mock everything but the class you want to test.” Thinking about it, a much better motto would be: “Test every component in an environment that is as production-like as possible and mock only if there is a good reason to do so.” In fact, Wikipedia says the same and even lists valid reason for mocking. The valid reasons I personally experienced are the following: speed and simulating rare behavior. There are many instances where we mocked for speed: using an in-memory database instead of a remote one or stubbing calls to remote services. The rare behavior that we simulated was when testing handling of network errors. Instead of simulating real network errors (plugin out cables or intercepting packet traffic??) we just stubbed the interface that does the networking and let it throw a SocketException. That's a perfectly valid reason to use a mock!

While reading the Wikipedia article I noticed the words “indirect input” and “indirect output” of objects under test. This made me think that some uses of mocks might just be needed because of a bad software design. Before taking out the mocking framework, shouldn't we check if the code can be written with direct input and output? For small functionality, input is just method parameters and output is the method result. For bigger functionality, input is given via setters or constructor parameters and output again is the result of the method under test or getter methods called later. For even bigger functionality, some of the input (or context) will be supplied in the form of other objects that first have to be constructed. When there's lots of those objects, tests will obviously not just span the object under test, but also the classes of those other objects. Maybe you think that's too big for a unit test, but I think it's just fine for a big unit test. If those other classes have complex behaviors they will have their own unit tests that will have ran green every time before the bigger test runs. So we don't risk to test too much at once, since the lower layer of the application is already tested.

Feature-driven, design-guided, and tests-in.

I just typed an answer to a new comment on my last post. Apparently it became to long to be submitted as a comment itself, so here I am turning it into a new post. I starts out with some "loud thinking" but ends with some nice insights.

Hi James, thanks a lot for your comment. You're asking just the right questions and those questions help me see more clearly, what "my problem" with TDD is.

Now, ten days after I wrote that post and through your comment, I realize that there's actually a big gap in how TDD is summarized (especially Uncle Bob's version with the three laws) and in how TDD is actually successfully practiced. I find that when the three rules are taken literally (we tried that in some Dojos), then the development and actual design becomes cluttered with detail of each test case and it's less feature-driven as well as less pattern-guided than I would like. On the other hand, if I'm looking at successful agile development with lots of unit tests, then the three rules are just not visible in the process.
I think that maybe two social processes are at work here: on the one hand, good practices spread thru pair programming and people reading a lot of open-source code, but those practices often don't have catchy names. On the other hand, there's a very catchy concept called TDD and very simple "three rules" and people saying that just by following those rules and refactoring, everything else will follow. For example, some people say that good design automatically follows from testability because only loosely coupled systems are easily testable.

So, the reason I wrote this blog post is that the simple, catchy way, TDD is explained, just won't work. It's also just not true that TDD gives you an easy way to tell when you're done. I am currently working on a medium-complex system (roughly developed by a four-person team over two years) with high unit and integration test coverage and we repeatedly had incidents just because we forgot to add something here or there which didn't get caught by the tests. However, our code is simple enuf that those missing parts would become obvious if we just had a final code review after every iteration where we check all production and test code against (a longish) list of the specific level of done for the project. (Which includes error handling, logging, monitoring, etc.) That review is what we now regularly do. Sometimes we find missing things in tests, sometimes we find them in the source, in each case it's easily fixed before going live. So the seemingly obvious things like "TDD always gives you 100% coverage" or "with TDD you always know when you're done" are just not relevant in practice.

My conclusion after working in a "high unit-test coverage" project are that not tests should come first, but the design of very small parts (a method or a small class) should be first instead. The design is primarily guided by the user (caller) of that unit. Design is always finding a sweet spot between a desired feature on the one hand and technical considerations just as available technologies, efficiency, and -of course- testability, on the other hand. I don't think it matters whether you write the implementation (of a small unit) or its tests first as long as you get all tests to pass before you tackle the next unit. (Personally I prefer implementing it first, because the implementation often is a more holistic description of the problem. Only for complex algorithms (which I find to be rather rare), writing tests first seems to give a better start at properly understanding the problem.) By starting with the design (which most often is an interface specification), I find it much easier to think about the method or class in a holistic fashion and also figure out a set of test cases that's small yet covers everything I need. Would you say that this process is still TDD?

Fast tests with high coverage are very important to me, not least because refactoring is very important to me. But I don't like the term "test-driven" because the driver of development is always some external (non-technical) need, such as a feature or some resource-restriction ("make it faster"). Tests are just a technical tool (albeit an important one) and it's the design that creates interfaces which both fulfill customer needs and technical standards. I think of my development rather as "Feature-driven", "design-guided", and last not least "integrated-testing" (because tests are an integral part of the code). Maybe the term "tests-in" is more catchy? As long it isn't "driven...". After all, model-driven also didn't work that well... ;-)

How to write good software and why baby-step TDD is a scam

First off, I am obviously not going to tell you all about writing good software in a single blog post about TDD. Writing good software takes a lot of learning and a lot of practice. There have been countless books written on the subject and since this post isn't about a book list for software engineers either, I'll just mention one to give you an idea: Object-oriented software construction by Bertrand Meyer.
The company I work at has quite a large software development department and quite a good leadership for the latter. Our managers promote autonomy (developers choose the technologies and methods they think are best suited for the work) and learning on and off the job. For example, we have regular (voluntary) coding dojos (practice sessions) where a bunch of developers sits together to solve some simple problems with some new approaches. This is certainly an important part of writing good software.
Recently, we experimented with Test-Driven-Development (TDD), which some people also read as Test-Driven-Design. TDD as my colleagues introduced it to the rest of us is based on the following three rules:

1. You are not allowed to write any production code unless it is to make a failing unit test pass.
2. You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.
3. You are not allowed to write any more production code than is sufficient to pass the one failing unit test.

(Something most proponents of TDD would add is a fourth step to refactor the code while the tests are green, but when TDD is introduced and defined this step is usually not mentioned.)
Our company DOJOs and some reflection upon them have taught me that this is plain bullshit and here's why. In the last two decades, the profession of software development has embraced methods like automated (unit and integration) testing, iterative development, early testing (also called "tests first"), merciless refactoring, design patterns, automated builds and many more. All of those practices are great if done right. Now TDD comes along and claims to condense many of them together into an integrated framework based on the above rules. Going back and forth between tests and code is obviously iterative. Tests obviously have to be automated. You obviously need refactoring, because otherwise TDD will produce terrible code. So TDD dresses itself up as the natural evolution of agile development. But the truth is: TDD is a perversion of agile which over-applies agile principles in a way that doesn't make any sense any more.
Somebody who claims to do TDD either doesn't follow the three rules above or they're doing helplessly bad development. TDD is a scam because it contributes nothing new to the set of agile practices. If someone using “TDD” succeeds writing good code, it is due to the other agile practices, not due to the three rules above. TDD even obscures and ignores a lot of other important methods. SCRUM, for example, tells us to define minimal features and implement them including production code, automated tests, and all that's needed to deploy and run the feature live. SCRUM offers a lot of advice on what a minimal feature is, how to split stories and what's small enuf not to need any further splitting. TDD, on the other hand, splits iterations too much, ignoring SCRUM's advice. Design by Contract tells us how to write minimal interfaces by considering both the needs of the client and the provider and describing the interface succinctly in code. TDD, on the other hand, says that interface should emerge while they instead drown into a plethora of special cases. Finally, testing methods teach us how to design good (and minimal) test cases, get good coverage, and test most where it is needed most. TDD, on the other hand, says nothing about where you start, how to continue, or when you are done. Tests are always green, but when do you have enuf tests?
Think about that: there have been countless example demos of TDD on the internet, on conferences, in practice sessions, but have you ever even seen a small program development finished with TDD? To the contrary, the only thing I see are epic failures. (Thanks, Fred, for the great link!)
So, can we please forget about this exaggerated baby-step TDD, stick to established best practices, and move on writing good software?

Addendum, months later: I saw a good example of TDD in Freeman & Pryce's book "Growing Object-Oriented Software". Their interpretation is much better than the baby-step TDD seen in blogs. The book starts by summarizing established best practice OO design. Their example study is much more elaborate and the problem domain is actually related to the kind of software that professional Java developers are writing for money. If you want to know about the real thing, you have to take the time to read something longer than a couple blog posts.

Refactoring examples: little steps and big smells

My friendly coworker shared a video of Uncle Bob live refactoring some code. Since I love refactoring I was very excited to watch it, but a few minutes into the video my excitement turned into horror, disappointment and anger. Uncle Bob refactors a piece of smelly code, but instead of removing the cause of complexity (namely too many things being done at once), he just spreads the complexity out into many different methods which communicate with each other via member variables. The result looks cleaner and certainly has good naming and short methods, but it still has way too much complexity. And what's worse, with everything spread out in so many pieces, it's much harder to refactor to really simplify it to the core. And what's the worst of worst: even forty years after the invention of such useful principles as "command-query-separation", "separation-of-concerns", and functional programming, Uncle Bob happily violates all those great principles to clumsily cultivate complexity and call the result "Clean Code" and sell it for money. Skip the jump to see the code, good and bad.

Cleaner Code

My team of software developers at work has decided (with some consultation by our team leader) to have a biweekly gathering to discuss a chapter of “Clean Code”. I am on vacation just now and had to miss the first meeting, but I am just reading the book on the train home and here's a little insight I want to share. I am talking about the last example of Chapter 2 in the section "Add meaningful context".

I think that the general strategy of giving a bunch of variables a context by putting them in a separate class is good, so I don't object with the point of the book.
However, I also think that this particular example can be improved in another way, which gets rid of the variables altogether by making the code simpler and shorter.

First of all, the naming of the method is wrong. Most of it is concerned with formatting the GuessStatistics, so I'd rename it "formatGuessStatistics" and refactor the call to print out to the calling method. This will also rid us of the dependency to however the statistics are printed.

Now, let's recognize that the method actually does two things: first, recognize the plural which is applied to all numbers but "1" and results in a different verb and plural "s", and second, replace the number "0" with the word "no". Instead of flattening those two choices into three cases, we should seperate the concerns.

private String formatGuessStats(char candidate, int count) {
    final String number = count==0 ? "no" : Integer.toString(count);
    if (count == 1) {
        return String.format("There is 1 %s", candidate);
    } else {
        return String.format("There are %s %ss", number, candidate);
    }
}

Maybe you'll think that I introduced bad redundancy by repeating the word "There ". I, however, think that such a little bit of redundancy is of no harm, especially since in this case it helps us remove abstraction and see more directly what the code is doing. I also think that the redundancy is only accidental a mirrors redundancy in the English language to which we convert here. If, for example, our PO decides that the singular case should read "There's" instead of "There is", our simplified (yet redundant) variant will be a bit easier to change.

Now, let's look at some further minor improvements of this code. Observing that the "number" variable is only used in the second part, we can move it down into the else block.

private String formatGuessStats(char candidate, int count) {
    if (count == 1) {
        return String.format("There is 1 %s", candidate);
    } else { 
        final String number = count==0 ? "no" : Integer.toString(count);
        return String.format("There are %s %ss", number, candidate);
    }
}

Also we could simplify some more and use the handy "%d" instead of the wordy "Integer.toString". If you are tempted to add a comment to the else-block saying something like "// handle plural case", you can as well factor it out to a second method.

private String formatGuessStats(char candidate, int count) {
    if (count == 1) {
        return String.format("There is 1 %s", candidate);
    } else { 
        return formatPluralGuessStats(candidate, count); 
    }
}

private String formatPluralGuessStats(char candidate, int count)  {
    if (count == 0) {
        return String.format("There are no %ss", candidate);
    } else {
        return String.format("There are %d %ss", count, candidate);
    }
}

Incidentally, this leaves us with code that doesn't contain any local variables any more at all. Given that it is so simple now, we could go back to using just one method and sort the cases in increasing order of "count":

private String formatGuessStats(char candidate, int count) {
    if (count == 0) {
        return String.format("There are no%ss", candidate);
    } else if (count == 1) {
        return String.format("There is 1 %s", candidate);
    } else { 
        return String.format("There are %d %ss", count, candidate);
    }
}

Admittedly we now have reintroduced the three cases from the original code. But isn't it so much more direct and clear?

Which variant do you prefer? The original, the final, or any of the intermediate ones?

PS: When continuing to read the book, I found that some of the principles I used in doing this refactoring are also introduced in the book. Apparently not all of the examples used comply with all the rules given. In particular I got very upset about the use of a parameter for output in a later example and went on to write a long rant about why this is bad and how it can be avoided. Two chapters later, Uncle Bob himself states that this is bad and gave the same alternative techniques on how to avoid the problem. I guess this means that at least Uncle Bob agrees with my own principles of coding... PPS: Bloggers new composition interface almost doesn't suck anymore. Good job, guys! Keep it up!

strategic jamming (how I just played in the Google code jam)

First, thanks to Stefan Schubert for alerting me to the contest. (Also, many thanks to all people who formed ICPC teams with me back in the good old times. What I did today was based on what I learned with and from you.)

I was surprised that the contest should only take two hours, since I know that in the past, I spent many hours working on problems like those from the code jam. But I actually like that it's so short: it encourages to be prepared for it and allows people who don't have much time to compete without much disadvantage. (Others can spend more time on preparation, but not more time during the contest.) After all, the smartest people often have a lot of different interests and are less likely to spend much time just on one thing, especially when they can produce a good-enuf solution in a rather short time.

So keeping this spirit in mind, I first found out, how many points I would need to advance to the next round and then solve the simplest problem which gets me enuf points. In this case, it was Problem C.

As usually, I solved the problem on paper first, then I wrote the code. Also as usually, I struggled with the input parsing part since Java has so many classes and methods for IO I always forget which one to use. Luckily, I remembered correctly which one to use. (I also tried googling "icpc java parse input" and "code jam java parse input", but none of each gave me any usable hints.)

Finally, I initially found two bugs in my program, the first was actually a bug in my test data, since I had produced an inconsistent input. The other bugs was forgetting two lines of code because I got distracted while writing. I found that bug by adding some debugging output, added the two lines and it the program worked on the sample input given in the problem statement. It also worked on the "small" data set of the contest and then I immediately went ahead and processed the "large" input. Let's see if I made it into the next round ^_^.

Overall it took me about 90 minutes to submit this one problem, not counting the time to log in and find the number of points needed to advance (I just didn't see it on the scoreboard initially) and find the problem that I want to solve. (Although that was pretty easy: I just took a problem that was worth enuf points and had the highest percentage of people who already solved it.)

If I do the next round, I will prepare myself a little by collecting some source code for the IO overhead that I can reuse. Also maybe let myself inspire by some contest-specific programming techniques to be found in example solutions.

Before starting I briefly thought about programming in Haskell, but since I have the some IO troubles there and less online resources to detrouble myself, I stayed with Java. Sure, Haskell is way more fun, but the real fun is solving the problem in one's head and on paper.

There are already more than 10,000 participants (still enuf time for more people to join) over 1,400 of which have solved all four problems. The fastest one used only 40 minutes for all four problems!! So I guess when the contest becomes serious and only the 1,000 best participants advance, it won't be for me any more.

Update: I did indeed make it into the next round. In fact, I am among the best 11,000 participants out of 18,000, which means I left 7,000 other very smart participants behind! In case you want to follow me in the next round, my handle is bob406.

literal arrays and lists in Java

In Python and other modern languages, you can say things like:

for x in ['cat', 'window', 'defenestrate']:
    print x, len(x)

In Java, there is no direct syntax for array or list literals. The only thing that Java has are “Array Initializers” that can be used right where an array is declared. Thus the above Python code becomes:

String[] xs = {"cat", "window", "defenestrate"};
for (String x : xs)
    System.out.println(x);

For people who need a lot of literal arrays, this can be a bother, especially when you need small arrays on the inside of expressions. Fortunately, Java's new “variable length argument lists” can help here, because they transform any number of arguments into just.... an array. So this little helper function does the job:

static  T[] array(T... elems)
{
    return elems;
}

Now I can happily write:

for (String x : array("cat", "window", "defenestrate")) 
       System.out.println(x);

Java's standard library even provides a function to create literal lists. Ironically this function is placed in class Arrays although it neither eats nor produces an array.

public static  List asList(T... a)

The nice thing about this factory method is that it provides a list which is backed by an array, so you get a list of fixed size – that saves a little bit of memory and can be useful in some situations. Unfortunately, the documentation says nothing about what happens when you call size-changing methods (like add()) on the resulting list. It's a typical omission of the Java documentation. One has to figure out all the details by oneself. Unfortunately, many Java programmers don't even know that there are such details and then they are surprised when it hits them.

I just did a quick history research. It seems that the helpful function asList(...) exists in Java at least since 1998 (Java 1.2, with the Collections framework). Given that Java really became famous around and after this time, it is really deplorable, that most Java tutorials introduce lists like this:

List ls = new ArrayList();
ls.add(1);
ls.add(2);
for ( Integer i : ls )
    // do stuff

When they instead could write:

for ( Integer i : Arrays.asList(1, 2) )
    // do stuff

Admittedly, the “for ( i : ls )” syntax and variables length argument lists have only been deployed with Java 1.5 in 2004. Curiously, the simplified for loop has become widespread already, while varargs and list literals haven't.

Update: I was about to edit my program to make use of the simplified syntax when I discovered, that Java has anonymous array literals after all. I used Ecplise's “inline local variable” refactoring to get rid of the array dummy and what it produced was:

    return new Object[] {1, 2, 3};

Curious again: the feature is there (probably even since an early release), but people don't know about it!

Why I hate modern Computer Science

First of all, the field should actually be called "Computer Magic", because it doesn't look much like a science. One of the reasons is shown below.

I came upon this blog post which prepared an essay about coping and failure. Now, I am not going to say that computer scientists fail too often, because when working on really hard problems, failure is always part of the game. The problem with much of contemporary Computer Magic, however, is that many things are hard, which actually could be --actually, should to be!-- much simpler.

The author of the above-linked piece explains how for a very simple task, he needs to deal with a lot of computer-internal issues. All he wants to do is, given an already running new mail notification applet, add a functionality that displays the sender of a newly received mail. To do this, he has to deal with five different types of incompatible "strings", only to add:

This simplifies it a bit, but I’m too frustrated to spend more time explaining it.

This should really not be that hard! Given a properly defined interface (API) for the existing libraries, this task would be easy enough for a beginning programmer, not even a CS major, to accomplish. But in reality it needs a well-educated, experienced programmer and even for them, it is still hard and frustrating.

When I started to program, all I had was a BASIC interpreter with an integrated help system and some examples. There was no Google or even Internet access at that time. There were not many functions in the programming environment, all very basic, but well-documented. It was easy and fun to build things on top of that. One of the things we build for school was a little function library for linear algebra and then a simple user interface for that. You could do anything with this environment! Only that the really interesting things would need too much programming, because everything had to be built from scratch.

Nowadays, expectations towards computers are way higher. Instead of a little linear algebra calculator, people would expect an on-line, collaborative learning tool with 3D visualization inclusive. Given how many libraries are available for all kinds of things, this is actually not too hard to do today. But the problem is, that most of the available libraries (and that's across the board, in any domain of CS) are simple too complicated to use, not well documented, and maybe even a little buggy.

Often times, the line between complexity and bugginess is blurred in a frightening way. I think this plays a big part in the field's misery and that's why I'll give you a bigger example to explain it.
Let's look at an imaginary Linear Algebra library, where I can subtract vectors "v" and "w" with the expression "v.subtract(w)". Let's say that I am writing a program that just crashes at this line, and after some debugging I find that during the crash we always have "v == w" and some more testing shows that an expression like "v.subtract(v)" actually always crashes! So, since I am a good person, I will go to the project website for the library and file a bug report. (Spending ten minutes just to sign up for the bug tracking system... I am signed up for so many of them already!)
The next day, I get a reply that my bug was closed and that I should have had a look at the mailing list archives or the web forum for beginning programmers (since there is no other "documentation" than archives from mail, forums, and IRC!) and then I would have seen, that "v.subtract(v)" is wrong code that should actually be "v.subtract(v.copy())".
Then I write back that this workaround doesn't make sense "v.subtract(v)" is the obviously right thing and it should work. With me being so critical of their project, there is actually a chance that the developers will simply ignore my message. But let's suppose that I will get a reply. In this case, the wording will probably be quite upset, saying that " 'v.subtract(v.copy())' is not a workaround, but it's the proper way to do it." And "if [I] knew anything about how the library works, it would be perfectly clear to [me] that I can't pass any object as an argument to its own methods."
Now, I am also getting upset and do my best to reply politely, but firmly, saying that this is precisely the purpose of any library: that one can use it without knowing what's going on behind the scenes.

Do you see the clash of cultures here? I think that the people who do such a bad software design are idiots that create more damage than doing good, because their library has more bugs than features and wastes many people's time. On the other hand, they are probably thinking that I am a stupid idiot who just can't understand that a vector is not the same as a "vector object" and I have to copy the object before passing it as an argument. (But I do know the difference, I am just saying that this is not the best way to do it.)

Please don't ask me what the technical reason for not passing objects to their own methods in this example is. I just made the example up based on similar experiences whose technical details I have thankfully forgotten. 

I think the problem with many libraries, programming languages and other tools is that most of their users just want the functionality and are willing to accept unneeded complexity, probably even thinking that they are just not as smart as the people who made it. And they are revering the people who made it and don't realize that those people are actually way too smart: they create things that are too complex and by consequence waste other people's time.

Greg Wilson writes in an article (PDF) about Free/Libre and Open Source Software's (FLOSS) culture:

The greatest harm done by FLOSS’s cold shoulder is that it chases people away. This is the real cost of any discrimination, including the passive discrimination created by unwelcoming environments. When users (and developers) feel that they’re not welcome, they’ll invest their time elsewhere. What remains is a community that suffers from a lack of diversity and new ideas. Only those who match the definition of a hard-core geek remain, and thus the cycle perpetuates itself.

I think that this statement at least partly generalizes to all of information technology: things here are complex and the people working here love to learn and deal with the intricacies of complex things. People who think that things should be simpler are simply turning away to careers more interesting to them. There are only a very few visionaries who actually understand the complicated things and at the same time work hard to simplify them. (Currently, it seems that Apple and Google employ most of them.)

To finally go back to the initial example of different string types a simple mail notification program has to deal with. Probably some of the complexity of the string types is needed to deal with different character encodings in email. Probably most of the complexity is just historically grown and unnecessary. Only some genius expert knows exactly which is which and how it could be simplified. Maybe even nobody knows all of it, because it all has been written by different people during a long period of time. It's insanely complex and makes your program crash. That why I hate modern Computer Magic.

learn to drive before you build a car, learn to read before you write

Although Computer Science as an education is widely available since several decades, many software-development- and in particular programming-jobs are still done by people with a different educational background. Computer Science (CS) departments are struggling to attract as many students as there is demand for educated Computer Scientists, but it seems that the more students they attract (on top of those who would take CS anyways), the more students will fail during their studies. While recruiters say "we need more graduates with CS degrees", professors whine "we are accepting to many students that have no talent."
In programming in particular there seems to be a divide between programming nerds who just seem to understand programming to a point of completing their assignments, while some others always seem to be estrange to programming and struggle even with simple assignments as soon as some variation into problems is introduced. The question which I want to write about here is: how can we teach programming in a way that helps all students?

I have been helping out with teaching programming since I was an undergrad myself. Seeing how awful student programs looked in the first year I helped with the programming lab, I decided to publish some of the best assignment solutions so that the students would have an example to imitate and an idea of how elegant programming solutions can be. Assignment submissions got much better in the second year!

Since that time, Pascal as a beginner's language has been replaced by Java, and Java by Python, but the problems are still the same. I asked one student with a particularly awful program where she he learned the particular construct which she was abusing there. "Oh, that's from my high school teacher."

When after more than half the course was over, the instructor finally taught students about testing. At that point, I started realizing one of the things that went wrong from the beginning: in our course, students were mostly taught to program little functions, but they were never taught how to use those functions properly and how to test them. I could write an entire blog post on the "test first" paradigm of programming, especially it's important in languages like Python which have no static checking whatsoever. But here, let's concentrate on something more fundamental: the actual understanding of programs, what they do, and how they work.

Most students seem to learn by example and the only official good examples of programs that they see in a typical university course are code snippets small enough to fit on a single lecture slide! Given that material, they are supposed to write entire programs with interworking parts. If I contrast that approach with other disciplines of writing, it just seems ridiculous! In literature, students will read entire novels plus probably some secondary literature, before they write essays of their own. In science, students will read a couple of text books plus tens of journal articles before composing a paper of their own. Furthermore, the first writings of students are usually of secondary nature themselves: summarizing, interpreting, or commenting on some primary work (i.e. an essay about a novel, or a paper about some previously published scientific findings). In programming on the other hand, students will get no reading material, and they have to compose primary works starting from their first labs and assignments!

Here are some ideas how to improve this deplorable situation.

• A large part of the exercises and assignments has to be devoted to reading and understanding programs that have been hand-selected by the instructor as examples of good programming.
• Possible exercises which train this are: writing documentation for a given (substantial) piece of code, writing test cases for a given program, write an example use for a given program module, and finally, extend a given program by a small feature.
  
• Writing of documentation has to be split into external (what does it do) and internal (how does it work).
• Testing can first be done for a programm that works according to its documentation, and as soon as the students can do this well, let them test and debug an incorrect (but still well-designed!) program.

The important part about all this, is that students have to read and really understand a large corpus of existing code that has specifically been chosen for its good design. Students will then appreciate how easy it is to understand, extend, and debug really well-written code. They will also see how the different programming constructs they know from lectures are best fit together.

I don't know how many of our field's education problems this approach to teaching is going to solve. But I am sure this is the way to go!

Python 3000 migration

I think it's a good thing that Python as a language removes some of its deprecated features and old special cases which no longer make sense.
Few programming languages make such a bold move, although more should. A language that grows ever more complex with time becomes a huge liability. Much of this complexity is unavoidable, but the avoidable one should be avoided! Making the necessary non-backward-compatible changes to a language is an expensive investment because the migration of existing libraries and applications becomes a project of its own for each of those libraries and applications. On the other hand, this investment also has an immense payoff in maintainability and extensibility in the long term.

Now, it has been more than six months since Python 3.0 has been released, but a naive search on Google doesn't turn up any migration information yet! Some people seem to have plans for migration, but there's no experience-report yet. Also, there is no list of libraries that have already moved or are planning to move. Unfortunately it seems that no application or library can be migrated until all of it's dependent libraries have done so. Of course, a project could migrate to 2.6, import feature from future and use the -3 compatibility checking option. But it seems that most are still waiting for the "upstream" projects to move.

Here are some references:

• PEP 3000 - "guidelines for Python 3000 development" by Guido van Rossum
• http://peadrop.com/blog/2009/04/05/porting-your-code-to-python-3/ 
• How are you planning on handling the migration to Python 3? - Discussion on stackoverflow.com

After the 3.0 release it quickly turned out that GvR's original advice: "start all your new projects with Python 3000" is unrealistic: the libraries are just not there yet. But what's realistic (and what I will do) is to write all new code with Python 2.6 and make it as much future-compatible as possible. Then the port to 3.x will be completely automatic, once the new libraries have come out.

choose your programming language

they used to say: choose your programming language according to the problem you have.

now finally people start admitting: better choose the programming language that is best for your type of person: http://www.paulgraham.com/vanlfsp.html