Coding with a Safety Net - Mystery Studio

The need for automated testing

Imagine you have to cross a chasm, and the only way to cross it is walking over a thin wire. If you were offered the option of having a safety net or not, what would you choose? If you're like me, you'll ask for two safety nets - you're never safe enough. Although some people enjoy the excitement of danger, most of us just want to get to the other side safely.

It's a surprise, then, that the vast majority of developers just walk across the wire without a net. Or maybe it's not so surprising; you're an excellent designer and a hell of a coder (we all are, right?); you don't need a safety net. Safety nets are for cowards.

However, the systems we develop tend to become very complex systems, with lots of components interacting with other components. With complexity growing exponentially as the number of components increases, the project becomes more fragile all the time - bugs get harder to catch and easier to introduce. Even worse, when those bugs are found, we tend to fix the effect and not the cause, in fear of breaking everything else. The project reaches a state of unmaintainabiliy, where just making sure everything works is a daunting task, and adding new features is unthinkable.

What can be done to keep code from creeping into that state? Firstly, bugs need to be caught as soon as they are introduced - even before they manifest themselves in production code. Secondly, we can't allow the code to be in a state where making deep changes is a scary thing, because making deep changes is the only way to fix bugs from their roots, and the only way to ensure that the code of an evolving app is evolved code and not a collection of patches.

Ideally, the decision of making deep changes to the code or not shouldn't be influenced by fear. However, refactoring, as these deep changes are often called, requires a great deal of courage to get into; therefore, many opportunities to refactor are lost because of fear. Code quality tends to degrade; it's clear that we need a safety net to dive into refactoring without fearing the consequences.

Enter the Unit Test

We aren't used to testing. We often write a couple of little classes which are so simple that just can't be wrong. More often than not, bugs tend to be in those simple pieces of code that we assume must work. That's our first mistake.

The obvious solution is testing every line of code we write. This is a huge task, and a very boring one, if done by hand. Even worse, whenever something changes somewhere in the code, all the code should be tested again - our second usual mistake is to believe that code is isolated and forget that we are dealing with a tremendously complex system; complex systems tend to fail in complex ways. Who isn't afraid of the term side-effect?

Fortunately, computers enjoy doing repetitive tasks, and they do them quickly. They need just a little bit of help from our part to do all the testing for us.

Testing is better understood with an example. Let's build a Date class - one of those classes which is too simple to fail. However, we won't start by writing the class, but some code to ensure the class will be working correctly :


void testDate (void)
{
	Date pD1(20, 6, 2003);	
	assert(pD1.getDay() == 20);
	assert(pD1.getMonth() == 6);	
	assert(pD1.getYear() == 2003);
	
	pD1.addDays(5);
	assert(pD1.getDay() == 25);
	assert(pD1.getMonth() == 6);	
	assert(pD1.getYear() == 2003);

	pD1.addDays(31);
	assert(pD1.getDay() == 26);
	assert(pD1.getMonth() == 7);	
	assert(pD1.getYear() == 2003);
}


int main (int argc, char** argv)
{
	testDate();
	return 1;
}

It's amazing how helpful is to write calls to a non-existent class. When you design a class with pen and paper or with a modeling tool, it's easy to overlook some details that become obvious when you start writing code. Anyway, that fragment of code won't compile because we still haven't defined the Date class. So let's do the minimum work necessary to make it compile :


class Date
{
public:
			Date			(int nDay, int nMonth, int nYear);
	virtual ~Date			(void);
		
		int	getDay		(void) const;
		int	getMonth	(void) const;
		int	getYear		(void) const;

		void	addDays		(int nDays);

private:
		int	m_nDay;
		int	m_nMonth;
		int	m_nYear;
};

Date::Date (int nDay, int nMonth, int nYear)
{
}

Date::~Date (void)
{	
}

int Date::getDay (void) const
{
	return 0;
}

int Date::getMonth (void) const
{
	return 0;
}

int Date::getYear (void) const
{
	return 0;
}

void Date::addDays (int nDays)
{	
}

Now, the program will compile, but when you run it, the following message will appear :


utdemo: utdemo.cpp:64: void testDate(): Assertion `pD1.getDay() == 20' failed.

Of course it failed, since there's no code to make it work. So we fill in some of the methods :


Date::Date (int nDay, int nMonth, int nYear)
	: m_nDay(nDay), m_nMonth(nMonth), m_nYear(nYear)
{
}
		
int Date::getDay (void) const
{
	return m_nDay;
}

int Date::getMonth (void) const
{
	return m_nMonth;
}

int Date::getYear (void) const
{
	return m_nYear;
}

Running the program again, we see the progress - the program fails, but it fails later :


utdemo: utdemo.cpp:70: void testDate(): Assertion `pD1.getDay() == 25' failed.

Makes sense. We didn't implement addDays() yet. Let's do it :


void Date::addDays (int nDays)
{	
	static int nMonthDays[] = { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 };
	
	m_nDay += nDays;
	while (m_nDay > nMonthDays[m_nMonth])
	{
		m_nDay -= nMonthDays[m_nMonth];
		m_nMonth++;
		
		while (m_nMonth > 12)
		{
			m_nMonth -= 12;
			m_nYear++;
		}		
	}		
}

It's too trivial to be wrong. However, it is. Running the code produces this output:


utdemo: utdemo.cpp:92: void testDate(): Assertion `pD1.getDay() == 26' failed.

A closer look at the code reveals the obvious bug - a classic off-by-one bug. We should access nMonthDays[m_nMonth - 1] instead of nMonthDays[m_nMonth]. Chaning that makes the program run without complaints.

The bug was obvious; however, off-by-one bugs are so common that they have their own name! Without the test, would it be caught immediately? Probably not - finding it would have taken several hours tracking a seemingly unrelated and random bug with, say, the function that checks the expiration time for the demo of your last game.

Moral : Don't assume your code works, no matter how trivial it is. If you're so sure it works, prove it.

Accumulating tests

Let's add another method, this time to subtract days. Learning from our last mistake, we write the test first :


	pD1.subtractDays(31);
	assert(pD1.getDay() == 25);
	assert(pD1.getMonth() == 6);	
	assert(pD1.getYear() == 2003);

Next, the function. We copy, paste and modify addDays() to get this :


void Date::subtractDays (int nDays)
{	
	static int nMonthDays[] = { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 };
	
	m_nDay -= nDays;
	while (m_nDay < 1)
	{
		m_nDay -= nMonthDays[m_nMonth - 1];
		m_nMonth--;
		
		while (m_nMonth < 1)
		{
			m_nMonth += 12;
			m_nYear--;
		}		
	}		
}

Again, a very easy method. Not surprisingly, the first run doesn't work :


utdemo: utdemo.cpp:117: void testDate(): Assertion `pD1.getDay() == 25' failed.

The bug is slightly more subtle this time. First, the month days must be added, not subtracted. Second, the month must be decremented before adjusting the days :


void Date::subtractDays (int nDays)
{	
	static int nMonthDays[] = { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 };
	
	m_nDay -= nDays;
	while (m_nDay < 1)
	{
		m_nMonth--;
		m_nDay += nMonthDays[m_nMonth - 1]; 		
		
		while (m_nMonth < 1)
		{
			m_nMonth += 12;
			m_nYear--;
		}		
	}		
}

This leads to another question - what happens if m_nMonth < 1 before adjusting it? This is the case when subtracting 10 days from january the 5th. m_nMonth is decremented, so the day adjustment will read the nMonthDays at index -1. We write a new test to test our theory :


	Date pD2(5, 1, 2003);
	pD2.subtractDays(10);
	assert(pD2.getDay() == 26);
	assert(pD2.getMonth() == 12);	
	assert(pD2.getYear() == 2002);

Guess what?


utdemo: utdemo.cpp:123: void testDate(): Assertion `pD2.getDay() == 26' failed.

So, in fact, we prevented the bug from appearing in "real" code. We modify the code :


void Date::subtractDays (int nDays)
{	
	static int nMonthDays[] = { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 };
	
	m_nDay -= nDays;
	while (m_nDay < 1)
	{
		m_nMonth--;
		
		while (m_nMonth < 1)
		{
			m_nMonth += 12;
			m_nYear--;
		}		
		
		m_nDay += nMonthDays[m_nMonth - 1]; 
	}		
}

And, finally, we have the code in working condition. Not only the new method we just added, but all of the code. If you were a human tester, would you have tested every single method of each class in the rest of your code?

Moral : Accumulate tests. Run them all every time. When you find a bug in production code, first create a test which reproduces the bug by failing. Then fix the code, so the test passes. Keep the new test - it has become part of your safety net.

What about refactoring?

We have working code, and we can prove it's working. However, it's ugly; there's clearly no need for addDays() and subtractDays(). We'll refactor the code so both functions add or subtract the required number of days from m_nDays and then call one common function, adjustDate(). While we are at it, let's say we want to save some bytes, and store the year as a byte instead of an integer - we'll actually store the years since 1900.

What do you think are the probabilities of doing these changes simultaneously and ensure we don't break anything? The right answer is, "who cares?". We'll do the changes and run our tests. When all tests pass, we'll be sure we've done it right.

First of all, we add a new method, setYear(). Together with getYear(), the two methods will make the adding and subtracting of 1900 years transparent. Then, we make the proposed changes to addDays() and subtractDays(). We end up with something like this:


int Date::getYear (void) const
{
	return m_nYear + 1900;
}


void Date::setYear (int nYear)
{
	m_nYear = nYear - 1900;	
}


void Date::adjustDate (void)
{
	static int nMonthDays[] = { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 };
		
	while (m_nDay > nMonthDays[m_nMonth - 1])
	{
		m_nDay -= nMonthDays[m_nMonth - 1];
		m_nMonth++;
		
		while (m_nMonth > 12)
		{
			m_nMonth -= 12;
			m_nYear++;
		}		
	}
	
	while (m_nDay < 1)
	{		
		m_nMonth--;
		
		while (m_nMonth < 1)
		{
			m_nMonth += 12;
			m_nYear--;
		}		
		
		m_nDay += nMonthDays[m_nMonth - 1];
	}		
}


void Date::addDays (int nDays)
{	
	m_nDay += nDays;	
	adjustDate();
}


void Date::subtractDays (int nDays)
{	
	m_nDay -= nDays;
	adjustDate();	
}

We've done the changes really carefully. This time we can't have screwed up, right?

Beeeeep. Wrong :


utdemo: utdemo.cpp:119: void testDate(): Assertion `pD1.getYear() == 2003' failed.

The very first test failed! We have just catched a side effect of the changes we made : the new code works perfectly, but we forgot to take the 1900 trick into account in the class constructor! (Of course, we should have used get/set methods everywhere; this shows the advantages of encapsulation, which isn't the topic of this article, but we got our bug anyway). We fix the constructor :


Date::Date (int nDay, int nMonth, int nYear)
	: m_nDay(nDay), m_nMonth(nMonth)
{
	setYear(nYear);
}

And all our tests are running again, so we can be confident that the changes we introduced to the code aren't breaking anything. Moreover, we introduced changes to already working code without fearing something might break somewhere else - in fact, it did, but we caught it as soon as it happened.

Moral : Accumulating tests ensures that new code doesn't break existing code, thus minimizing the possibility of introducing side-effects.

Conclusions

We have shown the necessity and advantages of automated testing. The article presents just the general idea of testing; every major language has frameworks that help automate the process.

The key points of this article are the following :

Write tests first. This will ensure you know exactly how the class must behave.
Accumulate tests. Each test is the result of a feature that must work or the result of a previous bug.
Run all the tests every time. This way, you know the code works as a whole, all the time.

Unit testing has its limitations, too. For example, user interfaces and anything that has to do with the visual appearance of the application are extremely difficult to test. However, it's an invaluable tool when writing support code, or when modifying existing code, especially if you didn't write it yourself.

The need for automated testing

Enter the Unit Test

Accumulating tests

What about refactoring?

Conclusions

Further reading