This article originally appeared in the August 2010 edition of GroovyMag, the Groovy and Grails magazine. Parts 6 and 7 are currently available for download from the magazine’s site, and more will come each month. Previous articles in this series are on the Canoo website: Part 1: Eliminate Waste, Part 2: Build Quality In, Part 3: Create Knowledge, and Part 4: Defer Commitment. Lastly, if you like this, you may want to check out some of my older blog posts from my personal site under the ‘craft’ category. Enjoy!
The Groovy Programming Language advertises itself as an “agile and dynamic Language for the JVM”, but what does this mean exactly? This series of articles explains Lean Software Development, and shows how your choice of programming language can make your entire process remain nimble and adaptive. Each month’s article will cover one of the seven Lean Software Development principles and explain how Groovy and the associated ecosystem help eliminate waste, defer commitment, and build quality into your product.
About this series
Groovy is an agile programming language. In order to explain what this means, these articles are structured around the seven principles of Lean Software Development. The previous installments included a short introduction to Lean, explained how an expressive language can eliminate waste in unit testing, showed how the easyb Behavior Driven Development framework and Groovy Builders can build quality into your life-cycle, demonstrated how the Spock Testing Framework can create project knowledge, and discussed how tools like Real Options and dynamic languages defer commitment. In later months we’ll see how Groovy’s Grapes module system, alongside modules like Groovy Web Services, streamline interactions between development and other groups like operations and QA (an important aspect of respecting people). Finally, we’ll look at optimizing the whole by applying Groovy to domains that cross department boundaries: how Gradle improves the build process and easyb serves as a collaboration mechanism between different organization roles. The series will end with a discussion on the best practices of mixing Java and Groovy.
Some of the code examples are basic while others are advanced. The articles explore why the features of Groovy are important rather than the mechanics of any one Groovy feature. Hopefully you’ll leave with some new ideas about how to use Groovy, how to convince your team that Groovy is worthwhile, or most importantly, how to increase your productivity.
Lean Revisited
Part 1 of this series contained a more in-depth explanation of Lean, and I won’t repeat it here. But as a quick refresher, here are the 7 lean principles along with a summary for those principles covered previously:
- Eliminate Waste – Reduce work in progress and half done work
- Build Quality In – Instead of settling for finding defects, optimize your process to create fewer defects
- Create Knowledge – Create fast feedback cycle and encode project information in executable form
- Defer commitment – Delay decisions until the last responsible moment
- Deliver Fast
- Respect People
- Optimize the Whole
Principle 5 – Deliver Fast
The easiest way to deliver quickly is to deliver less. There is a kind of mathematical simplicity to that statement. Short iterations are a reflection of this principle: pick one thing to do, finish it, release it, then move to the next thing. The assertion agile makes is that you’ll go faster over time by continuing to work in small batches. The rhythm of the iteration is the same each time: take a small piece of work, finish it, integrate and release, then start over.
It is clear and logical that developing features in small increments means you deliver sooner, and there is value in that. Delivering a partial system on an earlier calendar date means the customer starts accruing a return on investment before project completion, and this can even result in the system paying for itself before the project is even finished. But will delivering smaller increments really increase your output in the long term? The answer from queuing theory research is a resounding “yes”.
Queuing theory is the study of waiting in lines, or queues, and concerns itself with making your wait as short as possible. It can be applied to software development if we look at our overall development process as a system that takes ideas and visions as input and produces working, tested, and documented software as output. In software production, delivering fast means getting ideas and visions to flow faster through the system, producing working software sooner. Studying how queues work, whether it is supermarket queues, bank teller lines, our your team’s product backlog, has produced several findings that should influence how you schedule and manage work:
- As variability increases, delivery becomes slower – The more people that show up at a bank, then the longer they will each on average have to wait. The more features crammed into your product or iteration backlog, then the longer each takes to deliver.
- As batch size increases, delivery becomes slower – Guess what happens when there is one user story in your backlog that is twice as large as all the others? The same thing that happens when someone shows up in a supermarket express lane pushing a cart overstuffed with goods: processing time increases and delivery becomes slower.
- As utilization increases, delivery becomes slower – If your web servers are operating at 100% capacity then your operations team (possibly you!) should be getting paged so that the problem can be fixed. In a traffic jam, everyone has to wait longer, and the same applies to the development process. You shouldn’t schedule your servers or your team to be utilized 100%. It leaves no time for task switching, maintenance, improvements, or innovation.
There is always more work to do to improve your system, even for teams with a small batch size, steady arrival rates, steady processing rates, and built in slack time. The best way to increase throughput in a system is to find the biggest bottleneck and fix it. Work often piles up when developers hand off features to QA. If so, then it’s time to take more ownership of quality and start writing your own automated acceptance tests. Easyb is an excellent choice for writing Groovy based acceptance tests. If work is piling up in operations, waiting to be deployed to the test environment or production environment, then it’s time to help with project and deployment automation. Currently, Gradle is the best choice for project automation tools, and its features outweigh the more limited scripting abilities of the competitors like Maven, GMaven, or Gant, and the front end to Gradle is conveniently Groovy based.
The worst way to increase throughput in a system is to increase productivity in a non-bottleneck area. If your deployment process has no bottlenecks, then migrating from Maven to Gradle will not help you deliver faster. Improving test coverage in a system with no defects and rework isn’t going to help deliver fast. The only locations in your process that should be improved are the bottlenecks, and even then it makes sense only to work on the worst bottleneck, fix it, and then move on to the next. Addressing capacity bottlenecks can reduce cycle time even in organizations that are already efficient.
I spend most of my day writing code, some of it Groovy and some of it Java. In the Java projects, I’m always struck by how large and comprehensive small changes quickly become, even early on in a feature change or bug fix. In Java, have you ever refactored the public API of a class and then tried to run the tests only to find that a dozen or so of them won’t compile? You need to laboriously integrate your changes into the test files before checking that your changes even work. This always comes as an interruption to my workflow. What I wanted to do was make a small change and test it, but what I got was a huge and unexpected variation in the arrival of goods. It was as if the javac compiler showed up in my supermarket express lane and cut right in front of me with a shopping cart packed full with compile errors. Against my desire, batch size increased, and the arrival of goods (the work I needed to get done) started to vary. Both queuing theory and my personal experience tell me that this slows me down and less work gets done overall.
What I love about programming Groovy is that you can take a small batch approach to your code. To run a Groovy test, the only source unit that needs to compile and run is the test class you’re trying to execute. And even that is an exaggeration: if you try to run a single test method through your IDE then the other methods can contain quite invalid code!
The small batch cycle of Groovy is this: make your changes to the production code; rerun the single unit test you care about. Iterate until your changes are complete and tested, then run the entire test suite, integrating your change back into the rest of the project when needed. Take a small piece of work, finish it, integrate and release, then start over. Small batch testing is better than the alternative, just as small batch iterations are better than the alternative.
Neal Ford once said, “I would rather program Groovy in vi than Java in IDEA.” While I don’t entirely agree with the statement (uh, ever heard of emacs?), the truth behind is that the Flow of Groovy is to have two files open: the production code and the unit test. You don’t need to have any more files than that open at one time, and you don’t exactly need a fancy IDE that allows you to navigate giant software projects with science fiction accuracy. You need two windows, and nowadays they both fit on one decently sized monitor. Of course, with the excellent Groovy support in the IDEs available, there is no reason to switch back to 1980s technology. You can have your small batch and your fancy IDE too.
The Deliver Fast principle complements the Defer Commitment principle covered last month. The faster you can deliver, the longer you can delay decisions. To deliver fast, let the lessons learned from queuing theory guide your process improvements. Reduce batch size, even out variations, and create a little slack time. Then find the biggest bottleneck in your process and apply a little Groovy magic to make it go away.
Sidebar: Little’s Law and throughput
Little’s Law is a simple and intuitive observation with big ramifications. It states: Inventory = Arrival Rate × Flow Time. You are already familiar with this formula from visiting supermarkets. When the line is 5 people long, and the clerk takes 2 minutes to checkout each person, then your wait will be 10 minutes. The Inventory is 5 people and the Flow Time is ½ a person per minutes. Your Arrival Rate is therefore 10 minutes: 5 = n × ½.
Little’s Law implies that you can deliver faster by reducing work in process, or the amount of items in the system. As your inventory diminishes, items flow through the process faster. When lines are shorter at the supermarket, then more customers can make it through the checkout counters. And you will move user stories to done faster by reducing the amount of stories in-process.
Be careful, as this is a simplification. Reducing work-in-progress without making any other changes will also reduce throughput. Small batches, slack, and reduced variability should still be your goals. As H.H. Munro said, “A little inaccuracy sometimes saves a ton of explanation.”
Next steps
Delays are costly. Time spent waiting is clearly a waste: you are by definition doing nothing. Delays, on the other hand, are a different type of waste more akin to lost opportunity cost. You may follow the advice here, and bring a lean software project to market in 6 months. Or maybe you don’t and it takes you 7 months. A naïve way to calculate the extra cost is one month of paying the development team. But what about the cost of not having your product on the market? You must count the cost of lost market share, a later return on investment, and less revenue. Delays have costs (paying the team for a longer time) but they also have expenses (less revenue and market share). With so many variables, decisions are tough.
To simplify decisions, put on your accountant hat and create a profit and loss worksheet estimating costs, expenses, and income. And be sure to make them all in the same unit: dollars, euros, whatever. Next time you are debating whether it is better to deliver in two weeks with an extra feature are now without the feature, simply consult the P&L sheet and see which is more profitable. With an economic model, the team becomes empowered to look after the business’ best interests themselves. And data and knowledge-based decisions provide an easy-out when arguing with passionate product owners and developers. The Poppendieck text [1] provides some excellent templates to follow.
Next Month: Respect People
Fostering pride in workmanship is a key part of Respecting People, and the cultural change of allowing new ideas and languages like Groovy to flourish in your workplace can improve morale, attract new employees, and retain the ones you have. Next month we explore how using Groovy applies to respecting people and get back to some code examples with Grapes and GroovyWS.
Learn more
Several references were used to create this article, and all are enjoyable, informative reads in their own right:
[1] Lean Software Development – Poppendieck
[2] Implementing Lean Software Development – Poppendieck
Need help with agile? Canoo is ready to help. It’s what we do. Call or email any time.
Hamlet D’Arcy Canoo Engineering AG
Hamlet D’Arcy has been writing software for over a decade, and has spent considerable time coding in Java, Groovy, and C++. He’s passionate about learning new languages and different ways to think about problems, and is keenly interested in code quality, testing, and static analysis. He’s a committer on several projects including Groovy, CodeNarc, and JConch, and is a contributor on a few open source projects (including Griffon, and the IDEA Groovy Plugin). He blogs regularly at http://hamletdarcy.blogspot.com and http://canoo.com/blog.
