28 Apr 2015 by dryobates
Behaviour Driven Development is great enhancement for Test Driven Development. It helps to look on project from user's perspective. Something that we often miss as programmers. In my opinion not all tools that we have are equally helpful in this.
In community there are generally two approaches to BDD tools. One type of tools is RSpec-like tools others are Cucumber-like tools.
Cucumber-like tools have separate feature description file that looks like user story and functions/methods are marked somehow to match those steps. Example from Ruby's Cucumber :
In RSpec-like tools feature description is part of a code. There's no separate pure text file. Instead feature description is mixed with code. Example from Ruby's RSpec :
RSpec-like tools are OK when your don't have to interact with non-programmers. But if your product owner is non-programmer then it more difficult to use RSpec as communication tool as was intended by BDD's authors .
On the other hand with Cucumber-like tools product owner can easily learn how to write a little formalized user stories and send you one file for each feature.
Then the only thing you have to do is to map steps into methods. What's important you don't have to map it verbatim. You can do more broad mappings (one method for many steps) as from your perspective different steps can do the same.
When you use RSpec on your own you can fall into another trap. Because code is so close you'll start thinking like a developer and try to write steps to be a little easier to implement. Then your you start doing architecture driven development or something other - not behaviour driven. It's OK as long as you still do what project owner really wants.
With separate text file you focus to work like a role from your feature file:
You know that format, don't you? "As a <role>" is a key part. We have to start thinking like a <role>. I don't say that you can't do this with RSpec-like tools nor that you won't fall in trap with Cucumber-like tools but the latter make my mind more focused on being scenario writer xor programmer at given moment.
OK. So you know that I prefer Cucumber-like tools. The most often language I use is Python. What tools it can offer?
I've found this:
It offered RSpec-like integration of feature and code but it looks to be dead. This makes me more confident that that RSpec direction is not what we should follow.
Lettuce, Behave and Freshen
Freshen, Lettuce and it's younger cousin Behave are almost identical from outside. Like all Cucumber-like tools they allow you to create separate file for feature. In order to mark your function to be considered as a step they all use decorators.
Example from Freshen's github :
Example from Lettuce's homepage :
Example from Behave's documentation :
As you see the youngest brother does not use that ugly global variable (world in Lettuce, glc, ftc, scc in Freshen) anymore, but it's still quite similar. There are several problems with that libraries.
Integration with frameworks
All those libraries needs quite a lot of code to integrate into a little more complex environments then pure python files. For example to integrate with Django Freshen needs django-sane-testing library . Lettuce has own application to integrate with Django which adds new command (harvest) to Django and adds ability to run it . Behave provides recipes how to integrate with Django. . Freshen is plugin for nose  so if you don't use nose as test runner you can't use it.
All of those tools force you to change your testing habits as their have nothing in common with unittests nor doctests. I have no idea how would it like integrating it with framework like Twisted which being asynchronous have special requirements :/
Other problem with Freshen, Lettuce and Behave is method they need that steps to be declared. Again with simple setup it works great. Create steps file and put steps into it. Problems arise when you have more complies setup. Let's use Django example again.
Consider that you have two applications. In both applications you write feature files and steps in them. What will happened when you have similar step's description in both feature files but you have to run different functions for them in each application? Behave would scream at you saying that you have to change one of the steps! With Freshen and Lettuce is much worse as it won't tell you that you have conflict. It would simply use first (or last) step that it has found.
I understand Freshen, Lettuce and Behave authors motivation. They want to provide some kind of sharing steps in order to complies with DRY rule. In case of Freshen and Lettuce that force me to watch on all feature files and steps in my whole project in order not make some mistake. That's is impossible in big projects.
Behave have a lesson from older brothers and is aware of that problem. But it force me to change steps that were written by product owner. OK, I could append numbers to steps in feature file and explain product owner that it's technical requirement but I don't think that's right way.
Integration with CI
That's probably not so important issue but because you have to adjust your CI to be able to read output from your regular unittest runner (builtin unittest, nose, py.test, trial or whatever you use) and custom output from Lettuce or Behave. They both support JUnit xml files. That's simplify tasks. But you still can't run both your unit tests and BDD tests together.
Morelia took different approach to the problem. Key motivation is not to invent another testing backend but to use familiar for every Python programmer unittests (which is based on xUnit framework commonly found in many other languages).
Example from Morelia's documentation :
Morelia uses standard TestCases and allows your preferred test runner to do the job. There's no problem with integration with frameworks, as you can use it's recommended way of running tests. There's also no problem with sharing steps. If you know how to DRY in Object Oriented program (and we all know because we do this day by day) then you have problem solved. Just call other methods/functions from your steps or use a mixin/inheritance/composition... Huh. We have a lot of tools in OO to DRY :)
Of course there's no problem with integration with CI if you can integrate your current unit tests.
First time I've seen Morelia I knew that it's architecture is a key to solve problem with sharing code in BDD. I'm probably biased to Morelia as after I seen how brilliant is idea behind it I start to hacking it. It still misses some bells and whistles that other mentioned here libraries have but it can be added in future. Freshen's, Lettuce's and Behave's architecture blocks them from solving step sharing problem in easy way.
Maybe you know some other Behave Driven Development tool for Python that also doesn't have mentioned limitations? I'd be happy to test them.
|||BDD as communication tool http://dannorth.net/introducing-bdd/|
|||Integration Django with Freshen http://devel.almad.net/docs/django-sane-testing/index.html|
|||Integration Django with Lettuce http://lettuce.it/recipes/django-lxml.html|
|||Integration Django with Behave https://pythonhosted.org/behave/django.html|