17 May 2015 by dryobates
Python is powerful dynamically typed language. We can benefit from it's dynamic nature doing a lot of things difficult to achieve in statically typed languages. But that nature has it's price which grows with a project. Unless you resolve into technics learned from statically typed languages.
What I talk about are of course formal interfaces. I see your grimace on your face. "What is he talking about? We have duck-typing" you'll say . Yes, with all my heart I am for duck-typing! But at some complexity level duck-typing is not enough.
In company I work we have over 800 KLOC spread across 50 projects and 3 dozens of libraries. It's quite a lot of code to maintain. Now consider that one library is quite generic (as it should be :) ) and provides service that acts on any object that has given set of methods. That library is used in most of our projects.
For the purpose experiment let that object expected by library have given methods:
So any object with this 3 methods can be passed into our service. Now let our service looks like this:
And we use it this way:
If "method1" is absent in expected_object we would like to know it before we do those heavy things. Ideally at service creation time. So let check for "method1" in init part:
We had to sacrifice duck typing. Holy Coding Purity please forgive us but "practicality beats purity"  Wait a minute. We need to check for more methods. We expect also "method2" and "method3" later in code. Let's check for that:
Huh. That looks to become not quite practical...
OK, let's try something different. We have unit tests. Let us check correctness in unittests:
OK, it works! So let us write similar tests for all other 50 projects...
Abstract Base Classes
When we'll finish the obvious thing will happen. "In this world nothing can be said to be certain, except death and taxes" once wrote Benjamin Franklin. Well in software's world nothing can be said to be certain, except change in functionality...
What if our library would expect one more method from passed object? So how to detect which objects needs to be modified and gain "method4"? If we ran tests it won't detect missing methods. Maybe we should use grep? But if method is created dynamically or aliased?
Searching in 800 KLOC doesn't look to be funny thing :/
In order to cope with a problem of object behaving as we expect, in PEP 3119 appeared Abstract Base Classes . I encourage you to read rationale in this PEP to get a feel why it can be helpful. Now, with a new tool, let us rewrite our library and tests:
Now if ExpectedObject has missing any of methods marked as abstractmethod in parent class then we would get an exception at time of objects creation:
One important thing you should note is that we still use duck typing. Abstract Base Classes help library clients to create compatible object interface, but they aren't enforced. You can still pass objects which do not inherit from ExpectedObjectBaseClass and it will still work unless you provide expected methods.
Abstract Base Classes and Django
OK, so we have ABCs and it looks like a great tool to keep our large code base coherent. Most of our projects are Django-based. Let's use ABCs on Django models:
Now if we create model:
Wow! Something went wrong. Django models use metaclasses too and it conflicts :(
So we could use adapters:
OK that would work but... if I made a mistake and do something like this?
Oh no... It's dead end :(
Isn't there any other method to use with Django models? Well we always use veteran on the field of Python's interfaces: zope.interface .
If you think I'll encourage you to use them on runtime then don't be afraid. I won't. As I have written before: with all my heart I am for duck-typing. Saying that I consider zope.interface  as convenient piece of software that in conjunction with tests can help you keep your projects easier to maintain.
So let's try with it:
With above solution we don't affect runtime behaviour. We use duck-typing and the only use for interfaces is for documentation purposes and pre-deployment checking. We could write such testing functions by ourselves but Zope's community did it for us.
Now that you see the point of using some kind of informal interfaces in Python I have to tell you that... I have lied to you a little :)
You see I have intentionally use wrong first test as an example. I have written this test:
Do you see a problem with above test? What object's behaviour it tests? None! Tests should check object's dynamic behaviour. That's why they are used even in statically typed languages. Above test checks for object's interface which has no value in running system. Important is not like it looks but how it behaves.
If we had a 100% tests coverage we won't have to check if object has correct methods or not. If something would change our tests would fail and point us error location with surgical precision.
Interfaces are great for documentation. It's very convenient way of telling "I need this" and as such ABCs are used in standard library.
Besides that in interfaces can be interested for:
- programmers coming from statically typed languages - switching to think in dynamically typed categories can be difficult and... I'm not sure is it always desired as new fresh ideas can come to Python from that world.
- lazy programmers that don't want write tests ;)
- for those which have a lot of legacy code and interfaces are one step forward to get this legacy fully testes in some future.
With over 10 years in Python professional programming I can't explain myself as newcomer in dynamically typed world. Although I consider myself as lazy (in terms of creative laziness ;) ) I'd rather put myself in the last group: I had a lot of legacy code to maintain. But I believe that in some day all those code will be fully tested. I try all my new code do with Test Driven Development and strongly encourage my co-workers to do the same. That gives us code 100% covered by tests. Of course it doesn't mean it will be 100% correct, but problems with incorrect object's interface can be easily detected with tests.
|||The Zen of Python https://www.python.org/dev/peps/pep-0020/|
|||Abstract Base Classes https://www.python.org/dev/peps/pep-3119/|
|||(1, 2) Zope Interfaces http://docs.zope.org/zope.interface/|