Post by Nelis Willers. This Notebook will/was also presented at the Gauteng Python Users Group. Follow the link at the end of this post to ensure you have the latest workbook. ed
Notes on Python Decorators¶
This document captures the essential elements of decorators, based on the work of others. The text here is brief, please consult the references for more details.
Don't have time?¶
If you don't have time to read this document, just go here and follow the instructions.
Use case¶
The pyradi module provides an environment for computational optical radiometry. The pyradi.ryplanck file implements the Planck equations as demonstrated here.
The Planck equations have a form similar to this: $$ M_{e\lambda}(T) =\frac{2\pi h c^2}{\lambda^5 \left(e^{hc/(\lambda kT)}-1\right)} =\frac{c_{1e\lambda}}{\lambda^5 \left(e^{c_{2\lambda}/(\lambda T)}-1\right)} $$ where temperature $T$ and wavelength $\lambda$ are numpy arrays of shape [T,] or [T,N] and [L,] respectively. The values of $T$ and $\lambda$ are independent of each other. The calculation of the equation must be done for all combinations of $T$ and $\lambda$, but such that the structure of the return variable is [L,1] or [L,N].
There are two issues in implementing these functions for computing:
There are twelve variations of this equation, with slightly different forms, for
- 3x frequency, wavenumber, and wavelength units,
- 2x radiant units and photon units, and
- 2x basic form and temperature derivative form.
The Planck equation is easy to calculate for a given temperature and wavelength, does not lend itself to N-dimensional vectorised calculation because of its relatively complex form. Some data preparation must be done prior to doing the Planck equation calculation.
A practical solution to this problem is to (1) pass the temperature and spectral data as N-dimensional arrays, but (2) to 'flatten' these to a simpler form, (3) do the calculations on the simple form and then (4) reshape the result back into the N-dimensional form. Of course, the flattening and reshaping must be the inverse of each other.
The flattening step essentially creates two new arrays (one for temperature and one for wavelength) that contain all possible variations of these two variables. This is done with numpy.meshgrid. The two [N,T] arrays are then reshaped to (NT,) arrays with numpy.ravel.
Each of the twelve equation variations requires exactly the same flattening and reshaping. It is indeed an arduous and error-prone task to write and maintain twelve copies of the same code, embedded in the twelve functions.
The implementation is hugely simplified by using the Python decorator to do the pre-processing (variable checking and flattening) and post-processing (reshaping) of the arrays. There is only one copy/version of this pre-post-processor decorator, used as a wrapper around all twelve function variations.
In the following code docstrings and much functional code are removed to focus on the salient content.
First, consider the Planck function wrapped in the decorator (indicated here as @fixDimensions).
Note that the user function focuses only on the Planck equation and not on array dimensions. The fixDimensions intercepts all the input and output from this function and process according to the decorator requirements.
@fixDimensions
def planckel(spectral, temperature):
return (pconst.c1el / ( numpy.exp(exP2)-1)) / (spectral ** 5)
@fixDimensions
def planckqn(spectral, temperature):
exP = pconst.c2n * spectral / temperature
exP2 = np.where(exP<explimit, exP, 1);
return pconst.c1qn * spectral**2 / (np.exp(exP2)-1);
Repeat this twelve times for the twelve functions.
The decorator function focuses only on manipulation of input and output data formats, and has no idea what is happening inside the Planck functions, which it wraps.
def fixDimensions(planckFun):
@wraps(planckFun)
def inner(spectral, temperature):
The decorator function reshapes and creates a meshgrid to provide all the combinations of input sample values and then flattens the arrays by using the ravel function.
# create flattened version of the input dataset
specgrid, tempgrid = numpy.meshgrid(spectral,temperature)
spec = numpy.ravel(specgrid)
temp = numpy.ravel(tempgrid)
At this point the Planck function is called with the flattened arrays.
planckA = planckFun(spec,temp)
After the function returns, the return values are handed back to the decorator function which then proceeds to reshape the array back into a format the user can work with.
# now unflatten to proper structure again, spectral along axis=0
return planckA.reshape(temperature.shape[0],-1).T
return inner
Decorators, why?¶
- add functionality to existing functions and classes.
- do timing on functions.
- perform pre- and post-formatting on data.
- diagnostics and testing.
- perform monkey patching.
Decorator concepts¶
The concepts underlying the Python decorator are described in detail in the references at the end of this notebook. This section serves to summarise, leaning rather heavily on the references in doing so.
This discussion is centred around Python 2.7, apparently the Python 3 syntax is somewhat different.
In twelve easy steps Franklin provides the ground work for building up to the punchline. For a more verbose discussion see Franklin's page. These are:
1. It all starts with Python functions that has a name, an optional list of parameters (positional and named/default) and can return a value.
2. Variables defined inside a function has scope only inside the function.
3. Variable names are resolved (for access) with the LEGB (local, enclosing, global, built-in) rule. The first name found is used. Inner scopes have read access to outer scopes, but not write/change access.
4. Variables has lifetime only in the scope that exists at any moment.
5. Variables passed as function parameters become local variables in the function.
6. Python allows nested functions, with the normal scoping rules. In this code inner is a local variable in the function outer, it just happens to be a function:
def outer():
x = 1
def inner():
print(x)
inner()
outer()
7. Functions are objects that/(whose names) can be passed as variables to functions and can be returned from functions.
def add(x, y):
return x + y
def apply(func, x, y):
return func(x, y)
apply(add, 2, 1)
In this example, inner is redefined every time when outer() is executed. The function outer returns the function name inner which can be called with ().
def outer():
def inner():
print('Inside inner')
return inner
print(outer)
foo = outer()
print(foo)
foo()
8. Python supports function closure, which means that non-global functions remember what their enclosing namespaces looked like at definition time, even if the enclosing environment (function in this case) goes out of scope. In this example the variable x goes out of scope when outer() returns inner to the variable foo, but inner still remembers the value when foo() is executed.
def outer():
x = 1
def inner():
print(x)
return inner
foo = outer()
print(foo.func_closure)
foo()
Here is an example of how closure can build custom functions:
def outer(x):
def inner():
print x # 1
return inner
print1 = outer(1)
print2 = outer(2)
print1()
print2()
9. A decorator is a function that takes a function name as an argument and returns a replacement function, which can be called elsewhere. Here is the principle at work for a function with no parameters:
def outer(some_func):
def inner():
print('before some_func - do whatever you want before')
ret = some_func()
print('after some_func - do whatever you want after')
return ret + 1
return inner
def foo():
return 1
decorated = outer(foo)
decorated()
If we want to keep the foo function name, just reassign it to the decorated function, then calling foo will call the decorated function.
Caveat IPython cell memory! Run the following cell a few times....
foo = outer(foo) # now becomes outer(foo), same as decorated above
print(foo)
foo()
10. The @ symbol applies a decorator to a function, as in
@decoratorname
def function():
pass
11. *args and **kwargs provides the means to pass along any number of function parameters. This allows one to wrap/decorate any function, irrespective of its signature.
By using *args (list of function arguments) and **kwargs (dictionary of function keyword arguments) the decorator can be applied to functions and class methods alike, with any number of arguments. The *args list and **kwargs dictionary are just unrolled into the new func call.
12. Simple example: printing function parameters before calling a function.
def logger(func):
def inner(*args, **kwargs):
print('Arguments were: {}, {}'.format(args, kwargs))
rtnVal = func(*args, **kwargs)
print('Return value is: {}'.format(rtnVal))
return rtnVal
return inner
@logger
def foo1(x,y=1):
return x * y
@logger
def foo2():
return 2
foo1(5,4)
foo2()
13. Here is an example by Farhad of the decorator applied to a class method:
def p_decorate(func):
def inner(*args, **kwargs):
return "<p>{0}</p>".format(func(*args, **kwargs))
return inner
class Person(object):
def __init__(self):
self.name = "John"
self.family = "Doe"
@p_decorate
def get_fullname(self):
return self.name+" "+self.family
my_person = Person()
print my_person.get_fullname()
Passing arguments to decorators¶
Decorators can be passed arguments by using an additional level of function nesting around the decorators seen thus far.
This Farhad example passes arguments to the decorator function:
def tags(tag_name):
def tags_decorator(func):
def inner(name):
return '<{0}>{1}</{0}>'.format(tag_name, func(name))
return inner
return tags_decorator
@tags('p')
@tags('strong')
def get_text(name):
return 'Hello '+name
print get_text('John')
You can also pass conditionals to decorators to change behaviour.
def skipIf(conditional, message):
def dec(func):
def inner(*args, **kwargs):
if conditional:
print(message)
else:
return func(*args, **kwargs)
return inner
return dec
logical = [True, False]
for log in logical:
print(log)
@skipIf(log, 'skipped function call')
def myFunc():
print('in myFunc')
myFunc()
Decorators and function attributes¶
The decorating wrapper as shown above overwrites the function's attributes, so the wrapped function's attributes are lost.
As of Python 2.5, the functools module contains functools.wraps, a decorator for updating the attributes of the wrapping function to those of the original function. This is as simple as decorating the inner function with @wraps(func). Farhad's example:
from functools import wraps
def tags(tag_name):
def tags_decorator(func):
@wraps(func)
def inner(name):
return "<{0}>{1}</{0}>".format(tag_name, func(name))
return inner
return tags_decorator
@tags("p")
def get_text(name):
"""returns some text"""
return "Hello "+name
print get_text.__name__ # get_text
print get_text.__doc__ # returns some text
print get_text.__module__ # __main__
Graham Dumpleton created a library called wrapt, going much beyond wraps. Dumpleton writes: "To provide a transparent object proxy for Python, which can be used as the basis for the construction of function wrappers and decorator functions.
It therefore goes way beyond existing mechanisms such as functools.wraps() to ensure that decorators preserve introspectability, signatures, type checking abilities etc. The decorators that can be constructed using this module will work in far more scenarios than typical decorators and provide more predictable and consistent behaviour." See the use of this module below.
Decorator classes¶
The decorator wrapper must be a callable, such as a function (shown above) or a class with a __call__() member function. Hence classes can also be used as decorators. Class decorators are slightly more involved than function decorators, but implements the same functionality. See Eckel's blogs
When using the class as a decorator, the class is initialised and remembers the original function, whereas function decorators reassign the original function name to the newly wrapped function name.
Use functools.update_wrapper() to make available the wrapped functions attributes to the decorator wrapping class.
from functools import update_wrapper
class myDecorator(object):
def __init__(self, f):
print('inside myDecorator.__init__()')
self.f = f
update_wrapper(self, f)
def __call__(self):
print "Entering", self.f.__name__
self.f()
print "Exited", self.f.__name__
@myDecorator
def aFunction():
"""aFunction does aThing"""
print('inside aFunction()')
print('Finished decorating aFunction()\n')
aFunction()
print('\nWrapped function name is {}'.format(aFunction.__name__))
print('Wrapped docstring is "{}"'.format(aFunction.__doc__))
Class decorators can also have parameters. This time the decorator class implementation is a lot more complex than the decorator function implementation.
class decoratorWithArguments(object):
def __init__(self, arg1, arg2, arg3):
"""
If there are decorator arguments, the function
to be decorated is not passed to the constructor!
"""
print('Inside __init__()')
self.arg1 = arg1
self.arg2 = arg2
self.arg3 = arg3
def __call__(self, f):
"""
If there are decorator arguments, __call__() is only called
once, as part of the decoration process! You can only give
it a single argument, which is the function object.
"""
print('Inside __call__()')
def wrapped_f(*args):
print('Inside wrapped_f()')
print('Decorator arguments: {} {} {} '.format(self.arg1, self.arg2, self.arg3))
f(*args)
print('After f(*args)')
return wrapped_f
@decoratorWithArguments("hello", "world", 42)
def sayHello(a1, a2, a3, a4):
print('sayHello arguments: {} {} {} {}'.format(a1, a2, a3, a4))
print('After decoration')
print('Preparing to call sayHello()')
sayHello("say", "hello", "argument", "list")
print('after first sayHello() call')
sayHello("a", "different", "set of", "arguments")
print('after second sayHello() call')
This seems overly complex and confusing, but I guess it might be required for some sophisticated applications. In Eckel's own words:
"Now the process of decoration calls the constructor and then immediately invokes __call__(), which can only take a single argument (the function object) and must return the decorated function object that replaces the original. Notice that __call__() is now only invoked once, during decoration, and after that the decorated function that you return from __call__() is used for the actual calls."
"Although this behavior makes sense -- the constructor is now used to capture the decorator arguments, but the object __call__() can no longer be used as the decorated function call, so you must instead use __call__() to perform the decoration -- it is nonetheless surprising the first time you see it because it's acting so much differently than the no-argument case, and you must code the decorator very differently from the no-argument case."
Decorating classes¶
All the examples thus far decorated/wrapped functions. Can classes be decorated? Based on this blog by da Palma, it seems that there are two options: (1) decorating a class by decorating each of its member functions and (2) where you write a class decorator and select which methods to decorate passing their names as the decorator arguments. I repeat his code below exactly as blogged.
def method_decorator(fn):
"Example of a method decorator"
def decorator(*args, **kwargs):
print "\tInside the decorator"
return fn(*args, **kwargs)
return decorator
class MyFirstClass(object):
"""
This class has all its methods decorated
"""
@method_decorator
def first_method(self, *args, **kwargs):
print "\t\tthis is a the MyFirstClass.first_method"
@method_decorator
def second_method(self, *args, **kwargs):
print "\t\tthis is the MyFirstClass.second_method"
if __name__ == "__main__":
print "::: With decorated methods :::"
x = MyFirstClass()
x.first_method()
x.second_method()
def method_decorator(fn):
"Example of a method decorator"
def decorator(*args, **kwargs):
print "\tInside the decorator"
return fn(*args, **kwargs)
return decorator
def class_decorator(*method_names):
def class_rebuilder(cls):
"The class decorator example"
class NewClass(cls):
"This is the overwritten class"
def __getattribute__(self, attr_name):
obj = super(NewClass, self).__getattribute__(attr_name)
if hasattr(obj, '__call__') and attr_name in method_names:
return method_decorator(obj)
return obj
return NewClass
return class_rebuilder
@class_decorator('first_method', 'second_method')
class MySecondClass(object):
"""
This class is decorated
"""
def first_method(self, *args, **kwargs):
print "\t\tthis is a the MySecondClass.first_method"
def second_method(self, *args, **kwargs):
print "\t\tthis is the MySecondClass.second_method"
if __name__ == "__main__":
print "::: With a decorated class :::"
z = MySecondClass()
z.first_method()
z.second_method()
Dumpleton's warpt module¶
I discoved Dumpleton's work in his Pycon 2014 video, after all of the above was written. The writeup above is still valid but it is recommended that you use the wrapt module for production work on decorators, wrappers and monkey patching. This module provides the required robustness to decorators to behave as expected (e.g., as functions) under all conditions.
Dumpleton shows in his talk how the techniques described above have weaknesses when it comes to accessing the wrapped function's attributes and descriptors. During the talk, Dumpleton smiled and made the remark "totally losing you at this point, I guess" and "I will let you digest that one...". This is indeed the case when he goes into the very deep details of fixing the weaknesses of the simple decorators. Fortunately you don't have to understand the inner details to use the wrapt module.
Dumpleton's estimate is that 90% of all decorator requirements can be met by simple function closure wrappers (as described above) - but you lose significant introspection functionality. He recommends that you use wrapt for developing libraries that will be used by other people.
The wrapt module results in more compact code and full introspection functionality; no inner function coding is required.
import wrapt
@wrapt.decorator
def pass_through(wrapped, instance, args, kwargs):
return wrapped(*args, **kwargs)
@pass_through
def function():
"""Function docstring"""
print('Using wrapt with a decorator function and no decorator arguments')
function()
print('Function name'.format(function.__name__))
print('Function doctring'.format(function.__doc__))
import wrapt
class with_arguments(object):
def __init__(self, myarg1, myarg2):
self.myarg1 = myarg1
self.myarg2 = myarg2
@wrapt.decorator
def __call__(self, wrapped, instance, args, kwargs):
return wrapped(*args, **kwargs)
@with_arguments(1, 2)
def function():
print('Using wrapt with a decorator class and decorator arguments')
function()
wrapt decorators can be enabled and disabled by setting a flag. If enabled, the wrapped function is returned, if disabled, the original function is returned.
for ENABLED in [True, False]:
print('\nDecorator enabled is {}'.format(ENABLED))
@wrapt.decorator(enabled=ENABLED)
def pass_through(wrapped, instance, args, kwargs):
print('In decorator, before calling function')
wraptfun = wrapped(*args, **kwargs)
print('In decorator, after calling function')
return wraptfun
@pass_through
def function():
print('Executing function')
print(type(function))
function()
Decorator applications¶
Context managers¶
This example is a very simple example of applying the @contextmanager decorator. The program flow is really simple: you (1) create some context, (2) yield when the context is used and (3) finalise your work when closing the context. the example below is a simple implementation of the with open() as f idiom using a function.
from contextlib import contextmanager
@contextmanager
def myfile(filename, filemode):
f = open(filename, filemode)
try:
yield f
finally:
f.close()
with myfile('test.txt','w') as mf:
mf.write('one line\n')
import os.path
if os.path.sep == '/':
!cat test.txt
else:
!type test.txt
The context manager can also use a class to set up and finalise the context.
class MyFile:
def __init__(self, filename, mode):
self.filename = filename
self.mode = mode
def __enter__(self):
self.thefile = open(self.filename, self.mode)
return self.thefile
def __exit__(self, *unused):
self.thefile.close()
with MyFile('test2.txt','w') as writer:
writer.write("Hello World from our new Context Manager!")
import os.path
if os.path.sep == '/':
!cat test2.txt
else:
!type test2.txt
Timing with decorators¶
import math
import time
def time_dec(func):
def inner(*args, **kwargs):
t = time.clock()
res = func(*args, **kwargs)
print('fn={} time={}'.format(func.func_name, time.clock()-t))
return res
return inner
@time_dec
def myFunction(a):
return math.atan(a)
print(myFunction(0.55))
Chaining decorators
Chaining decorators is simply wrapping decorators within decorators, within decorators....
def makebold(fn):
def inner(*args, **kwargs):
return "<b>" + fn(*args, **kwargs) + "</b>"
return inner
def makeitalic(fn):
def inner(*args, **kwargs):
return "<i>" + fn(*args, **kwargs) + "</i>"
return inner
@makebold
@makeitalic
def hello():
return 'hello world'
print hello()
Counting the number of times a function is called¶
def count(func):
def inner(*args, **kwargs):
inner.counter += 1
return func(*args, **kwargs)
inner.counter = 0
return inner
@count
def myFunction():
pass
myFunction()
myFunction()
print(myFunction.counter)
Redirecting stdout to logger¶
import logging
import sys
class LogPrinter:
'''LogPrinter class which serves to emulates a file object and logs
whatever it gets sent to a Logger object at the INFO level.'''
def __init__(self):
'''Grabs the specific logger to use for logprinting.'''
self.ilogger = logging.getLogger('logprinter')
il = self.ilogger
logging.basicConfig()
il.setLevel(logging.INFO)
hdlr = logging.FileHandler('logprinter.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
self.ilogger.addHandler(hdlr)
def write(self, text):
'''Logs written output to a specific logger'''
self.ilogger.info(text)
def logprintinfo(func):
'''Wraps a method so that any calls made to print get logged instead'''
def pwrapper(*arg, **kwargs):
stdobak = sys.stdout
lpinstance = LogPrinter()
sys.stdout = lpinstance
try:
return func(*arg, **kwargs)
finally:
sys.stdout = stdobak
return pwrapper
@logprintinfo
def somePrintingFunc():
print('Hello output')
somePrintingFunc()
import os.path
if os.path.sep == '/':
!cat logprinter.log
else:
!type logprinter.log
A decorator-based build system¶
Bruce Eckel built a build system using python decorators. His motivation is that build systems really need a language and that make and ant do not supply sufficient capability in this area. In my own view CMake is the better way to go here - is has a horrible language, but it works.
Repository of decorators¶
https://wiki.python.org/moin/PythonDecoratorLibrary has a large number of decorators for many different applications. The following is a list of some of these. The code won't work here, because some of the decorators are quite long and must be included in your code.
Use the accepts decorator for type checking:
@accepts(uint, utf8string)
def myMethod(ID, name):
print(ID, name)
myMethod(4, 'string')
Implement singletons:
@singleton
class Highlander:
x = 100
Highlander() is Highlander() is Highlander #=> True
id(Highlander()) == id(Highlander) #=> True
Highlander().x == Highlander.x == 100 #=> True
Highlander.x = 50
Highlander().x == Highlander.x == 50 #=> True
Memoizing a class. Caches a function's return value each time it is called. If called later with the same arguments, the cached value is returned (not reevaluated)
@memoized
def fibonacci(n):
"Return the nth fibonacci number."
if n in (0, 1):
return n
return fibonacci(n-1) + fibonacci(n-2)
print fibonacci(12)
Smart deprecation warnings (with valid filenames, line numbers, etc.)
@deprecated
def my_func():
pass
my_func()
Easy Dump of Function Arguments
@dump_args
def f1(a,b,c):
print a + b + c
f1(1, 2, 3)
Access control prevents users from getting access to places where they are not authorised to go.
@LoginCheck
def display_members_page():
print 'This is the members page'
References¶
Pycon 2014 videos:
Colton Myers: Decorators: A Powerful Weapon in your Python Arsenal
Graham Dumpleton: Advanced methods for creating decorators
Harrison: Hands-on Intermediate Python
The definitive work on decorators:
Dumpleton: wrapt package
Dumpleton: wrapt documentation
Decorator repository:
wiki.python.org
Blogs:
Knupp: Improve Your Python: Decorators Explained
Franklin: Understanding Python Decorators in 12 Easy Steps!
Eckel: Decorators I: Introduction to Python Decorators
Eckel: Python Decorators II: Decorator Arguments
Eckel: Python Decorators III: A Decorator-Based Build System
Farhat: A guide to Python's function decorators
Yasoob: Python decorators finally demystified
Simionato: PyPI decorator module
da Palma: Writing a class decorator in Python
Harrison: Guide to learning Python decorators
Hellman: contextlib – Context manager utilities
Context managers in Python
Python wiki: PythonDecorators
Chaining decorators
Some common uses
Python and module versions, and dates¶
%load_ext version_information
%version_information
Source and feedback¶
The source of this notebook is available at https://github.com/NelisW/PythonNotesToSelf
Feedback welcome: neliswillers at gmail.