6 Ways to Improve Performance in Python

You can download the project for this post by clicking HERE.

In this article we are going to describe a series of recommendations and best practices for programming in Python (version 2.7). Some of them apply to any programming language, and others consist of programming in a more “Pythonic” style whenever possible.

These recommendations apply to:

1. List comprehensions
2. Generators
3. Membership checks with data structures that hash their data.
4. Strings
5. Intermediate outputs
6. Caches

List comprehensions

Script: 01 – comprehensions.py

List comprehensions are a way of creating lists in Python from iterables (other lists, dictionaries, sets, etc.). As we are going to see, they don’t solve a new problem (the bad example will look familiar), but they give us a new syntax that makes Python run them faster.

def good_list():
    my_list = [value for value in range(elements)]

def bad_list():
    my_list = []
    for value in range(elements):
        my_list.append(value)

As we can see, in both cases we are filling a list with the numbers from 0 to elements-1. The second one is the classic way of solving the problem: we define a list and fill it with the elements one by one. The first one is how you are supposed to do it in Python. If we run this n times, measure the execution times and take the average, we get the following results (for n = 3):

As we can see, it is faster if we do it the way Python gives us. Some of you may be thinking that this is just a way of turning for loops into this Python form, and partly that’s true, with the caveat that every list comprehension can be converted into a for loop, but not every for loop can be turned into this kind of list. List comprehensions let us operate on the values, for example squaring the elements:

def square_good_list():
    my_list = [value * value for value in range(elements)]

Or also add only the elements that meet a certain condition, such as the even ones:

def good_list_validation():
    my_list = [value for value in range(elements)
        if value % 2 == 0]

Generators

Script: 02 – generator.py

Generators, as the name suggests, are used to generate sequences of values. The sequences are generated little by little. Here performance has two dimensions: computation time and memory usage.

Generators become indispensable when working with very large series: if we wanted to generate all the prime numbers and return them, it would be impossible, since we would run out of memory. With a generator, however, we can produce them each time we need one and we won’t run out of memory.

range is a generator in Python 3, but not in Python 2, where we have xrange. To see the difference more clearly:

• If in Python 2 we call range(3), the function generates all the elements [0,1,2] and returns them, whereas xrange(3) returns one element whenever we ask for it (each time we iterate)
• Xrange always uses the same amount of memory.

As we said, the good way to define a generator in Python 2 is with xrange

def good_generator():
    return (x*2 for x in xrange(elements))

def bad_generator():
    return (x*2 for x in range(elements))

We can also define our own generator like this:

def generator(element_number, start_value=0):
    while start_value < element_number:
        yield start_value
        start_value += 1

With the @profile decorator we can see how much memory is being used. To do that we need memory_profiler installed, which we can do with:

pip install -U memory_profiler

Line #    Mem usage    Increment   Line Contents

================================================

   17     12.7 MiB       0.0 MiB    @profile

   18                              def my_generator():

   19     12.7 MiB       0.0 MiB        return (x*2 for x in generator(elements))

Line #    Mem usage    Increment   Line Contents

================================================

   22     12.7 MiB       0.0 MiB    @profile

   23                              def my_generator_testing():

   24     12.7 MiB       0.0 MiB        for loop_count in my_generator():

   25     12.7 MiB       0.0 MiB            foo = loop_count + 3

Line #    Mem usage    Increment   Line Contents

================================================

   27     12.7 MiB      0.0 MiB   @profile

   28                             def good_generator():

   29     12.7 MiB      0.0 MiB       return (x*2 for x in xrange(elements))

Line #    Mem usage    Increment   Line Contents

================================================

   32     12.7 MiB      0.0 MiB   @profile

   33                             def good_generator_testing():

   34     12.7 MiB      0.0 MiB       for loop_count in good_generator():

   35     12.7 MiB      0.0 MiB           foo = loop_count + 3

Line #    Mem usage    Increment   Line Contents

================================================

   37     12.7 MiB      0.0 MiB   @profile

   38                             def bad_generator():

   39     43.7 MiB     31.0 MiB       return (x*2 for x in range(elements))

Line #    Mem usage    Increment   Line Contents

================================================

   42     12.7 MiB      0.0 MiB   @profile

   43                             def bad_generator_testing():

   44     43.7 MiB     31.0 MiB       for loop_count in bad_generator():

   45     43.7 MiB      0.0 MiB           foo = loop_count + 3

The time is slightly lower, but it isn’t really relevant. The most relevant thing in this example is the memory increase from 13MB to 31MB with just 1,000,000 integer elements. In fact, since these sizes fit in memory, we can see that the “bad” way is faster, because once it has everything in memory it can access it more quickly. The times were taken without the @profile decorator, so if you run it with the decorator on, you will get higher times.

More information about Generators in Python and examples in our dedicated article by clicking here.

Membership checks with data structures that hash their data

Script: 03 – membership_checks.py

Whenever we find ourselves using the word in, it will be much more efficient with a set than with a list. The reason is that sets compute the hash of the elements they contain, so to know whether an element is in the set, Python computes the element’s hash and checks whether it has it. This takes constant time. In a list, the interpreter has to go element by element checking whether each one is the element we want to know is in the list.

Let’s not forget that a set stores its elements uniquely and unordered, so it may not fit our problem. Let’s define a list and a set to check it:

my_set = {value for value in range(elements)}
my_list = [value for value in range(elements)]

start = time.time()
999999 in my_set
end = time.time()
good_time = end - start

start = time.time()
999999 in my_list
end = time.time()
bad_time = end - start

As we can see, since it is the last element, the list time is greater than the set time:

start = time.time()
5 in my_set
end = time.time()
good_time = end - start

start = time.time()
5 in my_list
end = time.time()
bad_time = end - start

On the other hand, if we search near the beginning of the list, searching the list is faster

Strings

Script: 04 – strings.py

This section applies to many programming languages, including Java. Strings are immutable objects, which means that if we concatenate with the ‘+’ operator we are creating new objects, so it is not a good idea to concatenate many of them, to avoid creating a new object with each concatenation. Strings should always be handled with functions designed for it, such as join.

def good_string_joiner():
    ''.join(examples)

def bad_string_joiner():
    final_string = ''
    for c in examples:
        final_string += c

The execution time results are:

If we want to insert strings in the middle of another string, the cleanest way is to use a placeholder and the % operator, for example:

print ("Ms. %s! My %s crawled in my %s and then I %s it."
       "Can I have another one?" % ("Hoover", "worm", "mouth", "ate"))

Another option is the format function, which also lets us define the order of substitution.

print ("Ms. {}! My {} crawled in my {} and then I {} it."
       "Can I have another one?".format("Hoover", "worm", "mouth", "ate"))

print ("Ms. {0}! My {2} crawled in my {1} and then I {3} it."
       "Can I have another one?".format("Hoover", "worm", "mouth", "ate"))

As a result we get:

Ms. Hoover! My worm crawled in my mouth and then I ate it.Can I have another one?
Ms. Hoover! My worm crawled in my mouth and then I ate it.Can I have another one?
Ms. Hoover! My mouth crawled in my worm and then I ate it.Can I have another one?

Intermediate outputs

Script: 05 – intermediate.py

This is another problem that is not specific to Python. Every time we do a print, we are making a call to the operating system, and these calls are synchronous, so until the operating system finishes, we can’t continue. It is very common to use them for debugging, which is not a good practice. For debugging we have debuggers!

def good_intermediate():
    for loop_count in range(elements):
        print ('Hi how are you')

def bad_intermediate():
    for loop_count in range(elements):
        print ('Hi')
        print ('How')
        print ('Are')
        print ('You')

The execution time results are:

Caches

Script: 06 – caches.py

Caches are very useful when we have to do complex calculations that may be repeated. Or, for example, if we have to access a database a lot and want to avoid that latency. If the calculation is not complicated or the latency is small, it isn’t worth storing these results, because caches don’t have zero cost. You have to look at what eviction policy they have, you have to add elements, query those elements to see whether we already have them or not, etc. Python 3 has its own cache system built in; for Python 2 there are many alternatives, and here we will use repoze.lru, which we can install with:

pip install repoze.lru

In this case we will use the @lru_cache decorator, where we specify the cache size. This function has a sleep to simulate an expensive operation

@lru_cache(maxsize=16)
def cached_expensive_function(value):
    time.sleep(0.1)
    return value * 2

def non_cached_expensive_function(value):
    time.sleep(0.1)
    return value * 2

If the function is not expensive (we remove the sleep), we see that the cache becomes counterproductive:

@lru_cache(maxsize=16)
def cached_inexpensive_function(value):
    return value * 2

def non_cached_inexpensive_function(value):
    return value * 2

References

To write this tutorial we used the following article as a reference: 7 Tips For Python Performance (Packt)