Introducing Functional Programming to Pythonistas

Development cycle

When we update our software we rely on extensive tests to check the code. These tests may take hours to complete, resulting in a long feedback loop. FP may provide a faster feedback loop to improve our productivity.

Reusability

When we integrate software to build services, we are often dealing with a complex stack of components that do not share or re-use common computations. For example, we have many independent TLS setup implementations. Such components rely on complex state management that is difficult to compose. FP provides effective tools to define and re-use the common computations.

System operations

When we operate software, we are often faced with runtime errors such as AttributeError or ReferenceError. For example, when we restart our CI scheduler, we may not be able to re-enqueue some changes when they are in an undefined state. This kind of issue is nasty because:

• The underlying bugs are hard to debug.
• Operating the software in production is stressful.
• Even though we spend a lot of time and energy to prevent those issues, they can still occur.

FP may result in lower bug counts to increase reliability.

However, learning FP has a significant cost of its own. The next section describes the strategies I used to introduce FP.

Why

Function composition is a mechanism that combines two functions. It is similar to object inheritance or object composition, but uses general purpose primitives that can be applied to a large set of problems. In particular, using function composition results in a more consistent interface compared to stateful objects.

How

I shared Scott Wlaschin's talk The Power of Composition and looked for practical use-cases in our code base. For example, instead of writing this computation:

for build in get_builds():
    if build.status == "NODE_FAILURE":
        continue
    print(build.log_url)
    ...

I would suggest using this function composition:

def completed_build(build):
    return build.status != "NODE_FAILURE"

def process(build):
    print(build.log_url)
    ...

list(map(process, filter(completed_build, get_builds())))

Benefits

• Break a computation in smaller pieces.
• Re-use common functions like map and filter.
• More declarative.

Results

Thinking in terms of function composition requires a change in habit, though I think this first concept is easy to introduce. A colleague even started to use functools.reduce when map was not enough.

I recommend Julien Danjou's post An Introduction to Functional Programming with Python as a follow-up.

To build up on function composition, I then introduced pure functions.

Why

For a given input, a pure function always produces the same output. Thus, it is possible to truly study and understand such functions independently from others and they provide a solid building block for complex computations.

How

I introduced Haskell for a small side project where we needed a new service to forward statsd metrics from UDP packets to two backends: telegraph and prometheus. We implemented the service in Python (source) and Haskell (source) in pair programming.

Then I wrote a couple of blog posts about the language: Getting Started with Haskell on Fedora and Haskell for python developpers. I also demonstrated how we could use the language for other use-cases such as documentation management and software factory packages update.

Benefits

• Haskell features state-of-the-art programming paradigms.
• Small footprint, the service has been running continuously without any issues.
• More robust, the Haskell network library reports when UDP packets are not being delivered, something we were not able to achieve with Python.
• Short development cycle where most of the bugs are prevented by the compiler.

Results

There is a lot going on in Haskell and there is a steep learning curve. Learning Haskell is a significant investment and the project I used to introduce the language was too small to allow for the team to gain adequate experience using the language. Though new projects are a good opportunity to explore new languages.

Why

As Graham Hutton states in the What is a Monad? computerphile video, Monad is one of the most important developments in programming languages in the last twenty-eight years.

How

I introduced the concept through a code kata using the Monadic Parser Combinators paper to implement a RPM spec file parser. I went through the paper by re-implementing each functions in pure Python. Then I showed how the same concepts can be used for error handling and to manage side effects.

I also presented a couple of Python libraries such as toolz and effect.

Benefits

Monad is a general purpose building block that can be applied for many, if not all, computations. Combined with pure functions, monads enable a clean composition for effectful computations.

Results

Explaining this concept from the ground up is a difficult exercise and using an existing combinator library might have been a better strategy. Moreover, Python is not the most ergonomic language to write purely functional style code, resulting in non idiomatic implementations.

Even though pure monadic functions did not spark much joy for my team, there is still a lot to learn from FP. Thus, I switched gears to the more practical type system concept.

Why

The fundamental purpose of a type system is to prevent the occurrence of execution errors during the running of a program.

—Luca Cardelli

How

Since python version 3.6, we can add type annotations and use the mypy type checker. So instead of using such object:

def get_build():
    return dict(status="SUCCESS", log_url="http://example")

I would suggest to use a dataclass:

@dataclass(frozen=True)
class Build:
    status: str
    log_url: str

def get_build() -> Build:
    return Build("SUCCESS", "http://example")

I performed a code kata on dataclass and wrote a blog post.

Benefits

• Type checker may prevent runtime errors.
• Re-usable, a library providing dataclass is safer to use.

Results

This was well received and the benefit of mypy is quite practical, but dataclasses are difficult to introduce in an existing code base. Thus, we were not able to benefit much from it.

Why

A strong type system provides stronger safety and definitely excludes certain classes of programming errors.

How

I presented algebraic data type to enable type composition in the form of:

• Sum (union)
• Product (record)

For example, the previous build type can also be defined as

data build = Error | Success(log_url)

For another side project, I presented different languages featuring a strong type system: PureScript, ELM and ReasonML. We picked ReasonML as the team was already familiar with React, and I wrote a blog post about this choice: Software Factory Resources in Reason.

Benefits

• Easier to review and maintain.
• Safer dependencies updates, most breaking changes are prevented by the compiler.

Results

Even though we applied this concept to a web application, using a strong type system was a very valuable experience. We could have used Python with some tricks, for example, by enabling the strict and disallow-any-expr options. This is impractical because Python is dearly missing support for tagged union and pattern matching.

To learn more about strong type systems, I recommend watching this talk by Jane Street: Why Functional Programming Doesn't Matter.

Why

Using a general purpose configuration DSL such as Dhall enables a functional approach to configuration management.

How

Instead of using a data serialization language like JSON or YAML, I introduced the Dhall language to define our configurations.

In the sf-infra project, where we define our infrastructure, I showed how by using Dhall records we could define resources such as servers, volumes, and networks. Then I showed how we could use functions to transform those records into configurations such as the inventory or the monitoring, and how we can re-use those resources in other projects.

For the zuul-operator I wrote a Dhall function to convert the high level definition of a Zuul service into a collection of Kubernetes resources and ConfigMap.

Benefits

• Prevents runtime error, configuration is tested and valid by construction.
• Enables configuration refactoring.
• Fosters re-use through a powerful dependency and import system.

Results

Adopting a new syntax for configuration is a major difficulty. Though I showed how this functional language can be used for all our configuration needs, such as Kubernetes resources, container definitions, CI pipelines and Ansible playbooks.

Why

To capture knowledge in a way that is understandable for both the computer and humans, now and in the future.

How

Comparing with object's names, documentations and tests, I showed how type annotation could capture the knowledge more effectively.

In the context of a couple of mob programming sessions, we added types to undocumented code, resulting in an insightful documentation. Then by increasing the type coverage, we were able to refine the initial knowledge of the code.

We used the TypedDict annotation to incrementally document the distroinfo library output.

Benefits

• Incremental, each modification adds knowledge.
• Mypy verifies the knowledge is correct.

Results

This concept is relatively easy to demonstrate and it yielded the most engagement from my team.