..

Packaging Pulumi with Poetry

Introduction

Although Terraform seems to be the current front-runner in the infrastructure-as-code department, I've recently run across Pulumi as an excellent alternative. In this post, I'd like to run through how to integrate Pulumi's Python with Poetry (a PEP-517 compatible packaging manager). Once you have everything integrated together, you can benefit from the code quality tools available in the Python ecosystem as you build out your infrastructure.

This post assumes that you are already familiar with the basics of using Pulumi with Python. The goal is to work through how Pulumi code may be packaged.

Install Poetry

Let's start off with getting Poetry installed. Head on over to the installation instructions. If you're on a Mac and using Homebrew, this can be as simple as running "brew install poetry". I'll wait here while you get it installed.

Just to make sure things are working correctly, run "poetry --version" and verify that it actually prints out a version number. At the time of this writing, the output should resemble something like "Poetry version 1.0.10".

Create Your Project

Now that you have Poetry installed, go ahead and create your new project. Simply run "poetry new my-pulumi-project" and "cd my-pulumi-project".

Next, add common tools that you use to the dev environment by running "poetry add --dev flake8 mypy pytest-cov".

Now, add Pulumi itself and the appropriate target module that you plan on using. For example, if you'd like Pulumi to automate your Cloudflare infrastructure, you should run something like "poetry add pulumi pulumi_cloudflare".

Now that all of our dependencies are specified, lets organize the source a bit -- the defaults just dump it into the project folder. Lets move it into a "src" directory to make it a bit cleaner. Run "mkdir src" and "mv my_pulumi_project src" while in the "my-pulumi-project" directory.

To tell Poetry where you moved things to, update the "pyproject.toml" file and add the following under the "[tool.poetry]" section:

packages = [
    { include = "my_pulumi_project", from = "src" }
]

Configure Mypy

In your project directory, create a file called "mypy.ini" with the following contents. I highly recommend that you use Mypy on every Python project you work on -- I've personally caught many, many bugs that I wouldn't have noticed otherwise.

[mypy]

# Import discovery
mypy_path = src
namespace_packages = true

# Disallow dynamic typing
disallow_any_unimported = true
disallow_any_expr = true
disallow_any_decorated = true
disallow_any_explicit = true
disallow_any_generics = true
disallow_subclassing_any = true

# Untyped definitions and calls
disallow_untyped_calls = false
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
disallow_untyped_decorators = false

# None and Optional handling
no_implicit_optional = true
strict_optional = true

# Configuring warning
warn_redundant_casts = true
warn_unused_ignores = true
warn_no_return = true
warn_return_any = true
warn_unreachable = true

# Suppressing errors
show_none_errors = true
ignore_errors = false

# Miscellaneous strictness flags
allow_untyped_globals = false
allow_redefinition = false
implicit_reexport = false
strict_equality = true

# Configuring error messages
show_error_context = true
show_column_numbers = true
show_error_codes = true
pretty = true
color_output = true
error_summary = true
show_absolute_path = true

# Advanced options
show_traceback = true
raise_exceptions = true

# Miscellaneous
warn_unused_configs = true
verbosity = 0

Setup Tox

If you don't already have Tox installed, follow the installation instructions. If you're on a Mac and using Homebrew, you can simply run "brew install tox".

Tox is useful for testing your code under various environments and making sure your code quality tools all pass. Once you have it configured well, you can call it from something like Github Actions to automate your deployment workflow when you commit to git. To get started, add the following to a file called "tox.ini" in your project directory:

[tox]
isolated_build = true
envlist = py38

[testenv]
whitelist_externals = poetry
commands =
    poetry install -v
    poetry run flake8 src/ tests/
    poetry run mypy src/
#    poetry run pytest --cov-report term-missing \
#                      --cov=my_pulumi_project \
#                      --cov-fail-under=100 \
#                      tests/

[flake8]
max-complexity = 10
max-line-length = 160

Afterwards, run the "tox" command from your project directory to see if everything is working correctly. Once you add a few unit tests, uncomment the pytest lines in your tox.ini file.

Configure Pulumi

Now that our basic project structure is coming together, lets setup Pulumi itself. Simply add a file called "Pulumi.yaml" into your project directory with the following contents:

name: my-pulumi-project
runtime: python
description: A demo Pulumi project using Poetry.
main: src/my_pulumi_project

Create a simple way to test locally

We're in the home stretch. To make things easier to test locally, lets add a simple way to run a Pulumi preview. Create a file named "Makefile" in your project directory with the following contents:

ifndef CLOUDFLARE_EMAIL
$(error CLOUDFLARE_EMAIL is not set)
endif

ifndef CLOUDFLARE_API_KEY
$(error CLOUDFLARE_API_KEY is not set)
endif

all:
    poetry run pulumi preview

If you're using something other than Cloudflare, swap out the env variable checks there for whatever you feel is appropriate (or just delete them).

Now, when you want to see what actions Pulumi will take before you commit, you can simply run "make".

Next Steps


Now that you have a basic project structure for Pulumi using Poetry, Mypy, Flake8, and pytest, you're on your way towards infrastructure as code in style. It's time to head on over to the Pulumi documention to get started. Create a file called "src/my_pulumi_proect/__main__.py" as the entry point for your Pulumi code.

Once you're ready, configure your deployment pipeline to run code quality checks with tox, then "poetry run pulumi ..." to actually apply your new infrastructure.