Structuring a simple Python project

Python is mostly an elegant and simple language. As such, it is the tool of choice for engineers, academics, and other “accidental developers” who need to write code outside of a dedicated software development company. Many of these developers quite like coding, would like to share and cooperate on code and make reusable tools. Python packaging, testing, distributing is, in contrast to the core language, an absolute mess. The foundations are a swamp, on top of which is built an increasing array of complex tools: tox, pavement, pytest, nosetest, pip, conda, virtualenv. To the uninitiated, this appears as a complex mass which you struggle even to work out what anything is for. I still don’t know whether I should use setuptools or distutils see here.

There are a variety of templates, “cookie-cutter” recipes for example cookiecutter-pypackage, python-project-template. But I have found them a bit too complex for my needs.

The problem is a lack of middle ground: a simple project layout, a simple way to write and run tests, and a simple way to package and distribute their code. We don’t all need to use continuous integration. Why should we have to install yet another task runner, template engine or similar which “simplifies” the process, but requires a different configuration file. I honestly don’t know whether I need a requirements.txt, an environment.yml, a setup.cfg, a travis.yml a tox.ini or whatever. Developers take note, every time you write a tool which needs another config file at top level, you increase the clutter and complexity of developing a project. Sure, all of these things are useful and probably have a place

This post is aimed at providing a structure and a process suitable for small-time developers who still wish to follow good practice, like me. It is my own attempt to boil down all the myriad options and information into a something usable.

Terminology

In my mind project > package > module. And

  • project corresponds to a repository on GitHub
  • project usually contains one python package, in a subdirectory, often with the same name as the project
  • project contains other stuff, such as the docs, README, tools for helping other developers
  • the package contains multiple python modules

In the template, I have named them such, so you have the directory structure:

/project
/project/setup.py
(and license, README, etc)
/project/doc
/project/tests
/project/package
/project/package/module.py

Tools

The tools I have settled on are:

  • git + GitHub for version control and hosting
  • anaconda for python control and virtual environments
  • pytest for testing
  • sphinx for documentation using napoleon extension

Instructions

  1. Clone the template repo:

    git clone https://github.com/samwisehawkins/project

  2. Change the name of the project and python environment

    mv project myproject 
    vi myproject/environment.yml  --> change environment name to match
    
  3. Change the package name, python modules etc. Generally adapt project to your needs.

  4. Edit setup.py to reflect name changes

  5. Remove git history and re-initialise

    cd myproject 
    rm -rf .git 
    git init
    
  6. Create a virtual environment with the same name as the project

    cd myproject 
    conda env create -f environment.yml 
    source activate myproject
    
  7. Install package into new virtual environment, using develop so it is symbolically linked to source code

    python setup.py develop
    
  8. Run tests

    pytest
    
  9. Create documentation structure (use project/doc as target)

    sphinx-quickstart
    
  10. Edit doc/index.rst to auto-include modules and members:

    vi doc/index.rst
    .. automodule:: package.module
       :members:
    
  11. Build the docs

    cd doc
    make html
    
  12. Package and publish (coming soon).

2 thoughts on “Structuring a simple Python project

Leave a Reply

Your email address will not be published. Required fields are marked *