How to create a Python package - Part 1

In the past I always was not really sure how to properly create a python package. I somehow got everything to work but it often was a mess and I wasn't sure if I did something horribly wrong. So this blog post will outline the process of creating a python package from start to finish. It serves as a tutorial for you and a great reminder for me. I always had to browse through lots of blog posts to find everything I need so now I want to collect the basic things in one post. If you follow these instruction your package will

  • have a GitHub repository
  • be listed on PyPi
  • have automated builds and testing with Travis CI
  • be easily testable locally
  • properly handle logging

Prerequisites

This is a list of things that you should own or have knowledge of before we begin.

  • git 
  • a GitHub account (if you do not know what GitHub is you can view this great introduction)
  • a PyPi account (this is the place where the python package are stored and can be installed from)
  • experience with python virtual environments (this is optional but highly recommended)
  • the following python packages (and basic knowledge about pip)

Step 1: Create a new Github repository

The first thing that you should do when starting a new project is creating a new GitHub repository. This will serve as a central point for storing your code. You can easily work with multiple machines and you do not risk losing your progress if your machine fails. You have versioning and issue tracking and so on. So, head over to GitHub and create a new repository.

 Create a new GitHub repository

Create a new GitHub repository

 Choose your package name and license (optional)

Choose your package name and license (optional)

In the next step you will need the URL to your new repository. It should look like this: http://github.com/yourAccount/yourPackageName 

Step 2: Syncing with travis

Travis CI is a tool that will automatically build and test your package. And the best thing is: it is integrated with GitHub and free. You just have to go to travis-ci.org and log in with your GitHub account. Normally a sync with GitHub will start automatically and it often takes some time. If that does not happen you can manually trigger the sync in your account settings. There you can also activate Travis CI for your package. The Travis CI URL for your package is of the following form: http://travis-ci.org/yourAccount/yourPackageName 

 You have to switch on Travis CI for your package

You have to switch on Travis CI for your package

Step 3: the folder structure

One thing I really struggled with for a long time is a proper folder structure for the python package. And I know that there is not the one perfect solution. Nonetheless I will present the folder structure that I used for my recent project soccer.

The cool thing is, that you do not have to create the folders yourself. We will use the tool cookiecutter to create it. I assume that you have some kind of workspace or git folder on your machine that serves as an entry point for your projects. Now follow these steps:

  • Clone the following repository: https://github.com/kragniz/cookiecutter-pypackage-minimal
git clone https://github.com/kragniz/cookiecutter-pypackage-minimal
  • Create a new cookiecutter project using the pypackage-minimal cookiecutter
cookiecutter cookiecutter-pypackage-minimal/
  • Answer the questions provided by cookiecutter
$ cookiecutter.exe cookiecutter-pypackage-minimal/
author_name [Louis Taylor]: Hendrik Hilleckes
author_email [louis@kragniz.eu]: hhllcks@gmail.com
package_name [cookiecutter_pypackage_minimal]: package
package_version [0.1.0]: 0.0.1
package_description [An opinionated, minimal cookiecutter template for Python packages]: An example package
package_url [https://github.com/borntyping/cookiecutter-pypackage-minimal]: https://github.com/hhllcks/package
readme_pypi_badge [True]:
readme_travis_badge [True]:
readme_travis_url [https://travis-ci.org/borntyping/cookiecutter-pypackage-minimal]: https://travis-ci.org/hhllcks/package

Now you should have your package structure ready. It should look like this:

  • package
    • __init__.py
  • tests
    • __init__.py
    • test_sample.py
  • .gitignore
  • README.rst
  • setup.py
  • tox.ini

STEP 4: Sync with GITHUB

Now we have to sync our package with GitHub. To do that you first have to initialize git in your package. Switch to the folder and call git init.

cd package
git init
Initialized empty Git repository in C:/Users/hilleckesh/git/package/.git/

The following commands will connect your local git repository with the one on GitHub.

git add .                                                # add your files 
git commit -m "initial commit"                           # commit 
git remote add origin https://github.com/hhllcks/package # adding the remote repository
git remote -v                                            # check remote connection
git pull origin master --allow-unrelated-histories       # download the current remote repository content
git push origin master                                   # pushing the new folder structure

If you refresh the homepage of your package you should already see a nice new readme.

gh_sync.png

Step 5: Get Travis CI working

To get Travis CI to work you need to add a .travis.yml file to your repository. It will tell Travis CI what to do. More about the file here. I will now show you my .travis.yml file and tell you what each part is for. Here is the file:

language: python
python: 
  - "3.6"

install: "pip install -r requirements.txt"

script: pytest

notifications:
  email:
    on_success: never # default: change
    on_failure: always # default: always

First we specify python as the programming language (Travis CI can be used with a lot of different programming languages) and the version. I personally use Python 3.6 at the moment. The install command tells Travis CI that it has to install some requirements. For example my package footballdataorg uses the requests package. Read more about the requirements.txt here. For the moment you can add an empty file called requirements.txt to your package. The script command tells Travis CI to run pytest after installing the package. More about pytest later. In short: you can write tests that will check if your code base still works. The last part is optional and you do not necessarily need it. Travis CI sends you a lot of mails when it tries to build your package. It changed the settings such that I only get a mail if the build failed.

After adding, commiting and pushing the two files (.travis.yml and requirements.txt) to your repository your first Travis CI build should be successful. You can identify this by looking at the little batch in your readme file.

travis_badge.png

Summary

This finishes the first part of my tutorial on creating a python package. You learned how to create a the python package and connect it to GitHub and Travis CI. And Travis CI has already successfully installed your package. In the next part we will look at the setup.py file and learn how to upload the package to PyPi. We will talk about testing and logging, too. 

Feel free to comment or tweet at me if you have a suggestion on how to improve my process of creating new python package.