Integrating GitHub with PyPi, Travis-CI, ReadTheDocs, and Code Climate

After change is pushed into GitHub, Travis is used to build and deploy PyPi package. Generated documentation is readable in PyPi, GitHub, and Read The Docs.

Results

Final results looks like this:

• GitHub repository with README
• PyPi package with readable documentation
• Read the Doc documentation (RTFD)
• Travis Status
• Code quality on Code Climate

How To Get There

Register and create new repository on GitHub. Register on PyPi and Read The Docs (RTFD). Sign up with GitHub account for Travis-CI and Code Climate.

Selecting Markup Syntax

PyPi requires reStructuredText, RTFD depends on Sphinx which is also using reStructuredText as markup. GitHub also supports reStructuredText, so it’s best to use reStructuredText.

Creating Python Package

First step is to create a python package. Detailed documentation is available here, but in practice it’s sufficient to use some existing setup.py, requirements.txt, and README.rst and modify them. For list of available classifiers for setup.py check following list. LICENSE file you will get for free when you create new repository on GitHub.

Generating Documentation

Now, when you have your initial README.rst file, it’s time to set up documentation generation. Sphinx tutorial describes whole process in details. You need to run: pip install Sphinx and sphinx-quickstart in your package directory. When you answer some questions, you will get conf.py which contains instructions for sphinx/RTFD and Makefile for generating documentation.

You can generate you documentation with make html and see it with chromium-browser _build/html/README.html. Although, it will look completely fine, it will be reported as failed build in RTFD, if you don’t include:

.. toctree::
   :maxdepth: 2

However, if you try to generate documentation in the same way, as it will be generated by PyPi with

python3 setup.py --long-description | rst2html.py > pypi-doc.html

You will get error

<stdin>:11: (ERROR/3) Unknown directive type "toctree".

To avoid this, it’s necessary to preprocess README.rst before it’s used as long description in PyPi documentation. I have achieved that by introducing two special tags – PYPI-BEGIN and PYPI-END which mark section which should be omitted for PyPi. Then setup.py contains following helper function:

def fix_doc(txt):
	return re.sub(r'\.\. PYPI-BEGIN([\r\n]|.)*?PYPI-END', '', txt, re.DOTALL)

with open('README.rst') as fileR:
    README = fix_doc(fileR.read())

...
  long_description=README,

Now, your documentation is updated on RTFD, it’s readable on GitHub and it will also look good on PyPi.

Version, that you have specified in setup.py should match version specified in conf.py.

Deploying to PyPi

It’s great to have automatically generated documentation, but without package itself it’s useless. So, lets create PyPi package. To do that, you can use Travis-CI. They have documentation how to use travis client as well as how to set up travis config to deploy packages on PyPi.

First, you have to install travis client with gem install travis and then you have to create .travis.yml file. You can also use following template.

language: python
python:
- '2.7'
- '3.3'
- '3.4'
- '3.5'
- '3.6'
install: pip install -r requirements.txt
deploy:
  provider: pypi
  user: user_name
  password:
    secure: somelongstring
  on:
    tags: true
    branch: master

To make it work, you have to do few more steps. Login to GitHub with travis client with travis login. You have to use your GitHub credentials in this step. Then you have to change user_name to your user name, that you use on PyPi in .travis.yml. Last step is to run command travis encrypt --add deploy.password and type your password for PyPi. This will modify .travis.yml file. Now it will contain some real value for key secure.

If you password contains some special characters, you should check documentation how to escape them properly.

Based on our configuration, deployment will be executed only when new tag is created, so we have to create one with following commands.

git tag 0.1 -m "Initial commit"
git push --tags origin master

It would be also nice if tag would match version used in setup.py.

Code Climate

Code Climate will analyze your code after each push. Integration with Travis is easy. Generate token and specify it in .travis.yml.

after_sucess:
- CODECLIMATE_REPO_TOKEN=yourtoken codeclimate-test-reporter

Congratulations!!! Now, you have your package available on PyPi.

Deploying New Version

To deploy new version you have to change its number in setup.py and conf.py. After that, you can create new tag and push changes into GitHub repository with

git tag 0.2 -m "New version"
git push --tags origin master

Troubleshooting

Wrong Python Used For Sphinx

I have received this error, when I have on my Ubuntu package python-sphinx which is using Python 2. Solution is to change SPHINXBUILD = python -msphinx to SPHINXBUILD = python3 -msphinx in Makefile.

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/sphinx/__main__.py", line 14, in
    sys.exit(main(sys.argv))
  File "/usr/lib/python2.7/dist-packages/sphinx/__init__.py", line 51, in main
    sys.exit(build_main(argv))
  File "/usr/lib/python2.7/dist-packages/sphinx/__init__.py", line 61, in build_main
    from sphinx import cmdline
  File "/usr/lib/python2.7/dist-packages/sphinx/cmdline.py", line 14, in
    import optparse
  File "/usr/lib/python2.7/optparse.py", line 419, in
    _builtin_cvt = { "int" : (_parse_int, _("integer")),
  File "/usr/lib/python2.7/gettext.py", line 584, in gettext
    return dgettext(_current_domain, message)
  File "/usr/lib/python2.7/gettext.py", line 548, in dgettext
    codeset=_localecodesets.get(domain))
  File "/usr/lib/python2.7/gettext.py", line 483, in translation
    mofiles = find(domain, localedir, languages, all=1)
  File "/usr/lib/python2.7/gettext.py", line 440, in find
    for nelang in _expand_lang(lang):
  File "/usr/lib/python2.7/gettext.py", line 133, in _expand_lang
    from locale import normalize
ImportError: cannot import name normalize
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python2.7/dist-packages/apport/__init__.py", line 5, in
    from apport.report import Report
  File "/usr/lib/python2.7/dist-packages/apport/report.py", line 12, in
    import subprocess, tempfile, os.path, re, pwd, grp, os, time
  File "/usr/lib/python2.7/tempfile.py", line 32, in
    import io as _io
  File "/usr/lib/python2.7/dist-packages/sphinx/io.py", line 11, in
    from docutils.io import FileInput
  File "/usr/lib/python2.7/dist-packages/docutils/io.py", line 18, in
    from docutils.utils.error_reporting import locale_encoding, ErrorString, ErrorOutput
  File "/usr/lib/python2.7/dist-packages/docutils/utils/__init__.py", line 21, in
    from docutils.utils.error_reporting import ErrorOutput, SafeString
  File "/usr/lib/python2.7/dist-packages/docutils/utils/error_reporting.py", line 47, in
    locale_encoding = locale.getlocale()[1] or locale.getdefaultlocale()[1]
AttributeError: 'module' object has no attribute 'getlocale'

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/sphinx/__main__.py", line 14, in
    sys.exit(main(sys.argv))
  File "/usr/lib/python2.7/dist-packages/sphinx/__init__.py", line 51, in main
    sys.exit(build_main(argv))
  File "/usr/lib/python2.7/dist-packages/sphinx/__init__.py", line 61, in build_main
    from sphinx import cmdline
  File "/usr/lib/python2.7/dist-packages/sphinx/cmdline.py", line 14, in
    import optparse
  File "/usr/lib/python2.7/optparse.py", line 419, in
    _builtin_cvt = { "int" : (_parse_int, _("integer")),
  File "/usr/lib/python2.7/gettext.py", line 584, in gettext
    return dgettext(_current_domain, message)
  File "/usr/lib/python2.7/gettext.py", line 548, in dgettext
    codeset=_localecodesets.get(domain))
  File "/usr/lib/python2.7/gettext.py", line 483, in translation
    mofiles = find(domain, localedir, languages, all=1)
  File "/usr/lib/python2.7/gettext.py", line 440, in find
    for nelang in _expand_lang(lang):
  File "/usr/lib/python2.7/gettext.py", line 133, in _expand_lang
    from locale import normalize
ImportError: cannot import name normalize
Makefile:23: recipe for target 'html' failed
make: *** [html] Error 1

Travis Permissions

If you are receiving error message user-name has not granted Travis CI the required permissions, please log in via travis-ci.com, it’s caused by using travis login --pro instead of travis login. You should modify all commands to not to use --pro flag.

---

Tags:

codeclimate, github, howto, integration, open-source, pypi, python, readthedocs, rtfd, travis-cl,

Archive

2017-07-03