Python and Go : Half III

Sequence Index

Python and Go: Half I – gRPC
Python and Go: Half II – Extending Python With Go
Python and Go: Half III – Packaging Python Code
Python and Go: Half IV – Utilizing Python in Reminiscence

Introduction

In the earlier put up we compiled Go code to a shared library and used it from the Python interactive shell. On this put up we’re going to complete the event course of by writing a Python module that hides the low stage particulars of working with a shared library after which package deal this code as a Python package deal.

Recap – Structure & Workflow Overview

Determine 1

Determine 1 reveals the circulate of knowledge from Python to Go and again.

The workflow we’re following is:

• Write Go code (CheckSignature),
• Export to the shared library (confirm)
• Use ctypes within the Python interactive immediate to name the Go code
• Write and package deal the Python code (check_signatures)

Within the earlier weblog put up we’ve carried out the primary three components and on this one we’re going to implement the Python module and package deal it. We’ll do that is the next steps:

• Write the Python module (checksig.py)
• Write the undertaking definition file (setup.py)
• Construct the extension

Python Module

Let’s begin with writing a Python module. This module may have a Pythonic API and can disguise the low stage particulars of working with the shared library constructed from our Go code.

Itemizing 1: checksig.py

01 """Parallel verify of information digital signature"""
02 
03 import ctypes
04 from distutils.sysconfig import get_config_var
05 from pathlib import Path
06 
07 # Location of shared library
08 right here = Path(__file__).absolute().dad or mum
09 ext_suffix = get_config_var('EXT_SUFFIX')
10 so_file = right here / ('_checksig' + ext_suffix)
11 
12 # Load features from shared library set their signatures
13 so = ctypes.cdll.LoadLibrary(so_file)
14 confirm = so.confirm
15 confirm.argtypes = [ctypes.c_char_p]
16 confirm.restype = ctypes.c_void_p
17 free = so.free
18 free.argtypes = [ctypes.c_void_p]
19 
20 
21 def check_signatures(root_dir):
22     """Test (in parallel) digital signature of all information in root_dir.
23     We assume there is a sha1sum.txt file beneath root_dir
24     """
25     res = confirm(root_dir.encode('utf-8'))
26     if res shouldn't be None:
27         msg = ctypes.string_at(res).decode('utf-8')
28         free(res)
29         elevate ValueError(msg)

Itemizing 1 has the code of our Python module. The code on traces 12-18 may be very a lot what we did within the interactive immediate.

Strains 7-10 take care of the shared library file identify – we’ll get to why we’d like that after we’ll speak about packaging beneath. On traces 21-29, we outline the API of our module – a single operate referred to as check_signatures. ctypes will convert C’s NULL to Python’s None, therefore the if assertion in line 26. On line 29, we sign an error by elevating a ValueError exception.

Notice: Python’s naming conventions differ from Go. Most Python code is following the usual outlined in PEP-8.

Putting in and Constructing Packages

Earlier than we transfer on to the final a part of constructing a Python module, we have to take a detour and see how putting in Python packages work. You possibly can skip this half and head over to code beneath however I imagine this part will allow you to perceive why we write the code beneath.

Right here’s a simplified workflow of what’s Python’s pip (the cousin of “go set up” within the Go world) is doing when it’s putting in a package deal:

Determine 2

Determine 2 reveals a simplified circulate chart of putting in a Python package deal. If there’s a pre-built binary package deal (wheel) matching the present OS/structure it’ll use it. In any other case, it’ll obtain the sources and can construct the package deal.

We roughly break up Python packages to “pure” and “non-pure”. Pure packages are written solely in Python whereas non-pure have code written in different languages and compiled to a shared library. Since non-pure packages include binary code, they have to be constructed specifically for an OS/structure mixture (say Linux/amd64). Our package deal is taken into account “non pure” because it incorporates code written in Go.

We begin by writing a file referred to as setup.py, this file defines the undertaking and incorporates directions on how one can construct it.

Itemizing 2: setup.py

01 """Setup for checksig package deal"""
02 from distutils.errors import CompileError
03 from subprocess import name
04 
05 from setuptools import Extension, setup
06 from setuptools.command.build_ext import build_ext
07 
08 
09 class build_go_ext(build_ext):
10     """Customized command to construct extension from Go supply information"""
11     def build_extension(self, ext):
12         ext_path = self.get_ext_fullpath(ext.identify)
13         cmd = ['go', 'build', '-buildmode=c-shared', '-o', ext_path]
14         cmd += ext.sources
15         out = name(cmd)
16         if out != 0:
17             elevate CompileError('Go construct failed')
18 
19 
20 setup(
21     identify="checksig",
22     model='0.1.0',
23     py_modules=['checksig'],
24     ext_modules=[
25         Extension('_checksig', ['checksig.go', 'export.go'])
26     ],
27     cmdclass={'build_ext': build_go_ext},
28     zip_safe=False,
29 )

Itemizing 2 reveals the setup.py file for our undertaking. On line 09, we outline a command to construct an extension that makes use of the Go compiler. Python has built-in assist for extensions written in C, C++ and SWIG, however not for Go.

Strains 12-14 outline the command to run and line 15 runs this command as an exterior command (Python’s subprocess is like Go’s os/exec).

On line 20, we name the setup command, specifying the package deal identify on line 21 and a model on line 22. On line 23, we outline the Python module identify and on traces 24-26 we outline the extension module (the Go code). On line 27, we override the built-in build_ext command with our build_ext command that builds Go code. On line 28, we specify the package deal shouldn’t be zip protected because it incorporates a shared library.

One other file we have to create is MANIFEST.in. It’s a file that defines all the additional information that have to be packaged within the supply distribution.

Itemizing 3: MANIFEST.in

01 embody README.md
02 embody *.go go.mod go.sum

Itemizing 3 reveals the additional information that ought to be packaged in supply distribution (sdist).

Now we are able to construct the Python packages.

Itemizing 4: Constructing the Python Packages

$ python setup.py bdist_wheel
$ python setup.py sdist

Itemizing 4 reveals the command to construct binary (wheel) package deal and the supply (sdist) package deal information.

The packages are inbuilt a subdirectory referred to as dist.

Itemizing 5: Content material of dist Listing

$ ls dist
checksig-0.1.0-cp38-cp38-linux_x86_64.whl
checksig-0.1.0.tar.gz

In Itemizing 5, we use the ls command to indicate the content material of the dist listing.

The wheel binary package deal (with .whl extension) has the platform data in its identify: cp38 means CPython model 3.8, linux_x86_64 is the operation system and the structure – identical as Go’s GOOS and GOARCH. Because the wheel file identify modifications relying on the structure it’s constructed on, we needed to write some logic in Itemizing 1 on traces 08-10.

Now you need to use Python’s package deal supervisor, pip to put in these packages. If you wish to publish your package deal, you may add it to the Python Package deal Index PyPI utilizing instruments reminiscent of twine.

See instance.py andDockerfile.test-b` within the supply repository for a full construct, set up & use circulate.

Conclusion

With little effort, you may prolong Python utilizing Go and expose a Python module that has a Pythonic API. Packaging is what makes your code deployable and precious, don’t skip this step.

In case you’d wish to return Python sorts from Go (say a checklist or a dict), you need to use Python’s in depth C API with cgo. You can too take a look on the go-python that may ease loads of the ache of writing Python extensions in Go.

Within the subsequent installment we’re going to flip the roles once more, we’ll name Python from Go – in the identical reminiscence house and with virtually zero serialization.