How I’m a Productive Programmer With a Reminiscence of a Fruit Fly

A love letter to instruments that modified all the pieces for me.

Programming Over the Years

Programming received vastly extra diverse in comparison with once I began dabbling in AmigaBASIC within the mid-Nineteen Nineties. Again then you might purchase one very huge ebook in regards to the pc you’re programming and had been 99% there. That ebook, full with earmarks and Put up-its, lay subsequent to you whereas hacking into your monochrome editor, all the time in attain.

These days it could occur that the ebook in your frontend internet framework is thicker than what a C64 programmer wanted to jot down an entire recreation. However, the data for all the pieces that we have to write code at this time is often no a couple of click on away.

No person can think about to pay for developer documentation – each Microsoft and Apple provide their documentation on the internet totally free for everybody. And don’t get me even began about open-source tasks!

In occasions of npm, PyPI, and GitHub, it’s arduous to elucidate that requiring something past what your working system gives, was controversial choice that needed to be weighted judiciously. Usually, you shipped your dependencies alongside along with your product^.

The brand new availability is nice and selection is wholesome, however it results in fragmentation of the data that you’ll want to be productive.

Folks have dozens of tabs open ^{with API docs for the packages they’re utilizing in the meanwhile. As somebody who has labored from rural South Africa, I can inform you that online-only docs aren’t solely an issue when when your Web dies altogether. Particularly if you’ll want to use the search operate of the positioning.}

Should you’re like me, and are a polyglot working with a number of programming languages with monumental sub-communities every (even inside Python, Flask + SQLAlchemy + Postgres is a really completely different beast from writing asyncio-based community servers), it hurts my head to even think about one may bear in mind the arguments of each technique I’m utilizing. Primarily, for those who’re actually like me and barely bear in mind your individual cellphone quantity.

That’s why it was such a life-changing occasion for me when I discovered Sprint in 2012.

API Documentation Browsers

Your thoughts is for having concepts, not holding them.

Sprint offers me the superpower of getting all related APIs one key press away:

• I press ⌥Area and a floating window pops up with an activated search bar,
• I begin typing the tough title of the API or subject,
• I select from the options and land on the image throughout the official challenge documentation,
• I press Escape, the floating window disappears and I can begin typing code instantly as a result of my editor is in focus once more.
• If I overlook what I simply learn, I press ⌥Area once more and the window pops up on the similar place.

All of that is blazing quick – I need to make such a spherical journey beneath 2 seconds. The forgotten bliss of native functions – sure I find out about ‌https://devdocs.io.

Whereas Sprint is a $30 Mac app, there’s the free Home windows and Linux model referred to as Zeal, and a $20 Home windows app referred to as Velocity. In fact there’s additionally at the least one Emacs bundle doing the identical factor: helm-dash.

Which means: you may have this API bliss on any platform! Within the following I’ll solely write about Sprint, as a result of that’s what I’m utilizing, however except famous in any other case, it applies to all of them.

I don’t know why, however there’s additionally numerous plugins for functions like Alfred or VS Code. I similar to to press the identical key mixture regardless of the place I’m and might drill down into the docs. For this reason I don’t discover the numerous internet API browser fascinating in any respect – I need to be out and in of the docs inside 5 seconds.

The one factor they’ve in frequent is the format of the native documentation.

Documentation Units

All of them use Apple’s Documentation Set Bundles (docsets) which might be directories with the HTML documentation, metadata in an XML-based property listing and a search index in a SQLite database:

some.docset
└── Contents
    ├── Information.plist # ← metadata
    └── Sources
        ├── Paperwork # ← root dir of HTML docs
        │   └── index.html
        └── docSet.dsidx  # ← SQLite db w/ search index

When you’ve got a bunch of HTML information in your disk, you may convert it right into a docset that may be consumed by Sprint. It’s simply HTML information with metadata. And because it’s HTML information on your disk, all this works offline.

Subsequently, docsets can substitute documentation that you just already maintain domestically in your pc for sooner and/or offline entry with out doing something particular. Simply bundle it up into the required listing construction, add an empty index, and fill out easy metadata.

Shazam! Now you may conjure them with a single keypress and eliminate them with one other.

Let’s circle again to the boring historical past lesson from the start: there’s a myriad of tasks that I exploit throughout numerous platforms – day-after-day. And I’m not speaking nearly programming APIs right here: Ansible roles, CSS lessons, HAProxy configuration, Postgres (and SQL!) peculiarities…it’s rather a lot.

And whereas Python and Go core documentation ship with Sprint, and whereas Godoc documentation may be added straight by URL^{, regardless of how arduous Sprint will attempt: within the fragmented world of recent software program growth, it would by no means be capable of ship all the pieces I want.}

Sphinx

The largest notable hole for me is Sphinx-based docs that dominate (not solely) the Python ecosystem.

Sphinx is a language-agnostic framework to jot down documentation. Not simply API docs or simply narrative docs: all of it, with wealthy interlinking. It was notorious for forcing reStructuredText on its customers, however these days increasingly more tasks use the fantastic MyST bundle to do it in Markdown. When you’ve got any preconceptions in regards to the look of Sphinx documentation, I urge you to go to the Sphinx Themes Gallery and see how fairly your docs may be. It’s written in Python, however it’s used extensively, together with Apple’s Swift, the LLVM (Clang!) challenge, or wildly in style PHP tasks.

And it gives the precise lacking piece: an index for API entries, sections, glossary phrases, configuration choices, command line arguments, and extra – all distributed all through your documentation any approach you want, however all the time mutually linkable. I discover this excellent notably for those who observe a scientific framework like ‌Diátaxis.

The important thing part that makes this attainable is technically simply an extension: intersphinx. Initially made for inter-project linking – therefore the title – it gives a machine-readable index for us to take. That index grew so in style that it’s now supported by the MkDocs extension mkdocstrings and pydoctor. You’ll be able to acknowledge intersphinx-compatible documentation precisely by that index file: objects.inv.

And that’s why, 10 years in the past nearly to the day, I’ve began the doc2dash challenge.

doc2dash

doc2dash is a command line device which you can get from my Homebrew faucet, obtain one of many pre-built binaries for Linux, macOS, and Home windows from its launch web page, or set up from PyPI.

Then, all it’s important to do is to level it at a listing with intersphinx-compatible documentation and it’ll do all the pieces obligatory to present you a docset.

Please notice that the title is doc2dash and never sphinx2dash. It was all the time meant as a framework for writing high-quality converters, the primary ones being Sphinx and pydoctor. That hope sadly didn’t work out, as a result of – understandably – each group needed to make use of their very own language and instruments.

These instruments often look fairly one-off to me although, so I’d prefer to re-emphasize that I might love to work with others including assist for different documentation codecs. Don’t reinvent the wheel, the framework is all there! It’s only a bunch of strains of code! You don’t even must share your parser with me and the world.

The truth that each Sprint and doc2dash have existed properly over a decade and I nonetheless see associates have a bazillion tabs with API docs open has been positively heartbreaking for me. I maintain displaying individuals Sprint in motion they usually maintain saying it’s cool, and put it on their sometime listing. Barring one other nudge, sometime by no means comes.

Whereas the fruit-fly a part of this text ends right here, let me attempt to provide you with that nudge with a step-by-step tutorial such that at this time turns into sometime!

Tutorial: Convert and Submit Your Docs

The purpose of this tutorial is to show you the right way to convert intersphinx-compatible documentation to a docset and the right way to submit it to Sprint’s user-generated docset registry, such that others don’t must duplicate your work.

I’ll assume you might have picked and put in your API browser of selection. It doesn’t matter which one you employ, however the tutorial makes use of Sprint. For optionally submitting the docset on the finish, you’ll additionally want a primary understanding of GitHub and its pull request workflow.

I will probably be utilizing this tutorial as an event to lastly begin publishing docsets of my very own tasks, beginning with structlog. I counsel you choose an intersphinx-compatible challenge that isn’t supported by Sprint but and whose documentation’s tab you go to most frequently.

Let’s do that!

Getting doc2dash

Should you’re already utilizing Homebrew, the simplest approach to get doc2dash is to make use of my faucet:

$ brew set up hynek/faucet/doc2dash

There are pre-built bottles for Linux x86-64 and macOS on each x86-64 and Apple silicon, so the set up needs to be very quick.

Until you already know your approach round Python packaging, the following finest approach are pre-built binaries from the launch web page. At present it gives binaries for Linux, Home windows, and macOS – all on x86-64. I hope to supply extra sooner or later if this proves to be in style.

Lastly, you may get it from PyPI. I strongly advocate to make use of pipx and the simplest approach to run doc2dash with it’s:

$ pipx run doc2dash --help

Constructing Documentation

Subsequent comes the most important drawback – and supply of frequent function requests for doc2dash: you want the documentation in an entire, constructed kind. Normally that implies that it’s important to obtain the repository and determine the right way to construct the docs earlier than even putting in doc2dash as a result of most documentation websites sadly don’t provide a obtain of the entire thing.

My private heuristic is to search for a tox.ini or noxfile.py first and see if it builds the documentation. If it doesn’t, I search for a readthedocs.yml, and if even that lets me down, I’m looking out for information named like docs-requirements.txt or non-compulsory set up targets like docs. My ultimate hope goes via pages of YAML and inspecting CI configurations.

When you’ve managed to put in all dependencies, it’s often only a matter of make html within the documentation listing.

After figuring this out, it is best to have a listing referred to as _build/html for Sphinx or website for MkDocs.

Changing

Following the toughest step comes the simplest one: changing the documentation we’ve simply constructed right into a docset.

All it’s important to do is pointing doc2dash on the listing with HTML documentation and wait:

That’s all!

doc2dash is aware of the right way to extract the title from the intersphinx index and makes use of it by default (you may override it with --name). It is best to be capable of add this docset to an API browser of your selection and all the pieces ought to work.

Should you go --add-to-dash or -a, the ultimate docset is mechanically added to Sprint when it’s finished. Should you go --add-to-global or -A, it strikes the completed docset to a worldwide listing (one thing like ~/Library/Utility Help/doc2dash/DocSets) and provides it from there. It’s uncommon that I run doc2dash with out -A when creating docsets for myself.

Bettering Your Documentation Set

Sprint’s documentation has a bunch of suggestions of how one can enhance the docset that we constructed within the earlier step. It’s essential to notice that the following 5 steps are strictly non-compulsory and as a rule, I skip them, as a result of I’m lazy.

However on this case, I need to submit the docset to Sprint’s user-contributed registry, so let’s go the total distance!

Set a Major Web page

With Sprint, you may all the time search all put in docsets, however generally you need to restrict the scope of search. For instance once I kind p3: (the colon is important), switches to solely looking out Python 3 docset. Earlier than you begin typing, it gives you a menu beneath the search field whose first merchandise is “Major Web page”.

When changing structlog docs, this important web page is the index which may be helpful, however often not what I need. After I received to the principle web page, I need to browse the narrative documentation.

The doc2dash choice to set the principle web page is --index-page or -I and takes the file title of the web page you need to use, relative to the documentation root.

Confusingly, the file title of the index is genindex.html and the file title of the principle web page is the HTML-typical index.html. Subsequently, we’ll add --index-page index.html to the command line.

Add an Icon

Documentation units can have icons which might be proven all through Sprint subsequent to the docsets’s title and symbols. That’s fairly, but in addition useful to acknowledge docsets sooner and for those who’re looking out throughout a number of docsets, the place an emblem is coming from.

structlog has a cute beaver brand, so let’s use ImageMagick to resize the emblem to 16×16 pixels:

$ magick 
    docs/_static/structlog_logo_transparent.png 
    -resize 16x16 
    docs/_static/docset-icon.png

Now we are able to add it to the docset utilizing the --icon docset-icon.png possibility.

Help On-line Redirection

Offline docs are superior, however generally it may be helpful to leap to the web model of the documentation web page you’re studying proper now. A typical cause is to peruse a more recent or older model.

Sprint has the menu merchandise “Open On-line Web page ⇧⌘B” for that, however it must know the bottom URL of the documentation. You’ll be able to set that utilizing --online-redirect-url or -u.

For Python packages on Learn the Docs you may choose between the steady (final VCS tag) or newest (present important department).

I feel newest makes extra sense, for those who go away the consolation of offline documentation, thus I’ll add:

--online-redirect-url https://www.structlog.org/en/newest/

Placing It All Collectively

We’re finished! Let’s run the entire command line and see the way it appears to be like in Sprint:

$ doc2dash 
    --index-page index.html 
    --icon docs/_static/docset-icon.png 
    --online-redirect-url https://www.structlog.org/en/newest/ 
    docs/_build/html
Changing intersphinx docs from '/Customers/hynek/FOSS/structlog/docs/_build/html' to 'structlog.docset'.
Parsing documentation...
Added 238 index entries.
Patching for TOCs... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

Fantastic:

Discover the icon within the search bar and urgent ⇧⌘B on any web page with any anchor takes me to the identical place within the newest model of the web docs.

Automation

Since I need to create a brand new model of the docsets for each new launch, the creation must be automated. structlog is already utilizing GitHub Actions as CI, so it is sensible to make use of it for constructing the docset too.

For native testing I’ll benefit from doc2dash being a Python challenge and use a tox atmosphere that reuses the dependencies that I exploit when testing documentation itself.

The atmosphere installs structlog[docs] – i.e. the bundle with non-compulsory docs dependencies, plus doc2dash. Then it runs instructions so as:

[testenv:docset]
extras = docs
deps = doc2dash
allowlist_externals =
    rm
    cp
    tar
instructions =
    rm -rf structlog.docset docs/_build
    sphinx-build -n -T -W -b html -d {envtmpdir}/doctrees docs docs/_build/html
    doc2dash --index-page index.html --icon docs/_static/docset-icon.png --online-redirect-url https://www.structlog.org/en/newest/ docs/_build/html
    cp docs/_static/docset-icon@2x.png structlog.docset/icon@2x.png
    tar --exclude='.DS_Store' -cvzf structlog.tgz structlog.docset

Now I can construct a docset simply by calling tox -e docset. Till doc2dash helps hi-res icons, it additionally copies a 32×32 pixels huge model of the emblem straight into the docset.

Doing that in CI is trivial, however entails tons of boilerplate, so I’ll simply hyperlink to the workflow. Notice the upload-artifact motion on the finish that permits me to obtain the constructed docsets from the run summaries^.

At this level we now have a nice docset that’s constructed mechanically. Time to share it with the world!

Submitting

Within the ultimate step we’ll submit our docset to Sprint’s user-contributed repository, so different individuals can obtain it comfortably from Sprint’s GUI. Conveniently, Sprint makes use of an idea for the entire course of that’s most likely acquainted to each open-source aficionado^{: GitHub pull requests.}

First step is checking the Docset Contribution Guidelines. Thankfully we – or in some circumstances doc2dash – have already taken care of all the pieces!

So let’s transfer proper alongside, and fork the https://github.com/Kapeli/Sprint-Consumer-Contributions repo and clone it to your pc.

First, it’s important to copy the Sample_Docset listing into docsets and rename it whereas doing so. For me the command line is due to this fact:

$ cp -a Sample_Docset docsets/structlog

Let’s enter the listing with cd docsets/structlog and take it from there additional.

The principle step is including the docset itself – however as a gzipped tar file. The contribution information even offers us the template for creating it. In my case the command line is:

$ tar --exclude='.DS_Store' -cvzf structlog.tgz structlog.docset

You could have seen that I’ve already finished the tar-ing in my tox file, so I simply have to repeat it over:

$ cp ~/FOSS/structlog/structlog.tgz .

It additionally needs the icons moreover to what’s within the docset, so I copy them from them docset:

$ cp ~/FOSS/structlog/structlog.docset/icon* .

Subsequent, it might like us to fill in metadata within the docset.html file which is easy in my case:

{
    "title": "structlog",
    "model": "22.1.0",
    "archive": "structlog.tgz",
    "creator": {
        "title": "Hynek Schlawack",
        "hyperlink": "https://github.com/hynek"
    },
    "aliases": []
}

Lastly, it needs us to jot down some documentation about who we’re and the right way to construct the docset. After different examples, I’ve settled on the next:

# structlog

<https://www.structlog.org/>

Maintained by [Hynek Schlawack](https://github.com/hynek/).


## Constructing the Docset

### Necessities

- Python 3.10
- [*tox*](https://tox.wiki/)


### Constructing

1. Clone the [*structlog* repository](https://github.com/hynek/structlog).
2. Try the tag you need to construct.
3. `tox -e docset` will construct the documentation and convert it into `structlog.docset` in a single step.

The tox trick is paying off – I don’t have to elucidate Python packaging to anybody!

Don’t overlook to delete stuff from the pattern docset that we don’t use:

$ rm -r variations Sample_Docset.tgz

We’re finished! Let’s test in our modifications:

$ git checkout -b structlog
$ git add docsets/structlog
git commit -m "Add structlog docset"
[structlog 33478f9] Add structlog docset
 5 information modified, 30 insertions(+)
 create mode 100644 docsets/structlog/README.md
 create mode 100644 docsets/structlog/docset.json
 create mode 100644 docsets/structlog/icon.png
 create mode 100644 docsets/structlog/icon@2x.png
 create mode 100644 docsets/structlog/structlog.tgz
 $ git push -u

Trying good – time for a pull request!

Just a few hours later:

Large success: everybody can obtain the structlog Documentation Set now which concludes our little tutorial!

Closing

I hope I’ve each piqued your curiosity in API documentation browsers and demystified the creation of your individual documentation units. My purpose is to turbocharge programmers who – like me – are overwhelmed by all of the packages they’ve to bear in mind whereas getting stuff finished.

My largest hope, although, is that this text evokes somebody to assist me including extra codecs to doc2dash, such that much more programmers get to benefit from the bliss of API documentation at their fingertips.

I’ve finished a horrible job at selling doc2dash previously decade and I hope the following ten years will go higher!