.svg)
.svg)
I'm Aaron Pelz, an Engineering Manager at Pinwheel. I love building elegant APIs, using threads in Slack, and coming up with bad names for microservices. One thing I don't love is updating documentation, especially at a fast growing company where we're adding new functionality constantly! Following the launch of our new docs, I wanted to share the story of the technical changes behind the scenes, and how our engineering team made it happen while removing a lot of toil in the process. If you find keeping your docs up to date is cumbersome or error prone, or just want a peek under the hood, I hope you'll find this post helpful.
When Pinwheel first launched our Direct Deposit product, we were building fast. To get customers up and running quickly, we set up a site to host our API documentation powered by Docz and wrote a number of descriptive implementation guides that combined technical explanations with reference specs detailing our API's full behavior. With only a handful of people on the team, engineers deployed a feature then pushed out updates to the appropriate user guides. It was easy enough at first, but as the team and product grew, so did the number of manual edits needed every week. Engineers notoriously dislike process that doesn't scale, and pretty soon we found ourselves looking for a remedy.
The Fastest API in the West
Around the time of our public launch, we also embarked on a major technical upgrade: switching the framework that powered our public API from Express.js to FastAPI, a modern Python framework built on a speedy ASGI server. In addition to performance (a topic for another blog post), FastAPI comes with its own OpenAPI 3 specification generator. OpenAPI 3 is a popular specification that can describe REST APIs like ours, and putting our spec in the wild would make it easier for other developers to understand, test, and integrate with our product. And best of all, FastAPI could introspect on the endpoints we defined in the code itself, generating the entire spec for us—paths, schemas, and tags—out of the box. Using the generator instead of forcing engineers to manually update the docs was a no-brainer.
Paring it Down
After finalizing the swap to FastAPI internally, we eagerly embarked on the journey to wrangle the autogenerated OpenAPI spec into something we could share with the public, and quickly ran into some problems. To start, we have a number of private, internal endpoints that live in the same codebase as our public API. And like many tech companies, we are constantly deploying new versions of code, in testing and production environments, that include beta features not ready for public consumption. That presents a problem when everything is added to the docs by default! We needed a way to conditionally display and modify what FastAPI spat out.
To solve this, we decided to implement custom schema modifications that were based on environment variables present in our various development environments. Doing so let us see private and public endpoints together on our internal tools while keeping them secret from the public until we were ready to launch. The documentation we shared externally would be generated in the same production environment where users interact with our API.
import uuidfrom fastapi import APIRouter, Requestfrom pinwheel import settingsfrom pinwheel.models import PrivateModelfrom pinwheel.managers import get_private_modelrouter = APIRouter()@router.get( "/{object_id}", response_model=PrivateModel, include_in_schema=settings.ENV == "staging")def private_route( request: Request, object_id: uuid.UUID): return get_private_model(object_id)
view rawpinwheel.routes.py hosted with ❤ by GitHub
This was easy enough for entire routes, but modifying individual fields was an added layer of complexity. What if we needed to add a beta field to a response and keep it secret? Fortunately, FastAPI uses Pydantic under the hood to build schemas for requests and responses, which let us define reusable models in our code. When responses exit our servers, Pydantic objects are converted to Python dictionaries, then serialized to JSON. To enforce our API conventions programmatically while allowing environment-specific behavior, we built an abstract model class that all public models inherit from. Now we could pick and choose which fields were omitted in our production environment.
from abc import ABCfrom typing import List, Dict, Type, Anyfrom pydantic import BaseModel, Fieldfrom pinwheel import settingsclass PinwheelPublicBaseModel(BaseModel, ABC): _hide_fields_from_public_docs: List[str] class Config: @staticmethod def schema_extra( schema: Dict[str, Any], model: Type["PinwheelPublicBaseModel"] ) -> None: """ Custom JSON schema modifications for Pydantic models. https://pydantic-docs.helpmanual.io/usage/schema/ """ if settings.ENV == "production": properties = schema.get("properties", {}) for field in model._hide_fields_from_public_docs: properties.pop(field, None)class PublicModel(PinwheelPublicBaseModel): foo: str bar: int secret_beta_feature: str _hide_fields_from_public_docs = ["secret_beta_feature"]
view rawpinwheel.models.py hosted with ❤ by GitHub
Building it Up
Now that we were exclusively generating docs ready for the public in production, we wanted to enrich the OpenAPI spec. FastAPI has a hook to write custom OpenAPI generation code, and combining that with our custom models, we could add some extra flair. For example, to ship response examples along with every successful API response, we added an hook to iterate over all relevant endpoints and populate them with examples generated programatically.
import typesfrom fastapi import FastAPIfrom pinwheel.openapi import custom_openapiapp = FastAPI()app.openapi = types.MethodType(custom_openapi, app)
view rawpinwheel.app.py hosted with ❤ by GitHub
from typing import Dictfrom fastapi import FastAPIfrom fastapi.openapi.utils import get_openapifrom pinwheel.models import PinwheelPublicBaseModeldef add_examples_to_schema(openapi_schema: Dict) -> None: for path, verbs in openapi_schema["paths"].items(): for verb, request in verbs.items(): success_response = request["responses"]["200"]["content"]["application/json"] response_model_ref = success_response["schema"]["$ref"] success_response["examples"] = PinwheelPublicBaseModel.generate_examples( response_model_ref )def custom_openapi(app: FastAPI, *args, **kwargs) -> Dict: """ A custom wrapper around FastAPI's OpenAPI generator. """ openapi_schema = get_openapi( title="Pinwheel", description="Pinwheel is the API for Payroll", version="1.0.0", routes=app.routes ) add_examples_to_schema(openapi_schema) app.openapi_schema = openapi_schema return app.openapi_schema
view rawpinwheel.openapi.py hosted with ❤ by GitHub
This approach also made it easy to define an example in code a single time and reuse it throughout the docs. Repeating the tactic, we can format and enrich every endpoint with tags, additional formatting, and extra descriptions, making the spec more informative and easier to digest. And with everything generated programmatically, we could also write unit tests for the generated spec to ensure it was both fully featured and convention compliant.
Enter Stoplight
Now that we had our fresh, autogenerated API reference, we needed to merge it with our existing documentation to show it side by side with our product and user guides. At the same time, we brought on some amazing product and sales people who wanted to contribute to the docs without needing to check in changes to our Github repo. To get the best of both worlds, we launched a refresh of our docs with a tool called Stoplight, which displays guides side by side with endpoints directly from an OpenAPI spec. Engineers can push changes programmatically while anyone else can edit our guides directly on Stoplight's website. Stoplight also ships with a number of mocking, testing, and code generation tools to help developers test out and build on our product.
Stitching it all together, our docs have come a long way from the early days of manual updates. Now, a fresh copy of our API reference is generated whenever we deploy changes to our product, our commercial team can use the documentation as a foundation for sales conversations, and it's easier than ever to integrate with Pinwheel. FastAPI has lived up to its name, and months later we're building faster than ever.
If you never want to remember to write API reference docs again, join us — we're hiring!
