From PR to published — anatomy of an AI-drafted release note
People ask us what releasepls actually does in the ~30 seconds between a merge and a draft showing up in their inbox. Here's the whole loop, step by step.
1. GitHub fires the webhook
When a pull request is merged into the default branch, GitHub delivers a pull_request event to our app. The delivery includes a signature header (x-hub-signature-256) that we verify against the per-installation webhook secret. Anything that fails verification is rejected with a 401 before any work happens.
2. Deduplication
GitHub retries deliveries on 5xx responses for up to 24 hours. We key idempotency on x-github-delivery (a UUID), so a retry never produces a duplicate entry. The first attempt to insert the row is the one that wins; subsequent attempts hit the unique index and bail out early.
3. Tier enforcement
Before we spend a single Claude token we check two limits: the project owner's monthly entry cap and the private-repo gate (Pro and above). Hitting a limit short-circuits the flow and sends the owner an upgrade email instead of generating an entry they can't use.
4. PR context fetch
We call the GitHub API in parallel for:
- the PR object (title, body, author, labels, merge commit SHA)
- the commit list (capped at 100)
- the file list with additions/deletions
- the unified diff (truncated to ~8,000 tokens so it fits in the model request)
If the PR body references an issue ("Closes #42"), we fetch that issue too. The issue body is almost always written in customer language — it's the "what does the user feel?" framing that PRs sometimes lack.
5. The heuristic pre-score
A pure-function classifier runs on the metadata before any model call. It catches the obvious cases — chore/deps/ci/test labels, internal-only file paths, tiny diffs — and tags them with skip hints. This lets the inbox surface "you can probably skip this" without burning Claude tokens. Cheap, deterministic, and easy to test.
6. Claude drafts the entry
We send Claude Sonnet a structured prompt with the PR context and ask for a JSON object: { title, summary, body, category, version }. We pin the model version and the system prompt so output stays consistent. Token usage from the response is captured and stored on the entry row — that's how the founder admin dashboard knows your per-customer AI margin without us having to ask Anthropic.
7. The decision: draft or publish
By default the entry lands in your inbox as a draft. You can opt in to per-project auto-publish (everything) or label-based rules (auto-publish: yes ships, internal: yes archives). The inbox lets you publish, edit, schedule, or skip with single keystrokes. We deliberately default to "human in the loop" — AI drafts are good, not perfect, and the value of a changelog is the trust it carries.
8. Fan-out
On publish we send the changelog page out to:
- every verified subscriber (one-click unsubscribe in the footer)
- the in-app embed widget (cache-busted)
- the RSS feed
- the project owner (if email-on-publish is on)
Total wall time, end to end: about 4–10 seconds depending on the PR size and Claude latency. Total cost: typically under a cent per entry.
That's the whole machine. If something breaks at any step, the error lands in our admin dashboard with the delivery id, so we can replay it through GitHub's "Redeliver" button. Most failures are transient — a Claude timeout, a Postgres reconnect — and GitHub's built-in retry handles them.
The interesting work in releasepls isn't the model call. It's everything around it: signature verification, idempotency, tier gating, the heuristic skip filter, the inbox UX, the fan-out, the failure handling. The AI is one component in a workflow that has to be quietly reliable thousands of times a week.