Skip to content

Conformance harness

forgetest.RunProviderConformance asserts the half of the provider contract the compiler cannot see: which sentinel to return, that a nil result and a nil error never come back together, that a size bound you accepted is actually enforced.

import forgetest "gitlab.com/phpboyscout/go/forge/test"

func TestConformance(t *testing.T) {
    forgetest.RunProviderConformance(t, forgetest.ConformanceConfig{
        NewProvider: func() forge.Provider { return newTestProvider(t) },
        Capabilities: forgetest.Capabilities{
            GetReleaseByTag: true,
            ListReleases:    false, // platform has no listing
        },
        Owner: "acme", Repo: "tool", Tag: "v1.0.0",
    })
}

For the narrative version (why it exists, what to do about each finding) see author a provider. This page is the field list.

What the harness runs

Seventeen checks, in this order. Each is wrapped so a panic from the provider is reported as a violation rather than ending the test run. The first provider to panic would otherwise take every later check down with it.

Check Covers
GetLatestRelease never (nil, nil)
GetReleaseByTag declared support behaves and never returns (nil, nil); undeclared returns ErrNotSupported
ListReleases declared support behaves; undeclared returns ErrNotSupported
DownloadReleaseAsset the redirect contract: a body or a redirect or an error, never a nil reader with an empty redirect and no error
size bounds DownloadChecksumManifest and DownloadSignature honour the caller's maxBytes
ListRepositories containment in the namespace, non-empty and unique Path, fail-closed visibility decoding
ListRepositories early stop yield is not called again after it returns false, and stopping early is not an error
GetFile reads what exists, ErrNotFound for what does not, maxBytes enforced
GetSite ErrNotSupported or ErrNotFound where each belongs
SearchIssues Text is honoured rather than ignored; early stop; issue shape
GetIssue shape, and ErrNotFound rather than a bespoke error
ListComments shape and early stop
CreateIssue sanitisation by default, the Unsanitised opt-out, and at-most-once under a repeated IdempotencyKey
CreateSnippet shape, that the effective visibility is never weaker than requested, a create/get/list round trip, and cleanup
DraftPublisher a tag carrying no release is ErrNotFound rather than a silent success, the published release comes back and is no longer marked a draft, and publishing an already-published release succeeds — including on the repeat, where a provider returning the record it read still reports a draft
RepositoryCreator an empty Name and an empty owner are both refused before any request, a taken name is ErrAlreadyExists rather than the sentinel a status-only mapping produces, and a created repository comes back populated — including beside ErrNotHonoured, which never travels alone
Wikis a malformed path is refused rather than accepted, an absent page is ErrNotFound rather than ErrNotSupported, a create/update round trip where the path written is the path read back, and that ListWikiPages includes what GetWikiPage serves
IssueCommenter at-most-once under a repeated IdempotencyKey, and ErrNotFound for an issue that does not exist
PullRequestCommenter the same, plus that ListRequestComments returns the complete set — and it reports a fixture too small to prove pagination rather than passing on one
PullRequests Find returns an open one for the branch asked for, FindLastMerged returns a merged one with a non-zero MergedAt, and ResolveMergedCommit returns the commit on the target branch rather than the forge's recorded head, or ErrNotFound rather than a guess
GetReleaseByTag missing tag a tag that does not resolve returns ErrReleaseNotFound, not a bespoke error
refusals each status a forge can answer with is named by the right sentinel, including a 403 carrying rate-limit evidence, which is a rate limit rather than a permission failure
logging no ERROR-level record for a condition also returned as an error

Capabilities: declare honestly

The harness checks both directions: an undeclared capability must return ErrNotSupported, and a declared one must not. Over-declaring is caught because it would otherwise skip every check for that method while implying they ran.

Field Declare true when
GetReleaseByTag the platform has an addressable tag concept
ListReleases the platform exposes a release listing
Checksums the provider implements forge.ChecksumProvider and is configured to serve a manifest
Signatures the provider implements forge.SignatureProvider and is configured to serve one
Repositories the provider implements forge.Repositories and can enumerate Namespace
Contents the provider implements forge.Contents and can read FilePath
Sites the provider implements forge.Sites and its forge publishes sites
Issues the provider implements forge.Issues and can read IssueNumber
IssueFiler the provider implements forge.IssueFiler. See the warning below
Snippets the provider implements forge.Snippets and its forge still offers the feature. It writes, see below
PullRequests the provider implements forge.PullRequests and ConformanceConfig.PullRequests names fixtures it can serve. Read-only: the harness never opens one
Wikis the provider implements forge.Wikis and the project actually has a wiki. It writes, see below
DraftPublisher the provider implements forge.DraftPublisher and its forge models an unpublished release. The absent-release check writes nothing; the one that PUBLISHES is gated on DraftFixture.PublishTag
RepositoryCreator the provider implements forge.RepositoryCreator. Its validation and collision checks write NOTHING; the one create is gated separately on RepositoryFixture.CreateIn
IssueCommenter the provider implements forge.IssueCommenter and its forge has an issue tracker. It writes, see below
PullRequestCommenter the provider implements forge.PullRequestCommenter. It writes, see below
ReleasePublisher the provider implements forge.ReleasePublisher. Its three refusal checks write NOTHING; the one write is gated separately on ReleaseFixture.PublishTag

IssueFiler: true files real issues

Its writes are not undone. There is no delete in the contract, and most forges do not offer one.

Point NewProvider at a fixture. A provider aimed at a real tracker will leave real issues in it, visible to everyone watching the project, and the harness cannot tell the difference.

Snippets: true writes too, but cleans up

The check creates a snippet, reads it back, lists it and deletes it. The contract carries a delete, so unlike IssueFiler this is safe to run repeatedly.

A failure part-way through can still leave one behind, so point it at a scope where a stray snippet is acceptable. The harness reports it under DeleteSnippet when cleanup fails, rather than failing silently.

The commenter checks post real comments, and nothing takes them back

IssueCommenter and PullRequestCommenter write to the objects CommentFixture names, and this contract carries no delete. Declaring a number is what arms the write, so a blank field skips the round trip rather than guessing a target.

MissingIssueNumber and MissingPullRequestNumber are separate fields on purpose. Only GitHub shares a number space between the two; on GitLab, Gitea and Bitbucket a number that is a missing issue is very likely a live merge request, and one field would have the harness post "must not be posted" onto it.

PullRequestCommenter needs a fixture big enough to prove pagination

At-most-once is checked by posting twice with one key and asserting a single comment. That passes against a provider reading only the first page, so long as the target is short — the shape where a check reads as coverage and tests nothing.

Point PullRequestNumber at a request carrying more than one page of comments and set PullRequestSpansPages. Every supported forge defaults a listing to 20 or fewer, so more than 20 comments forces a second request. Below that the harness reports the fixture rather than pretending the check was meaningful.

DraftFixture.PublishTag publishes a real release

What it writes becomes visible to everyone the moment it succeeds — that is what the verb is for — and there is no delete here to take it back.

The tag may name a draft or an already-published release: the checks assert the same end state either way, which is what the idempotency guarantee means. So a fixture is not "used up" by a run, and a second run exercises the repeat path on the first call rather than the second.

MissingTag arms the absent-release check and the opt-out probe. Without it both are skipped rather than aimed at something real.

Declaring DraftPublisher: true with an empty Drafts is REPORTED, not quietly skipped. Both first-party adapters shipped that way and every check returned immediately, so the only thing asserted was that forge.As resolved — which reads as coverage. Set MissingTag at least; it writes nothing.

One rule the harness cannot check is the refusal of a tag carrying more than one release. Only GitHub permits the state, a conformant provider can no longer create it, and no fixture can arm it on a live forge — so it is proved per adapter against a stub. A new adapter for a forge that allows several releases per tag must bring its own proof.

RepositoryCreation.CreateIn creates a real repository

Everything else under RepositoryCreator writes nothing — an empty name and an empty owner must both be refused before any request, and the collision check needs only ExistingName, a repository that is already there. Prefer that: it proves the sentinel without the harness creating anything.

MissingNamespace arms the opt-out probe — the check that an undeclared provider answers ErrNotSupported. It has to be a namespace that does not resolve, because this capability's only method writes: a provider ignoring its own declaration must fail at the namespace rather than create something. Without it the probe is skipped rather than guessed at.

Setting CreateIn and CreateName arms the round trip, and there is no delete in this contract. The fixture is deliberately not cleaned up between runs: a second run then meets ErrAlreadyExists on the first call, which the check treats as the expected steady state rather than a failure.

One thing this check cannot see: a silently dropped DefaultBranch. forge.Repository carries no default-branch field, so the harness has nothing to compare against, and adding one for its benefit would be the tail wagging the dog. That guarantee is proved per adapter, on the two forges that discard the field.

Wikis: true writes, and the wiki must already exist

The round trip creates and updates a page at WikiFixture.WritePath, which stays afterwards. On GitHub a wiki does not exist until a human creates its first page in a browser, and has_wiki: true is a default rather than evidence — so an unarmed fixture there produces ErrNotSupported from every method, which is the contract working rather than a failure.

PullRequests needs an armed fixture or it proves nothing

The check that matters is ResolveMergedCommit, and it can only catch a provider that returns the forge's recorded head if the fixture's recorded head differs from the commit that landed.

Set PullRequestFixture.StaleHead to the head the forge reports and MergedCommit to the commit actually on the target branch. If the two are equal the harness reports the fixture, because a check that passes whatever the provider does is worse than no check: it reads as coverage.

Leaving StaleHead empty is allowed, and the check is then unarmed rather than passed. Point it at a pull request the forge auto-rebased.

Most providers declare Sites: false, and that is the capability pattern working rather than a gap: two of the four first-party forges have no site feature at all.

ConformanceConfig: required, optional, and what a blank field costs

An omitted optional field means the corresponding check is skipped, not assumed satisfied. That is deliberate, because a guessed fixture produces a verdict nobody can trust. It does mean a sparse config runs a much weaker suite than it looks like it is running.

Field Required when Blank means
NewProvider always t.Fatal, and the harness will not run
Capabilities always in practice every capability false, so only the opt-out protocol is checked
Owner, Repo always in practice passed through as empty strings to every method
Tag Capabilities.GetReleaseByTag the tag lookup has nothing to find
NewProviderWithLogger never the ERROR-level logging rule is unverified rather than satisfied
MissingTag never the ErrReleaseNotFound check on GetReleaseByTag is skipped, and the logging check has one fewer failing lookup to drive
NewProviderAt never every refusal check is skipped. See below
Namespace Capabilities.Repositories enumeration has no namespace to walk
FilePath, FileRef Capabilities.Contents the file read has nothing to fetch
MissingFilePath never the ErrNotFound check on GetFile is skipped
SiteRepo, MissingSiteRepo never consulted only when Capabilities.Sites is true
UnknownVisibilityPath never the fail-closed visibility check is skipped, and it is the only mechanical check on that rule
IssueNumber Capabilities.Issues GetIssue has nothing to fetch
MissingIssueNumber never the ErrNotFound check on GetIssue is skipped
IssueSearchText never the search runs unfiltered
EchoesCreatedBody never the sanitisation comparisons are skipped

Four of those are worth supplying even though nothing forces you to.

UnknownVisibilityPath names a repository your fixture reports without a privacy field, so the harness can prove the provider decodes fail-closed rather than letting an absent field become false and therefore public. Providers whose SDK models privacy as a plain bool are exactly the ones that need it.

EchoesCreatedBody declares that your fixture returns the body it was sent rather than a canned one. The sanitisation checks compare what came back against what went in, so a canned body would make the redaction check pass without redacting anything.

NewProviderWithLogger is the only way the harness can inject a logger, because NewProvider hands back an already-built provider. Without it, the rule that a provider never logs at ERROR for a condition it also returns goes unchecked.

NewProviderAt builds your provider against a URL the harness supplies, rather than against your fixture:

NewProviderAt: func(baseURL string) forge.Provider {
    return newProviderAt(baseURL, nil)
},

It exists because three of the refusals cannot be provoked from a fixture. Nothing makes a healthy forge rate-limit a test, so the harness stands up its own server, answers with one status and set of headers, and checks what your provider made of it. If your NewProvider already closes over a server URL, supplying this is usually a two-line refactor.

Without it the refusal mapping is unverified rather than assumed satisfied, which is the whole point: an unsentinelled refusal is the defect this check exists for, and it is invisible until a caller needs to branch on it.

What the refusal checks will not decide for you

Which not-found sentinel a 404 carries is yours to choose. The harness accepts either ErrNotFound or ErrReleaseNotFound, because what is missing depends on what your provider addressed. What it will not accept is a 404 carrying neither.

A server fault is checked with a 501, not a 500. Every SDK in this family retries a 500 with backoff, and a fixture answering 500 to every request pays that in full (twelve seconds, in the case that prompted the change) before the check can look at the error. 501 makes the same point at no cost.

Capturing logs in your own tests

forgetest.CaptureLogger returns a logger and a function yielding the records it collected. It wraps exactly as forge.NormaliseLogger does, so what you assert on is what a consumer would see, redaction included.