I constructed Sleuth (GitHub – ruth-tirusew/sleuth: experimental… ยท GitHub), a Go CLI device for working integration and state of affairs flows outlined in declarative YAML.
I constructed it to unravel a couple of particular complications I bumped into with API integration testing:
-
Silent API Drift: Endpoints altering payload shapes with out breaking commonplace assertions.
-
Complicated Step Dependencies: Having to manually sequence steps in YAML as an alternative of letting a graph deal with execution order.
-
Database Verification: Eager to confirm database side-effects mid-flow with out risking unintentional writes or state air pollution.
Iโm nonetheless experimenting with the concept and open to collaborations please do let me know what you suppose
I had a glance by means of the repo, particularly the behavioural snapshot checks. I believe that half is fairly attention-grabbing, since API adjustments that donโt break current assertions may be surprisingly straightforward to overlook.
One factor Iโm interested by is the way youโd advocate dealing with snapshot merging in a bigger challenge.
From what I perceive, -merge can develop the accepted response form based mostly on a number of profitable runs. That is smart for fields which might be genuinely elective, however I’m wondering if it might additionally make sudden API adjustments tougher to catch over time, particularly when totally different environments or API variations are concerned.
Possibly conserving separate baselines per surroundings or API model would assist? Iโm not totally certain what the very best method can be right here.
I havenโt run the CLI but, so that is extra of a query from studying the implementation and checks than a bug report.
Iโd have an interest to listen to the way you see groups managing these snapshots in CI as soon as they’ve fairly a couple of flows.
Thanks for the shut learn. Youโre proper about -merge: it solely ever loosens a snapshot, so merging in a run from a nasty surroundings or an previous API model means the snapshot then accepts that form.
What Iโd advocate:
- Maintain one snapshot per surroundings and API model, for instance snapshots//checkout.json, and evaluate every surroundings towards its personal baseline. Right now thatโs accomplished by file naming. Native per-environment help isnโt constructed but past recording the surroundings within the file.
- Use -merge just for a number of known-good runs of the identical circulation towards the identical goal. Thatโs what itโs for: studying which fields are actually elective. Donโt merge throughout environments that ought to behave otherwise.
- Maintain recording and merging out of CI. CI ought to solely evaluate, with -shapes FILE -shape-drift-fail breaking. Recording refuses to overwrite a snapshot that differs, and -force retains a .prev copy. Snapshot updates then present up as a reviewed git diff.
- Deal with required-to-optional adjustments as one thing to overview. Use assertions and contracts for the fields that should all the time be current.
