Testing Automation – Web site Suggestions

I constructed Sleuth (GitHub – ruth-tirusew/sleuth: experimental… · GitHub), a Go CLI instrument for operating integration and state of affairs flows outlined in declarative YAML.

I constructed it to unravel a number of particular complications I bumped into with API integration testing:

• Silent API Drift: Endpoints altering payload shapes with out breaking commonplace assertions.

• Complicated Step Dependencies: Having to manually sequence steps in YAML as a substitute of letting a graph deal with execution order.

• Database Verification: Desirous to confirm database side-effects mid-flow with out risking unintentional writes or state air pollution.

I’m nonetheless experimenting with the thought and open to collaborations please do let me know what you suppose

I had a glance by the repo, particularly the behavioural snapshot exams. I believe that half is fairly attention-grabbing, since API adjustments that don’t break current assertions could be surprisingly straightforward to overlook.

One factor I’m interested in is the way you’d suggest dealing with snapshot merging in a bigger challenge.

From what I perceive, -merge can increase the accepted response form primarily based on a number of profitable runs. That is smart for fields which are genuinely non-compulsory, however I’m wondering if it might additionally make sudden API adjustments tougher to catch over time, particularly when completely different environments or API variations are concerned.

Possibly protecting separate baselines per surroundings or API model would assist? I’m not fully certain what one of the best strategy can be right here.

I haven’t run the CLI but, so that is extra of a query from studying the implementation and exams than a bug report.

I’d have an interest to listen to the way you see groups managing these snapshots in CI as soon as they’ve fairly a number of flows.

Thanks for the shut learn. You’re proper about -merge: it solely ever loosens a snapshot, so merging in a run from a foul surroundings or an outdated API model means the snapshot then accepts that form.

What I’d suggest:

• Maintain one snapshot per surroundings and API model, for instance snapshots//checkout.json, and examine every surroundings towards its personal baseline. Immediately that’s carried out by file naming. Native per-environment help isn’t constructed but past recording the surroundings within the file.
• Use -merge just for a number of known-good runs of the identical circulate towards the identical goal. That’s what it’s for: studying which fields are actually non-compulsory. Don’t merge throughout environments that ought to behave otherwise.
• Maintain recording and merging out of CI. CI ought to solely examine, with -shapes FILE -shape-drift-fail breaking. Recording refuses to overwrite a snapshot that differs, and -force retains a .prev copy. Snapshot updates then present up as a reviewed git diff.
• Deal with required-to-optional adjustments as one thing to overview. Use assertions and contracts for the fields that should all the time be current.

I believe your “type” discipline is superfluous. type: “db” is straight adopted by “db: …” so it’s redundant and susceptible to interrupt on copy & pasting or enhancing.