Hey gophers,
I have to fetch 5000+ objects from the identical endpoint concurrently. The API endpoint returns 500 objects at a time in JSON format, and I’ve to iterate by way of the offset parameter to fetch the remainder of the objects. Every merchandise incorporates details about a particular product, leading to 5000+ distinctive merchandise.
I’ve two major necessities:
a) I wish to make all of the requests in parallel.
b) I wish to discover one of the best strategies for processing the information, together with find out how to save the information and essentially the most appropriate knowledge construction.
I believe that is most likely too obscure to get an excellent detailed response. However – usually APIs that return an offset achieve this as a result of they’re meant to be referred to as serially. Is there a method you may get a depend beforehand so you possibly can decide what number of units of 500 objects you have to pull?
Doing issues concurrently is a comparatively easy downside to resolve. Try this easy instance from The Go Programming Language. You may even learn this a part of that guide at no cost. Scroll to web page 36 of that PDF (web page 17 of the guide). I’ve advisable that guide usually and for good purpose; it’s glorious.
When it comes to knowledge constructions, in the event you’re speaking about Go structs to signify unmarshalled JSON, take some instance JSON and paste it into this instrument to generate a struct. That may not less than be a superb place to begin.
I’ve a whole lot of expertise interacting with exterior APIs and I can inform you this: getting one of the best efficiency right here might be going to contain some trial and error. You may attempt to get too intelligent with concurrency and end up fee restricted or saturate your/their connection. My finest recommendation is: begin with a easy resolution that works and don’t fear about efficiency an excessive amount of but. 5,000 merchandise doesn’t sound like an enormous quantity of information to me so efficiency won’t even be a priority.
In abstract: maintain it easy. Make one thing that works. Tweak it over time to attain higher efficiency. However that’s simply my $.02 and anyone else may need extra particular recommendation! Additionally may assist in the event you point out what API you’re utilizing.
