Hello, everybody! Thanks for making an attempt to assist, I shall be temporary.
To be taught Kafka, I’m making an attempt the next:
- I’ve a REST endpoint that pushes JSON request information into Kafka.
- I would like publishing into Kafka to be asynchronous
- this fashion HTTP handler can instantly return HTTP response 202
I’ve examine Kafka, and watched Confluent’s video on YouTube.
I’m nonetheless not capable of make a assured resolution between utilizing sync or async Kafka producer.
That is the place I want your assist. Earlier than I proceed, enable me to offer some code:
func SomeGinHandler(c *gin.Context) {
// assume we extracted information from request's JSON
// information is stuffed into someValue byte array
/* what comes is a really delicate spot,
as a result of Publish() could place information on the queue,
however nonetheless someway fail; now we have now "soiled" information within the queue
and I see no technique to take away it from queue at this second
*/
err = kafkaProducer.Publish(c.Request.Context(), []byte(someKey), someValue))
if err != nil {
c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
// the purpose is that above Publish() name mustn't block but in addition be dependable
// I'll not lose message from this HTTP handler
// studying by Kafka to this point, I worry this isn't attainable
// I assume I need to make a tradeoff, however what's the appropriate alternative??
// stick to synchronous producer, with ack = waitALL ??
// or perhaps async producer can someway work right here ?? how ??
c.JSON(http.StatusAccepted, gin.H{"message" : "request accepted"})
}
As I’ve described within the code feedback, I would like Kafka publishing code to instantly return, so REST handler doesn’t get blocked.
.
I’ll course of information in a separate microservice, utilizing Kafka client group.
I’ll not afford message loss, and after studying about async producer, I came upon it simply “fires and forgets” the message, so message loss is feasible.
Apparently, I need to make a tradeoff, so what method do you counsel within the above state of affairs, bearing in mind that REST endpoint shall be below heavy load?
Concerning libraries, I’m leaning in the direction of franz-go, however sarama appears okay too. I wish to keep away from CGO dependency, but when wanted confluent’s library will do too. You probably have another library in thoughts I’ll take it into consideration as nicely. The character of the issue I face is of design challenge, so I don’t anticipate library will magically resolve it.
