On this weblog we’d be discussing three separate case research of Redis working
out of reminiscence. All of the three case research have movies demostrating how the
debugging was executed.
All of the three movies have been ready for my workforce members to indicate methods to go about
debugging. The movies are being introduced “because it was recorded”.
First Case Examine
When a job fails in Sidekiq, Sidekiq places that job in
RetrySet and retries that
job till the job succeeds or the job reaches the utmost variety of retries. By
default the utmost variety of retries is 25. If a job fails 25 instances then that
job is moved to the DeadSet.
By default Sidekiq will retailer upto 10,000 jobs within the deadset.
We had a state of affairs the place Redis was working out of reminiscence. Right here is how the
debugging was executed.
How one can examine the deadset
1ds = Sidekiq::DeadSet.new 2ds.every do |job| 3 places "Job #{job['jid']}: #{job['class']} failed at #{job['failed_at']}" 4finish
Operating the next to view the newest entry to the dataset:
To see the reminiscence utilization following instructions have been executed within the Redis console.
1> reminiscence utilization useless 230042467 3 4> kind useless 5zset
As mentioned within the video great amount of payload was being despatched. This isn’t
the fitting approach to ship information to the employee. Ideally some form of id ought to be
despatched to the employee and the employee ought to be capable of get the mandatory information from
the database based mostly on the obtained id.References
- How one can improve the variety of jobs within the Sidekiq deadset or disable deadset
- Most variety of job retries in Sidekiq
- Most variety of jobs in Sidekiq Deadset
Second case research
On this case the Redis occasion of neetochat
was working out of reminiscence. The Redis occasion had 50MB capability however we have been
getting the next error.1ERROR: heartbeat: OOM command not allowed when used reminiscence > 'maxmemory'.
We have been pushing too many geo information data to Redis and that triggered the reminiscence to
refill. Right here is the video capturing the debugging session.Followings are the instructions that have been executed whereas debugging.
1> ping 2PONG 3 4> information 5 6> information reminiscence 7 8> information keyspace 9 10> keys *failed* 11 12> keys *course of* 13 14> keys *geocoder* 15 16> get getocoder:http://ipinfo.io/41.174.30.55/geo?
Third Case Examine
On this case the authentication service of neeto was
failing due to reminiscence exhaustion.Right here the variety of keys was restricted however the payload information was big and all that
payload information was hogging the reminiscence. Right here is the video capturing the debugging
session.Followings are the instructions that have been executed whereas debugging.
1> ping 2 3> information keyspace 4db0:keys=106, expires=86,avg_ttl=1233332728573 5 6> key * (to see all of the keys)
Final command listed all of the 106 keys. Subsequent we would have liked to search out how a lot reminiscence
every of those keys are utilizing. For that the next instructions have been executed.1> reminiscence utilization organizations/subdomains/bigbinary/neeto_app_links 2736 bytes 3 4> reminiscence utilization failed 510316224 (10MB) 6 7> reminiscence utilization useless 829871174 (29MB)