Three case research of debugging redis working out of reminiscence

On this weblog we’d be discussing three separate case research of Redis working
out of reminiscence. All of the three case research have movies demostrating how the
debugging was executed.

All of the three movies have been ready for my workforce members to indicate methods to go about
debugging. The movies are being introduced “because it was recorded”.

First Case Examine

When a job fails in Sidekiq, Sidekiq places that job in
RetrySet and retries that
job till the job succeeds or the job reaches the utmost variety of retries. By
default the utmost variety of retries is 25. If a job fails 25 instances then that
job is moved to the DeadSet.
By default Sidekiq will retailer upto 10,000 jobs within the deadset.

We had a state of affairs the place Redis was working out of reminiscence. Right here is how the
debugging was executed.

How one can examine the deadset

1ds = Sidekiq::DeadSet.new
2ds.every do |job|
3  places "Job #{job['jid']}: #{job['class']} failed at #{job['failed_at']}"
4finish
Copy
Operating the next to view the newest entry to the dataset:
To see the reminiscence utilization following instructions have been executed within the Redis console.
1> reminiscence utilization useless
230042467
3
4> kind useless
5zset

Copy
As mentioned within the video great amount of payload was being despatched. This isn’t

the fitting approach to ship information to the employee. Ideally some form of id ought to be

despatched to the employee and the employee ought to be capable of get the mandatory information from

the database based mostly on the obtained id.
References

How one can improve the variety of jobs within the Sidekiq deadset or disable deadset
Most variety of job retries in Sidekiq
Most variety of jobs in Sidekiq Deadset

Second case research
On this case the Redis occasion of neetochat

was working out of reminiscence. The Redis occasion had 50MB capability however we have been

getting the next error.
1ERROR: heartbeat: OOM command not allowed when used reminiscence > 'maxmemory'.
Copy
We have been pushing too many geo information data to Redis and that triggered the reminiscence to

refill. Right here is the video capturing the debugging session.

Followings are the instructions that have been executed whereas debugging.
1> ping
2PONG
3
4> information
5
6> information reminiscence
7
8> information keyspace
9
10> keys *failed*
11
12> keys *course of*
13
14> keys *geocoder*
15
16> get getocoder:http://ipinfo.io/41.174.30.55/geo?

Copy
Third Case Examine
On this case the authentication service of neeto was

failing due to reminiscence exhaustion.
Right here the variety of keys was restricted however the payload information was big and all that

payload information was hogging the reminiscence. Right here is the video capturing the debugging

session.

Followings are the instructions that have been executed whereas debugging.
1> ping
2
3> information keyspace
4db0:keys=106, expires=86,avg_ttl=1233332728573
5
6> key * (to see all of the keys)

Copy
Final command listed all of the 106 keys. Subsequent we would have liked to search out how a lot reminiscence

every of those keys are utilizing. For that the next instructions have been executed.
1> reminiscence utilization organizations/subdomains/bigbinary/neeto_app_links
2736 bytes
3
4> reminiscence utilization failed
510316224 (10MB)
6
7> reminiscence utilization useless
829871174 (29MB)

Copy