What k8s components can have an effect on Rubbish collector conduct contained in the pod – Technical Dialogue

Hiya. We now have a challenge the place go was used for two functions:

1. Construct executor file for our challenge
2. Construct helm libraries from helm.sh sources

Undertaking shortly saying is about deploying helm charts. Working of that challenge is occurring utilizing kubernetes job. And concern is that the identical job and identical picture with identical parameters, identical requests and identical limits(mainly identical job.yaml) in 2/3 k8s clusters works positive. However on third one it’s working into OOM and we wish to perceive what may cause it. After we’re speaking about job:

1. Reminiscence Requests and limits are the identical values
2. CPU requests and limits – barely completely different

In precise execution we observed:

1. CPU is throttling in limits, however i imagine it’s okay
2. Even when limits is 1m(actually the minimal doable) – it’s nonetheless usually consuming reminiscence on these 2 clusters

However we observed that if we disable rubbish collector utilizing this:
“debug.SetGCPercent(-1)”

reminiscence consumption will increase drastically. So we suspecting that on third setting rubbish collector not working as anticipated

Query is – what are potential k8s setting configurations can have an effect on rubbish collector, so it is going to behave in a different way on the cluster?

Sadly can’t share any supply code resulting from firm coverage. And debugging on downside setting shouldn’t be doable.

Any suggestions and recommendations can be useful

It sounds such as you’re going to only need to experiment utilizing trial and error. Since you possibly can’t debug within the setting in query, perhaps you possibly can create a neighborhood docker picture with restricted reminiscence to attempt to reproduce the error there? Additionally did you discover this possibly-related article?

Hello, @Dean_Davidson

Thanks for the hyperlink. Definitly one thing price to verify. Ye, we experimenting with completely different configuration, we haven’t modified the reminiscence requests/limits as for us it’s necessary to look at reminiscence rising in graphana. Nonetheless making an attempt to experiment and focus on with firm about risk to run job with elevated logging and reminiscence profile catching as in the long run it’s the one risk to verify root trigger. However we have to guarantee the utmost information is collected and to not ask one other spherical of debug on that poor env