Kubernetes Deep Dive Room

Sign Up Free or Log In to participate!

How are failing pods noticed by svc and what happens

I understand the svc is updating ep based on the API server’s events to scale pods up and down.. but what happens when a pod fails, does svc notice based on not being able to talk to the pod? And then it would update the ep (scratch out the pod?) also, some events would be thrown etc.. Who is responsible to respawn a failing pod?

2 Answers

Hi Simon.

First up, I’d say that the EP object updates itself based on watching the API (rather than the SVC updating the EP). All comms and events etc. go via the API server.

Re noticing a failed Pod… Without looking at the code, I’d say this. Obviously there are different circumstances. However, usually the kubelet of the node running the Pod will notice it has failed and inform the API server (it might perform a retry of its own first – I honestly can’t remember off the top of my head). It is then the API server’s resposibility to find it a new home. The EP object will be watching the API server, and will notice and update itself accordingly. The SVC object will obviously leverage the updated EP object as normal.


Simon Bicskey

Thanks Nigel,

Simon Bicskey

this way is also makes sense why ep and svc are different entities … ep makes the bean counting with API and svc just refers to it as it should know how many beans (i mean pods) are out there… smarties!

There is a Master tunable terminated-pod-gc-threshold  that plays part in this along with the other abstracts e.g. deployments, replica set, and scheduler. so if the pod is part of  ReplicationController, ReplicaSet, or Deployment, then its restart policy is to restart always. unlike if it was a job where its restart policy would be never. so if a pod dies the responsible controller will notice and restart a new one, if a node fails the same case applies. all of these happens via the API server coordination with other controllers and scheduler.

Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!

Get Started
Who’s going to be learning?