Kubernetes Ingress 502 / "Connection Refused by Upstream" — Service Chain Debug

Ingress → Service → Endpoints → Pod: a 502 means the chain broke at one link. Test each hop from the outside in and the failing layer names itself.

What you'll see

Root causes

Service has no endpoints (selector/port mismatch)

kubectl get endpoints <svc> — empty = selector labels don't match pods, or the service targetPort doesn't match container ports. Deploy-time label/annotation drift is the classic.

Ingress points at the wrong service name/port

kubectl get ingress -o yaml: backend.service.name/port must match the actual Service (namespace included). Typos survive because ingress admission is name-lazy.

Controller can't reach the pod network

NetworkPolicy blocking the controller's namespace, or the app listening only on 127.0.0.1 inside the container (bind address!).

Fix it

  1. Check the service's endpoints first (most common break)
    kubectl get endpoints <svc> ; kubectl get pods -l <selector> -o wide
  2. Verify the ingress backend matches the service
    kubectl get ingress <ing> -o jsonpath='{.spec.rules[*].http.paths[*].backend}' | python3 -m json.tool
  3. Test in-cluster from the controller's view
    kubectl run ctest --rm -it --restart=Never --image=curlimages/curl -- curl -s -o /dev/null -w '%{http_code}' http://<svc>.<ns>.svc.cluster.local:<port>/
  4. Check the controller's own log for the named upstream
    kubectl -n ingress-nginx logs deploy/ingress-nginx-controller --tail=20 | grep -i 'upstream\|refused' | head

Field note

kubectl get endpoints is the single highest-value command in this debug: empty endpoints = selector/targetPort problem, populated = network or controller problem. Apps binding to 127.0.0.1 break only under k8s (the pod IP isn't 127.0.0.1) — very common with Node apps: listen on 0.0.0.0.

Common questions

Why does direct port-forward work but the Ingress 502s?

port-forward goes straight to the pod; the ingress goes through Service/Endpoints. The break is almost always empty endpoints (selector/targetPort mismatch) or the controller's access to the pod network — check kubectl get endpoints.

The service name looks right. What else?

Cross-namespace backends need explicit namespace handling, and the port number must match the Service's declared port (not the container port) — spec.rules[].backend.service.port targets the service port name/number.

Related fixes

Ship it right the first time

Kustomize base with probes, PDBs, and zero-downtime rollouts already wired.

Kubernetes Production Blueprints — $27 →

One-time. Yours to modify. Instant download from the NinjaOps template store.