[nginx] Rejected requests during update of ingress deployment

[foxylion]

We are using nginx-ingress and hope to get zero downtime deployments. This means that nginx should also be upgraded without any downtime.

Currently we have specified some things in our deployment to ensure that there is always an nginx pod running.

spec:
  ...
  replicas: 2
  minReadySeconds: 10
  template:
    terminationGracePeriodSeconds: 60
    containers:
      - name: ginx-ingress
        ...
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1

Kubernetes now ensures that there is at least on pod of nginx-ingress running.
But when using tools like siege or jmeter we still see a small window where requests get rejected (about half of a second).

        Date & Time,  Trans,  Elap Time,  Data Trans,  Resp Time,  Trans Rate,  Throughput,  Concurrent,    OKAY,   Failed
2017-02-22 15:06:36,   1178,      45.18,           0,       0.10,       26.07,        0.00,        2.52,    1178,       2

Is this caused by ingress or do we have something mis configured in our Kubernetes cluster?

[georgecrawford]

I'm seeing exactly the same thing. I set replicas: 2, and during a deployment this scales to 4 then back to two. However, tailing the nginx logs with --v=2, I see a single config diff where two IPs are added and two are removed - at this point, curl fails for a second or two.

[glerchundi]

I don't know if you're already using it, but depending which version of k8s are you using, default rolling update didn't worked as expected, so I needed to explicitly write the strategy:

[...]
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  minReadySeconds: 10
[...]

And of course follow best practices in terms of SIGTERMs, blabla...

[georgecrawford]

Thanks - I got to that same conclusion last night, and things are better now. I do wonder if there are improvements coming to this project which will make this easier?

[foxylion]

@glerchundi In my case I updated the ingress-nginx deployment and not one of my own deployments. I would expect that ingress does handle an update (new version) of itself without any downtime. Or am I wrong?

Also something about the configuration

    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

Normally it should be enough to leave that to the default. This would result in terminating one instance, creating a new one and so on. This should graceful update the deployment when replicas is >= 2. Right?

[foxylion]

Okay, I found a solution for this particular problem. Delaying the termination of the controller pod for a second will not result in any lost requests. But I think this may be general thing in Kubernetes. If someone wan't to try this too, add this to the deployment configuration:

spec:
  replicas: 2
    spec:
      containers:
        ...
        lifecycle:
          preStop:
            exec:
              command: ["sleep, "1"]

[aledbf]

If someone wan't to try this too, add this to the deployment configuration:

Maybe you can add this as an example?

[foxylion]

@aledbf I'm not so sure it looks for me like it is not only an ingress problem. More of a general Kuberenetes issue. But it is also possible I'm misunderstanding something. Nonetheless it seems to work.

But sure I can add an example if you want.

I created an issue here: kubernetes/kubernetes#43576

[foxylion]

So here is a final update for everyone who is wondering how to achieve zero downtime deployments with the nginx ingress controller.

You have to delay the termination process of a pod for a significant amount of time.

In the worst case the nginx configuration is only updated every 10 seconds. This means it can take up to ten seconds until a terminating pod to be removed from the upstreams in nginx.

It is as mentioned above pretty easy to achieve this. Simply sleep for 15 seconds in a preStop hook:

lifecycle:
  preStop:
    exec:
      command: ["sleep, "15"]

Additionally your backend has to do a graceful shutdown. This means it must process all in flight requests and close keep alive connections correctly. (nginx, apache httpd and tomcat for example can handle this properly out of the box).

[tyranron]

@aledbf can we mention the info provided by @foxylion somewhere in docs?

Current Nginx Ingress Controller docs have section "Why endpoints and not services", but it's not obvious (especially for Kubernetes noobs) that this leads to the problem that rolling restart of the Pods does not work as expected out-of-the-box.

[aledbf]

@tyranron sure. Can you send a PR?