downward API extended test flake

[smarterclayton]

Uh oh:

Extended.[k8s.io] Downward API volume should update labels on modification [Conformance]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/10550/testReport/junit/(root)/Extended/_k8s_io__Downward_API_volume_should_update_labels_on_modification__Conformance_/

/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:124
Timed out after 120.001s.
Expected
    <string>: content of file "/etc/labels": key1="value1"
    key2="value2"
    ...
    content of file "/etc/labels": key1="value1"
    key2="value2"
    
to contain substring
    <string>: key3="value3"
    
/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:123

Jan 23 15:18:26.827: INFO: At 2017-01-23 15:16:12.227062384 -0500 EST - event for labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0: {default-scheduler } Scheduled: Successfully assigned labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0 to 172.18.4.242
Jan 23 15:18:26.827: INFO: At 2017-01-23 15:16:17.219264543 -0500 EST - event for labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0: {kubelet 172.18.4.242} Pulled: Container image "gcr.io/google_containers/mounttest:0.7" already present on machine
Jan 23 15:18:26.827: INFO: At 2017-01-23 15:16:17.945694933 -0500 EST - event for labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0: {kubelet 172.18.4.242} Created: Created container with docker id eae9aa4c3f70; Security:[seccomp=unconfined]
Jan 23 15:18:26.827: INFO: At 2017-01-23 15:16:19.558858783 -0500 EST - event for labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0: {kubelet 172.18.4.242} Started: Started container with docker id eae9aa4c3f70
Jan 23 15:18:26.827: INFO: At 2017-01-23 15:16:26 -0500 EST - event for labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0: {kubelet 172.18.4.242} FailedMount: MountVolume.SetUp failed for volume "kubernetes.io/downward-api/c8ddb323-e1a8-11e6-9825-0e53c2054dd0-podinfo" (spec.Name: "podinfo") pod "c8ddb323-e1a8-11e6-9825-0e53c2054dd0" (UID: "c8ddb323-e1a8-11e6-9825-0e53c2054dd0") with: remove /mnt/openshift-xfs-vol-dir/pods/c8ddb323-e1a8-11e6-9825-0e53c2054dd0/volumes/kubernetes.io~downward-api/podinfo/resolv.conf: device or resource busy

[smarterclayton]

@pmorie @ncdc have never seen this before

[smarterclayton]

High priority in case this is a severe issue under the covers

[ncdc]

This failed because the kubelet wasn't able to write the updated downward api file after the pod update was issued. The device or resource busy error trying to remove the old data is the problem. Not sure why it hit that error.

[derekwaynecarr]

given andy's comment, do people agree with a lower priority?

[smarterclayton]

Should the downward API retry? Does it already?

[ncdc]

It does retry, but I don't think retrying is specific to downward api. It's just the kubelet and the volume code trying to set up a volume mount.

[smarterclayton]

Ha! https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/5303

[ncdc]

@derekwaynecarr @sjenning @pmorie it's probably worth digging into this when someone has some free time to see what is holding the file open (assuming that's the reason).

[derekwaynecarr]

@pmorie -- can you look at this?