container cleanup fails to remove containers and prune etcd sometimes

[tnozicka]

If the issue is the clean up that shouldn't make the job fail.

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_install_update/4730/consoleFull

Ran 26 of 815 Specs in 747.166 seconds
SUCCESS! -- 26 Passed | 0 Failed | 4 Pending | 785 Skipped Aug 24 13:39:03.339: INFO: Error running cluster/log-dump.sh: fork/exec /data/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/cluster/log-dump.sh: no such file or directory
PASS

Ginkgo ran 1 suite in 12m27.601143992s
Test Suite Passed
[INFO] [CLEANUP] Beginning cleanup routines...
[INFO] [CLEANUP] Dumping cluster events to _output/scripts/conformance/artifacts/events.txt
Logged into "https://ip-172-18-7-176.ec2.internal:8443" as "system:admin" using existing credentials.

You have access to the following projects and can switch between them with 'oc project <projectname>':

  * default
    kube-public
    kube-system
    logging
    management-infra
    openshift
    openshift-infra
    openshift-template-service-broker

Using project "default".
[INFO] [CLEANUP] Dumping container logs to _output/scripts/conformance/logs/containers
[INFO] [CLEANUP] Truncating log files over 200M
[INFO] [CLEANUP] Stopping docker containers
[INFO] [CLEANUP] Removing docker containers
Error: No such image, container or task: e10b2c583072
json: cannot unmarshal array into Go value of type types.ContainerJSON
Error: No such image, container or task: 436016e23790
json: cannot unmarshal array into Go value of type types.ContainerJSON
Error: No such image, container or task: 95a17261f104
json: cannot unmarshal array into Go value of type types.ContainerJSON
Error: No such image, container or task: e575bcc0f9bd
json: cannot unmarshal array into Go value of type types.ContainerJSON
Error: No such image, container or task: 0dfcdd07f86b
json: cannot unmarshal array into Go value of type types.ContainerJSON
Error response from daemon: You cannot remove a running container 6c9004f2e4e1f633c61626d05dffc80563d532f7ddccc025184584ecaac25e3c. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container bd680458d7e7d8587623726f018e2d26844fa3c0b4c03a6bcfde825cf94bfb71. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 7c2736385c04345698ef1fc79c007852fe2a6df8866d727bcae649d7fd545237. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 8b54617e9c437731e57d9dcfb244700a35c27e96292856450919e158d60c226d. Stop the container before attempting removal or use -f
[INFO] [CLEANUP] Killing child processes
[INFO] [CLEANUP] Pruning etcd data directory
rm: cannot remove ‘/tmp/etcd/openshift-backup-etcd_backup_tag20170824115222’: Permission denied
rm: cannot remove ‘/tmp/etcd/openshift-backup-etcd_backup_tag20170824115200’: Permission denied
rm: cannot remove ‘/tmp/etcd/member’: Permission denied
[ERROR] test/extended/conformance.sh exited with code 1 after 01h 41m 57s
make: *** [test-extended] Error 1
++ export status=FAILURE
++ status=FAILURE
+ set +o xtrace
########## FINISHED STAGE: FAILURE: RUN EXTENDED TESTS [01h 43m 02s] ##########

[stevekuznetsov]

No, please look at jUnit output:

Summarizing 2 Failures:

[Fail] [Conformance][templates] templateservicebroker security test [It] should pass security tests 
/go/src/github.com/openshift/origin/test/extended/templates/templateservicebroker_security.go:161

[Fail] [Conformance][templates] templateservicebroker end-to-end test [It] should pass an end-to-end test 
/go/src/github.com/openshift/origin/test/extended/templates/templateservicebroker_e2e.go:148

Ran 238 of 815 Specs in 5330.335 seconds
FAIL! -- 236 Passed | 2 Failed | 4 Pending | 573 Skipped 

[stevekuznetsov]

Cleanup is best-effort and cannot fail a job.

[tnozicka]

@stevekuznetsov seems like I was wrong in the cause although the cleanup seem broken:

Error response from daemon: You cannot remove a running container

Why not use -f there?

This seems wrong as well:

rm: cannot remove ‘/tmp/etcd/openshift-backup-etcd_backup_tag20170824115222’: Permission denied
rm: cannot remove ‘/tmp/etcd/openshift-backup-etcd_backup_tag20170824115200’: Permission denied
rm: cannot remove ‘/tmp/etcd/member’: Permission denied

[stevekuznetsov]

I mean, yes, the cleanup is failing, but it doesn't matter. if you'd like to clean it up, you may, but it is not impacting any workflows today.