The opnfv/functest-kubernetes-security:v1.23 using kube_hunter test suite is not applicable on K8S cluster where strict PSP defined #533
Description
What happened
I executed the kube-hunter test suite on a K8S cluster where strict PSP defined.
podman run -it --env-file ~/opnfv/env \
-v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z \
-v ~/opnfv/config:/root/.kube/config:Z \
-v ~/opnfv/results:/home/opnfv/functest/results:Z \
-v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z \
-v ~/opnfv/cluster-admin.pem:/home/opnfv/functest/cluster-admin.pem:Z \
-v ~/opnfv/cluster-admin-key.pem:/home/opnfv/functest/cluster-admin-key.pem:Z \
opnfv/functest-kubernetes-security:v1.23 /bin/bash
run_tests -t kube_hunter
The log functest-kubernetes.debug.log:
...
2023-02-27 10:53:20,130 - kubernetes.client.rest - DEBUG - response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"kube-hunter\" in pod \"kube-hunter-mz5jk\" is waiting to start: CreateContainerConfigError","reason":"BadRequest","code":400}
2023-02-27 10:53:20,130 - functest_kubernetes.security.security - ERROR - Cannot run kube-hunter
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/functest_kubernetes/security/security.py", line 100, in run
self.deploy_job()
File "/usr/lib/python3.9/site-packages/functest_kubernetes/security/security.py", line 92, in deploy_job
self.pod_log = self.corev1.read_namespaced_pod_log(
File "/usr/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 22929, in read_namespaced_pod_log
return self.read_namespaced_pod_log_with_http_info(name, namespace, **kwargs) # noqa: E501
File "/usr/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 23048, in read_namespaced_pod_log_with_http_info
return self.api_client.call_api(
File "/usr/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/usr/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/usr/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/usr/lib/python3.9/site-packages/kubernetes/client/rest.py", line 239, in GET
return self.request("GET", url,
File "/usr/lib/python3.9/site-packages/kubernetes/client/rest.py", line 233, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '486f38ad-c75f-469c-8393-0fa6c71004f8', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 27 Feb 2023 10:53:25 GMT', 'Content-Length': '217'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"kube-hunter\" in pod \"kube-hunter-mz5jk\" is waiting to start: CreateContainerConfigError","reason":"BadRequest","code":400}
2023-02-27 10:53:20,151 - functest_kubernetes.security.security - ERROR - Cannot process results
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/functest_kubernetes/security/security.py", line 188, in run
self.process_results(**kwargs)
File "/usr/lib/python3.9/site-packages/functest_kubernetes/security/security.py", line 144, in process_results
self.details = json.loads(self.pod_log.splitlines()[-1])
IndexError: list index out of range
2023-02-27 10:53:20,151 - xtesting.ci.run_tests - INFO - Test result:
+---------------------+------------------+------------------+----------------+
| TEST CASE | PROJECT | DURATION | RESULT |
+---------------------+------------------+------------------+----------------+
| kube_hunter | functest | 20:00 | FAIL |
+---------------------+------------------+------------------+----------------+
kubectl describe pod -n kube-hunter-svrzt kube-hunter-mz5jk
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 58s default-scheduler Successfully assigned kube-hunter-svrzt/kube-hunter-mz5jk to cbis-sut1-worker-02
Normal AddedInterface 58s multus Add eth0 [192.168.72.105/32] from calico-network
Normal Pulled 6s (x7 over 57s) kubelet Container image "docker.io/aquasec/kube-hunter:0.3.1" already present on machine
Warning Failed 6s (x7 over 57s) kubelet Error: container has runAsNonRoot and image will run as root (pod: "kube-hunter-mz5jk_kube-hunter-svrzt(5090797f-f697-4841-9f73-a2d10c39f037)", container: kube-hunter)
The POD got the kubernetes.io/psp: restricted PSP
kubectl get psp restricted -o yaml
kubectl get psp restricted -o yaml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"policy/v1beta1","kind":"PodSecurityPolicy","metadata":{"annotations":{"seccomp.security.alpha.kubernetes.io/allowedProfileNames":"*","seccomp.security.alpha.kubernetes.io/defaultProfileName":"runtime/default"},"name":"restricted"},"spec":{"allowPrivilegeEscalation":false,"allowedHostPaths":[{"pathPrefix":"/etc/localtime","readOnly":true},{"pathPrefix":"/usr/share/zoneinfo/","readOnly":true},{"pathPrefix":"/dev/urandom","readOnly":true},{"pathPrefix":"/opt/bin/process-starter","readOnly":true}],"fsGroup":{"ranges":[{"max":65535,"min":1}],"rule":"MustRunAs"},"hostIPC":false,"hostNetwork":false,"hostPID":false,"privileged":false,"readOnlyRootFilesystem":false,"requiredDropCapabilities":["ALL"],"runAsUser":{"rule":"MustRunAsNonRoot"},"seLinux":{"rule":"RunAsAny"},"supplementalGroups":{"ranges":[{"max":65535,"min":1}],"rule":"MustRunAs"},"volumes":["configMap","emptyDir","projected","secret","downwardAPI","persistentVolumeClaim","hostPath","ephemeral"]}}
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
creationTimestamp: "2023-02-07T22:57:09Z"
name: restricted
resourceVersion: "473"
uid: e8242603-ca5d-4c49-a91e-adda30012e7f
spec:
allowPrivilegeEscalation: false
allowedHostPaths:
- pathPrefix: /etc/localtime
readOnly: true
- pathPrefix: /usr/share/zoneinfo/
readOnly: true
- pathPrefix: /dev/urandom
readOnly: true
- pathPrefix: /opt/bin/process-starter
readOnly: true
fsGroup:
ranges:
- max: 65535
min: 1
rule: MustRunAs
requiredDropCapabilities:
- ALL
runAsUser:
rule: MustRunAsNonRoot
seLinux:
rule: RunAsAny
supplementalGroups:
ranges:
- max: 65535
min: 1
rule: MustRunAs
volumes:
- configMap
- emptyDir
- projected
- secret
- downwardAPI
- persistentVolumeClaim
- hostPath
- ephemeral
When I turned off the PSP on systemwide (removing from kubelet config) the test case executed successfully.
Expected behavior
My question, is there any chance to make this test suite applicable to run on K8S cluster where PSP configured? If not, is there a way to skip those part which require higher privilege and execute the remaining (if any) part from the test suite? Or running this test suite is simply not applicable for such systems? In these case, is it possible to add a check to the test and run it before execute the test case itself, so it can print out "This test suite is no applicable to run PSP preconfigured K8S cluster" without even trying to run it.