Kubernetes Authentication failure 61 minutes after starting.

[cpressland]

Which project are you reporting a bug for?

kr8s

What happened?

When using kr8s as part of a FastAPI application authentication begins failing 61 minutes after starting.

Sample code:

from pathlib import Path

from fastapi import FastAPI, status
from fastapi.responses import JSONResponse, Response
from kr8s.objects import ConfigMap

app = FastAPI()

def get_current_namespace() -> str:
    """Get the current namespace."""
    return Path("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read_text().strip()


configmap = ConfigMap("example", namespace=get_current_namespace())
if not configmap.exists():
    configmap.create()

@app.get("/health/readiness")
def readiness() -> Response:
    """Readiness check endpoint, returns a JSON response with status 'ok'."""
    configmap.refresh()
    if configmap.exists():
        return JSONResponse({"status": "ok"}, status_code=status.HTTP_200_OK)
    return JSONResponse({"status": "error"}, status_code=status.HTTP_503_SERVICE_UNAVAILABLE)


@app.get("/health/liveness")
def liveness() -> Response:
    """Liveness check endpoint, returns a JSON response with status 'ok'."""
    return JSONResponse({"status": "ok"}, status_code=status.HTTP_200_OK)

Logs:

First health check:

2024-12-23T11:31:13.528988544Z INFO:     Started server process [1]
2024-12-23T11:31:13.529025895Z INFO:     Waiting for application startup.
2024-12-23T11:31:13.529142835Z INFO:     Application startup complete.
2024-12-23T11:31:13.529409517Z INFO:     Uvicorn running on http://0.0.0.0:6502 (Press CTRL+C to quit)
2024-12-23T11:31:38.700639717Z INFO:     10.40.16.115:49578 - "GET /health/liveness HTTP/1.1" 200 OK
2024-12-23T11:31:38.743960452Z INFO:     10.40.16.115:49580 - "GET /health/readiness HTTP/1.1" 200 OK

Failing health check 61 minutes later:

2024-12-23T12:32:38.699077059Z INFO:     10.40.16.115:47082 - "GET /health/liveness HTTP/1.1" 200 OK
2024-12-23T12:32:38.794716051Z INFO:     10.40.16.115:47084 - "GET /health/readiness HTTP/1.1" 503 Service Unavailable

Exception when calling an endpoint that forces a configmap.refresh():

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/app/lib/python3.13/site-packages/kr8s/_api.py", line 168, in call_api
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/lib/python3.13/site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://172.16.0.1/api/v1/namespaces/sealed-secrets-tools/configmaps/cluster-certificates'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.scope, self.receive, self.send
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/app/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/app/lib/python3.13/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/app/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/app/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/app/lib/python3.13/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/app/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/app/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/app/lib/python3.13/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/app/lib/python3.13/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        func, args, abandon_on_cancel=abandon_on_cancel, limiter=limiter
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/app/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 2505, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 1005, in run
    result = context.run(func, *args)
  File "/app/lib/python3.13/site-packages/sealed_secrets_tools/server/__init__.py", line 119, in get_certificates
    return JSONResponse({"clusters": list(get_configmap().keys())}, status_code=status.HTTP_200_OK)
                                          ~~~~~~~~~~~~~^^
  File "/app/lib/python3.13/site-packages/sealed_secrets_tools/server/__init__.py", line 33, in get_configmap
    configmap.refresh()
    ~~~~~~~~~~~~~~~~~^^
  File "/app/lib/python3.13/site-packages/kr8s/_async_utils.py", line 120, in run_sync_inner
    return portal.call(wrapped)
           ~~~~~~~~~~~^^^^^^^^^
  File "/app/lib/python3.13/site-packages/kr8s/_async_utils.py", line 92, in call
    return self._portal.call(func, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/anyio/from_thread.py", line 290, in call
    return cast(T_Retval, self.start_task_soon(func, *args).result())
                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/root/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/app/lib/python3.13/site-packages/anyio/from_thread.py", line 221, in _call_func
    retval = await retval_or_awaitable
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/kr8s/_objects.py", line 357, in refresh
    await self.async_refresh()
  File "/app/lib/python3.13/site-packages/kr8s/_objects.py", line 375, in async_refresh
    raise e
  File "/app/lib/python3.13/site-packages/kr8s/_objects.py", line 363, in async_refresh
    async with self.api.call_api(
               ~~~~~~~~~~~~~~~~~^
        "GET",
        ^^^^^^
    ...<2 lines>...
        namespace=self.namespace,
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as resp:
    ^
  File "/root/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.13/site-packages/kr8s/_api.py", line 186, in call_api
    raise ServerError(
        error_message, status=error, response=e.response
    ) from e
kr8s._exceptions.ServerError: Unauthorized

Anything else?

No response

[jacobtomlinson]

How is your application authenticating with Kubernetes? Is it using a service account, or something else like OIDC? This could be related to #125

[cpressland]

Thanks @jacobtomlinson - authentication is entirely driven by Service Accounts, with the app only getting ConfigMaps and Secrets from the same namespace.

[jacobtomlinson]

That makes sense, the default service account token expiration time is an hour.

When the API gives an Unauthorized error kr8s attempts a few times to reauthenticate before giving up. There must be a bug with the reauthentication that's specific to your environment, otherwise we would be seeing this as a more widespread problem.

kr8s/kr8s/_api.py

Lines 173 to 175 in 5e93587

	if e.response.status_code in (401, 403) and auth_attempts < 3:
	auth_attempts += 1
	await self.auth.reauthenticate()

kr8s/kr8s/_auth.py

Lines 76 to 77 in 5e93587

	if self._serviceaccount and not self.server:
	await self._load_service_account()

kr8s/kr8s/_auth.py

Lines 264 to 267 in 5e93587

	async with await anyio.open_file(
	os.path.join(self._serviceaccount, "token")
	) as f:
	self.token = await f.read()

There are a few places in the logic that I could see things returning too early and not actually loading the new token.

I've just pushed a new release v0.19.0 which contains #542 to add some more debug logging. Could you update to that version, then run your code again with the logging set to DEBUG? Then share the logs. I'm particularly interested in what happens when the token expires and the auth failures start happening.

[cpressland]

Right, I finally had time to tackle this properly, here is a full Kubernetes Manifest set that should replicate this:

apiVersion: v1
kind: Namespace
metadata:
  name: kr8s-test
  labels:
    team: cpe
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kr8s-test
  namespace: kr8s-test
data:
  test: "test"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kr8s-test
  namespace: kr8s-test
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kr8s-test
  namespace: kr8s-test
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kr8s-test
  namespace: kr8s-test
subjects:
  - kind: ServiceAccount
    name: kr8s-test
    namespace: kr8s-test
roleRef:
  kind: Role
  name: kr8s-test
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kr8s-test
  namespace: kr8s-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kr8s-test
  template:
    metadata:
      labels:
        app: kr8s-test
    spec:
      containers:
        - name: kr8s-test
          image: docker.io/python:latest
          command:
            - sh
            - -c
            - |
              pip install kr8s &&
              cat << 'EOF' > /tmp/script.py
              import logging
              logging.basicConfig(level=logging.DEBUG)
              from time import sleep
              import kr8s
              while True:
                  logging.debug(list(kr8s.get("configmap", namespace="kr8s-test")))
                  sleep(60)
              EOF
              python /tmp/script.py
          resources:
            limits:
              cpu: 100m
              memory: 128Mi
            requests:
              cpu: 100m
              memory: 128Mi
      serviceAccountName: kr8s-test

This produces the following set of debug logs every 60 seconds:

DEBUG:kr8s._auth:Loading kubeconfig
DEBUG:httpcore.connection:close.started
DEBUG:httpcore.connection:close.complete
DEBUG:httpcore.connection:connect_tcp.started host='172.16.0.1' port=443 local_address=None timeout=None socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x75b0b03b31b0>
DEBUG:httpcore.connection:start_tls.started ssl_context=<ssl.SSLContext object at 0x75b0b15cd9a0> server_hostname='172.16.0.1' timeout=None
DEBUG:httpcore.connection:start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x75b0b03b2f80>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'942695fd-c5dd-458f-a5bc-9db8ad63b62b'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'69fc3a70-5198-4527-8ef7-1e0a70bda0a2'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'c2560709-9506-40dd-95cd-b32516a231e4'), (b'Date', b'Mon, 24 Feb 2025 21:10:49 GMT'), (b'Transfer-Encoding', b'chunked')])
INFO:httpx:HTTP Request: GET https://172.16.0.1/api/v1/namespaces/kr8s-test/configmaps?limit=100 "HTTP/1.1 200 OK"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:root:[<ConfigMap kr8s-test>, <ConfigMap kube-root-ca.crt>]

For the iteration where everything geos wrong, this is what we see:

DEBUG:kr8s._auth:Loading kubeconfig
DEBUG:httpcore.connection:close.started
DEBUG:httpcore.connection:close.complete
DEBUG:httpcore.connection:connect_tcp.started host='172.16.0.1' port=443 local_address=None timeout=None socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x75b0b03b3b60>
DEBUG:httpcore.connection:start_tls.started ssl_context=<ssl.SSLContext object at 0x75b0b15cd9a0> server_hostname='172.16.0.1' timeout=None
DEBUG:httpcore.connection:start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x75b0b03b2710>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 401, b'Unauthorized', [(b'Audit-Id', b'acec4a73-47fa-44c0-95ce-3f390c240d1e'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'Date', b'Mon, 24 Feb 2025 21:11:49 GMT'), (b'Content-Length', b'129')])
INFO:httpx:HTTP Request: GET https://172.16.0.1/api/v1/namespaces/kr8s-test/configmaps?limit=100 "HTTP/1.1 401 Unauthorized"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:kr8s._api:Unauthorized 401 error, reauthenticating attempt 0
DEBUG:kr8s._auth:Loading kubeconfig
DEBUG:httpcore.connection:close.started
DEBUG:httpcore.connection:close.complete
DEBUG:httpcore.connection:connect_tcp.started host='172.16.0.1' port=443 local_address=None timeout=None socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x75b0b03b3fc0>
DEBUG:httpcore.connection:start_tls.started ssl_context=<ssl.SSLContext object at 0x75b0b15cfbf0> server_hostname='172.16.0.1' timeout=None
DEBUG:httpcore.connection:start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x75b0b03b3c50>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 401, b'Unauthorized', [(b'Audit-Id', b'3b4bb6ec-af45-40df-b440-1be70ecfff13'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'Date', b'Mon, 24 Feb 2025 21:11:49 GMT'), (b'Content-Length', b'129')])
INFO:httpx:HTTP Request: GET https://172.16.0.1/api/v1/namespaces/kr8s-test/configmaps?limit=100 "HTTP/1.1 401 Unauthorized"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:kr8s._api:Unauthorized 401 error, reauthenticating attempt 1
DEBUG:kr8s._auth:Loading kubeconfig
DEBUG:httpcore.connection:close.started
DEBUG:httpcore.connection:close.complete
DEBUG:httpcore.connection:connect_tcp.started host='172.16.0.1' port=443 local_address=None timeout=None socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x75b0b03b1a90>
DEBUG:httpcore.connection:start_tls.started ssl_context=<ssl.SSLContext object at 0x75b0b15cf9b0> server_hostname='172.16.0.1' timeout=None
DEBUG:httpcore.connection:start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x75b0b03b2a80>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 401, b'Unauthorized', [(b'Audit-Id', b'0d095c64-2cee-4754-9fdd-bed1b0e143a3'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'Date', b'Mon, 24 Feb 2025 21:11:49 GMT'), (b'Content-Length', b'129')])
INFO:httpx:HTTP Request: GET https://172.16.0.1/api/v1/namespaces/kr8s-test/configmaps?limit=100 "HTTP/1.1 401 Unauthorized"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:kr8s._api:Unauthorized 401 error, reauthenticating attempt 2
DEBUG:kr8s._auth:Loading kubeconfig
DEBUG:httpcore.connection:close.started
DEBUG:httpcore.connection:close.complete
DEBUG:httpcore.connection:connect_tcp.started host='172.16.0.1' port=443 local_address=None timeout=None socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x75b0b03b1ea0>
DEBUG:httpcore.connection:start_tls.started ssl_context=<ssl.SSLContext object at 0x75b0b15cf890> server_hostname='172.16.0.1' timeout=None
DEBUG:httpcore.connection:start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x75b0b03b2210>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 401, b'Unauthorized', [(b'Audit-Id', b'83aa3515-d7d5-4e34-9088-1130e3c863a3'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'Date', b'Mon, 24 Feb 2025 21:11:49 GMT'), (b'Content-Length', b'129')])
INFO:httpx:HTTP Request: GET https://172.16.0.1/api/v1/namespaces/kr8s-test/configmaps?limit=100 "HTTP/1.1 401 Unauthorized"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/kr8s/_api.py", line 171, in call_api
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://172.16.0.1/api/v1/namespaces/kr8s-test/configmaps?limit=100'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/script.py", line 6, in <module>
    logging.debug(list(kr8s.get("configmap", namespace="kr8s-test")))
                  ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/kr8s/_async_utils.py", line 146, in iter_over_async
    done, obj = portal.call(get_next)
                ~~~~~~~~~~~^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/kr8s/_async_utils.py", line 91, in call
    return self._portal.call(func, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/anyio/from_thread.py", line 290, in call
    return cast(T_Retval, self.start_task_soon(func, *args).result())
                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.13/site-packages/anyio/from_thread.py", line 221, in _call_func
    retval = await retval_or_awaitable
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/kr8s/_async_utils.py", line 139, in get_next
    obj = await ait.__anext__()
          ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/kr8s/asyncio/_helpers.py", line 58, in get
    async for resource in api.async_get(
    ...<9 lines>...
        yield resource
  File "/usr/local/lib/python3.13/site-packages/kr8s/_api.py", line 466, in async_get
    async for resource in self._async_get_single(
    ...<9 lines>...
        yield resource
  File "/usr/local/lib/python3.13/site-packages/kr8s/_api.py", line 512, in _async_get_single
    async with self.async_get_kind(
               ~~~~~~~~~~~~~~~~~~~^
        kind,
        ^^^^^
    ...<6 lines>...
        **kwargs,
        ^^^^^^^^^
    ) as (obj_cls, response):
    ^
  File "/usr/local/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/kr8s/_api.py", line 405, in async_get_kind
    async with self.call_api(
               ~~~~~~~~~~~~~^
        method="GET",
        ^^^^^^^^^^^^^
    ...<4 lines>...
        **kwargs,
        ^^^^^^^^^
    ) as response:
    ^
  File "/usr/local/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/kr8s/_api.py", line 192, in call_api
    raise ServerError(
        error_message, status=error, response=e.response
    ) from e
kr8s._exceptions.ServerError: Unauthorized

I'm not sure if those logs give you anything useful. But we're running AKS 1.31.2, I'm happy to go retest on a vanilla cluster with nothing else installed if you think it'll help?

[jacobtomlinson]

I tested this reproducer on a local cluster created with kind and it did not reproduce the error. I'll try again with a cloud hosted Kubernetes cluster.

[cpressland]

Thanks @jacobtomlinson - as above I'm more than happy to spin up a fresh cluster on Azure if it helps.

[jacobtomlinson]

Thanks for the offer @cpressland. If you want to do so and spend some time debugging it would be much appreciated, but equally I'd like to see this error with my own eyes to try and understand what is happening.

I can see in the logs you shared that we are attempting to reauthenticate but failing to. When we get into this state I want to figure out why the call to reauthenticate() isn't working.

kr8s/kr8s/_api.py

Line 181 in 33f2cde

await self.auth.reauthenticate()

We should see it call into the kr8s._auth.KubeAuth.reauthenticate() method.

kr8s/kr8s/_auth.py

Lines 71 to 83 in 33f2cde

	async def reauthenticate(self) -> None:
	"""Reauthenticate with the server."""
	async with self.__auth_lock:
	if self._url:
	logger.debug("URL specified manually")
	self.server = self._url
	else:
	if self._kubeconfig_path_or_dict is not False:
	await self._load_kubeconfig()
	if self._serviceaccount and not self.server:
	await self._load_service_account()
	if not self.server:
	raise ValueError("Unable to find valid credentials")

Then it should call kr8s._auth.KubeAuth._load_service_account() and read a new token from disk. Kubernetes mounts service account tokens as a file within the container.

kr8s/kr8s/_auth.py

Line 260 in 33f2cde

async def _load_service_account(self) -> None:

I want to add some more debug logging in these places so we can see what is happening.

The questions I want to answer are:

• Do we actually get to the _load_service_account() call, or is something stopping us?
• Is the token we read from disk different to the expired one we read during the initial authentication?

[jacobtomlinson]

I just ran the reproducer on fresh GKE, AKS and EKS clusters and left them running for over an hour and I'm not able to reproduce the issue. It seems like the credentials are either not expiring in the same way that you're seeing on your cluster, or the refresh is working and new tokens are being picked up.

On all three I'm seeing successful log messages way beyond the 61 minute mark.

DEBUG:kr8s._auth:Loading kubeconfig
DEBUG:httpcore.connection:close.started
DEBUG:httpcore.connection:close.complete
DEBUG:httpcore.connection:connect_tcp.started host='10.100.0.1' port=443 local_address=None timeout=None socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0xffff8e58e0d0>
DEBUG:httpcore.connection:start_tls.started ssl_context=<ssl.SSLContext object at 0xffff8fa7a060> server_hostname='10.100.0.1' timeout=None
DEBUG:httpcore.connection:start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0xffff8e58e260>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'9b2a3c31-0aa1-4ae0-8527-080dec97b6e7'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'28fb39c8-3a
INFO:httpx:HTTP Request: GET https://10.100.0.1/api/v1/namespaces/kr8s-test/configmaps?limit=100 "HTTP/1.1 200 OK"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:root:[<ConfigMap kr8s-test>, <ConfigMap kube-root-ca.crt>]

Note that I used Kubernetes v1.30.8 on GKE, v1.30.9 on AKS and v1.29.13 on EKS.

[jacobtomlinson]

I just merged #579 which adds some more logging. @cpressland could you run your reproducer again and share the log output when it starts failing?

[cpressland]

right, I changed the install command to: pip install git+https://github.com/kr8s-org/kr8s.git, so this should be the main branch instead of v0.20.4: log.txt

I decided to include everything from the pod rather than snippets just in case it includes something else useful to you.