Delete resource on host deletes resource inside vcluster

[SCLogo]

What happened?

With Vcluster 0.25.0 if I create a vcluster and delete kube-dns service on host, syncer deletes the same service inside vclsuter as well

What did you expect to happen?

Syncer should sync back the deleted resource to hostcluster

How can we reproduce it (as minimally and precisely as possible)?

create a vcluster with following values:

controlPlane:
  backingStore:
    etcd:
      deploy:
        enabled: true
        statefulSet:
          highAvailability:
            replicas: 3
          image:
            registry: private-registry
            repository: k8s/etcd
            tag: 3.5.21-0
          persistence:
            volumeClaim:
              storageClass: gp3-delete-encrypted
  coredns:
    deployment:
      image: private-registry/dockerhub/coredns/coredns:1.12.0
      replicas: 3
      topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              k8s-app: kube-dns
          maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
        - labelSelector:
            matchLabels:
              k8s-app: kube-dns
          maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
  distro:
    k8s:
      enabled: true
      image:
        registry: private-registry/ghcr
      version: v1.33.0
  statefulSet:
    highAvailability:
      replicas: 3
    image:
      registry: private-registry
      repository: ghcr/loft-sh/vcluster-oss
      tag: 0.25.0
    scheduling:
      podManagementPolicy: OrderedReady
experimental: {}
integrations:
  metricsServer:
    enabled: true
networking:
  replicateServices:
    fromHost:
      - from: istio-system/loki-stack
        to: istio-system/loki-stack
sync:
  fromHost:
    ingressClasses:
      enabled: true
    nodes:
      enabled: true
  toHost:
    ingresses:
      enabled: true
    networkPolicies:
      enabled: true
    persistentVolumes:
      enabled: true
    podDisruptionBudgets:
      enabled: true
    pods:
      rewriteHosts:
        initContainer:
          image: private-registry/dockerhub/library/alpine:3.21
    storageClasses:
      enabled: true

Anything else we need to know?

No response

Host cluster Kubernetes version

$ kubectl version
kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.30.4

vcluster version

$ vcluster --version
vcluster --version
vcluster version 0.25.0

VCluster Config

controlPlane:
  backingStore:
    etcd:
      deploy:
        enabled: true
        statefulSet:
          highAvailability:
            replicas: 3
          image:
            registry: private-registry
            repository: k8s/etcd
            tag: 3.5.21-0
          persistence:
            volumeClaim:
              storageClass: gp3-delete-encrypted
  coredns:
    deployment:
      image: private-registry/dockerhub/coredns/coredns:1.12.0
      replicas: 3
      topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              k8s-app: kube-dns
          maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
        - labelSelector:
            matchLabels:
              k8s-app: kube-dns
          maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
  distro:
    k8s:
      enabled: true
      image:
        registry: private-registry/ghcr
      version: v1.33.0
  statefulSet:
    highAvailability:
      replicas: 3
    image:
      registry: private-registry
      repository: ghcr/loft-sh/vcluster-oss
      tag: 0.25.0
    scheduling:
      podManagementPolicy: OrderedReady
experimental: {}
integrations:
  metricsServer:
    enabled: true
networking:
  replicateServices:
    fromHost:
      - from: istio-system/loki-stack
        to: istio-system/loki-stack
sync:
  fromHost:
    ingressClasses:
      enabled: true
    nodes:
      enabled: true
  toHost:
    ingresses:
      enabled: true
    networkPolicies:
      enabled: true
    persistentVolumes:
      enabled: true
    podDisruptionBudgets:
      enabled: true
    pods:
      rewriteHosts:
        initContainer:
          image: private-registry/dockerhub/library/alpine:3.21
    storageClasses:
      enabled: true

[mayooot]

I think that is expected action: Virtual Service will delete when Host is Service manually deleted, the delete action is in SyncToHost.

But in the v0.18 vcluster that did not happen.

@FabianKramm @cbron So do you think we need to fix it?

[SCLogo]

any update on this ?

[cbron]

@SCLogo looked into this, can you confirm that a restart fixes the issue ? The code to generate resources around coreDNS are run at startup.

[mayooot]

@SCLogo looked into this, can you confirm that a restart fixes the issue ? The code to generate resources around coreDNS are run at startup.

I think vcluster(vanilla kubernetes) resources should not be deleted when the host cluster deletes them. What do you think? I'm happy to implement this feature.

[cbron]

This situation is a bit different than normal resources because its part of control-plane. We would need to refactor some of the startup logic around coreDNS, and that will be more difficult.

[mayooot]

This situation is a bit different than normal resources because its part of control-plane. We would need to refactor some of the startup logic around coreDNS, and that will be more difficult.

In our production environment, we provide vclusters for users, and we have also encountered similar problems.
For example, when users deleted the CoreDNS service or pod in a vcluster, it led to some errors.
Therefore, we need to refactor the sync logic to at least involve CoreDNS.

[squ94wk]

Hi @mayooot,

thanks for raising the issue.
I looked into it a bit.

The Service is recreated at startup, but since it gets a new ClusterIP in the host cluster, existing Pods that still have the old ones configured, will try to look up names at a stale address. See spec.dnsConfig.nameserver on a synced Pod in host cluster.
I assume that's where most problems come from.

I want to say, this behavior wouldn't necessarily be different if the virtual object was always retained. While unlikely, another service may get the (freed) ClusterIP in the host cluster and sync would fail.

Why do you think this should be something that vCluster "fixes"?
I think it's unreasonable to expect everything to continue to work if someone with the necessary permissions attempts to delete a critical resource in the host cluster or the virtual cluster.

If you want to prevent your users to shoot themselves in the foot, you could also install a VAP or something similar.
But again, when is that not a deliberate action?
Let me know if I'm missing something.

[SCLogo]

Hello all, My problem is not only with coredns but all services. the restart did not solve the issue. so IF I deploy vcluster and deploy a simle nginx... then delete the svc on hostcluster. It will delete the svc from vcluster as well. restart did not bring back the svc.