[New Feature] Add option to run "Automated Cleaning" only for initial provisioning

[mauhau]

We are provisioning a larger set of machines with CAPM3. In these machines we have several disks:

• Small disk for operating system
• Disk for local storage
• For storage nodes: Several disks for Ceph

For a proper lifecycle management the following workflow is necessary:

Initial provisioning

1. Add server to Metal3 and inspect it
2. Boot server into IPA
3. Cleanup all disks
4. Provision node image

Node update

1. Boot server into IPA
2. Cleanup only operating system disk
3. Provision node image

We read the documentation for CAPM3 automatic cleanup and for BMO Cleaning According to this we decided to set automatedCleaningMode to metadata as

When enabled (metadata), automated cleaning kicks off when a node is in the first provisioning and on every deprovisioning

With the expectation, that there is no deprovisioning for node updates.

Unfortunately this is not the case: At node update all disks are cleaned and data is lost!

We found a workaround by setting in the property skip_block_devices'='[{"size":"> 300"}] in Ironic for all nodes after first provisioning.

Of course this is not a real solution.

Any help is appreciated.

[tuminoid]

@Rozzii @elfosardo @dtantsur

[elfosardo]

@mauhau thanks for the report
I see that for "node update" you mean updating the operating system by reprovisioning a node image on the same node, is that correct?
can you please clarify how you're executing the "node update" procedure?
if you trigger a node provision the cleaning will be executed

[mauhau]

Thank you for the reply, @elfosardo.

The node update gets triggered by Cluster-API / CAPM3 after something in the Cluster custom resource is changed like the image version or some settings.

If I understand your last sentence

if you trigger a node provision the cleaning will be executed

correctly this means there is currently no chance to avoid cleanup of every node in a cluster after changes to the Cluster custom resource?

[Rozzii]

I am a bit confused about this @mauhau, these two lines:

With the expectation, that there is no deprovisioning for node updates.

Unfortunately this is not the case: At node update all disks are cleaned and data is lost!

There is indeed deprovisioning happening when CAPI initiated a node upgrade, we simply do not provide upgrade option that would not utilize deprovisioning as we have to write a new disk image to the disk.

If you would like to avoid cleaning all the disks then there are 2 options one is to not use the automated cleaning because the root disk will be always cleaned as a preparation for disik image writing, or there is a way to exclude devices from cleaning via some Ironic config option, I am not sure right now what is the relevant ironic option but @elfosardo might know.

P.S.
Actually you are already using skip_block_devices'='[{"size":"> 300"}] that should be compatible with all the rootDeviceHints so you can specify serial number etc...,

P.S. P.S.
I am also working on these that will allow users to set node specific kernel args so skip_block_devices will be customizable on a per node basis :
#2576
metal3-io/metal3-docs#552

[mauhau]

Thank you for bringing more light into this, @Rozzii. And also the hint to this great upcoming feature!

We require the following:

• Cleaning all disk before initial provisioning
• No cleaning after initial provisioning

If understood correctly at the moment this can only be achieved by changing automatedCleaningMode after initial provisioning. Is that correct?

If so, we would maybe like to bring this feature into Metal3 by providing a Pull Request.

[Rozzii]

Yes you understand the issue properly.

hmmm... I can see the feature you are proposing useful for the wider community so I would be okay with it. This would require a proposal to metal3-docs in order to provide a forum for the community to discuss the proposal, implementation PR can be pushed parallel to the proposal.

My recommended approach would be to extend the automatedCleaningMode with a new value something like automatedCleaningMode: initial-provisioning-metadata or something like this.

[Rozzii]

/triage accepted
/kind feature