OCPBUGS-59735, OCPBUGS-60643: UPSTREAM: <carry>: temporary measure to deflake flaky tests

[bitoku]

What type of PR is this?

/kind flake

What this PR does / why we need it:

context: https://redhat-internal.slack.com/archives/C01CQA76KMX/p1755784753878089?thread_ts=1755692105.699389&cid=C01CQA76KMX

We're pretty sure that the flakiness is caused by CPU pressure on the cluster, but it'll take some time to decide what to do (e.g. increase node sizes, increases the number of nodes, or reducing the parallelism).

This is a temporary measure to deflake the tests and once we fix the CPU pressure on the cluster, we can drop this change.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[openshift-ci-robot]

@bitoku: No Jira issue with key OCPNODE-59735 exists in the tracker at https://issues.redhat.com/.
Once a valid jira issue is referenced in the title of this pull request, request a refresh with /jira refresh.

In response to this:

What type of PR is this?

/kind flake

What this PR does / why we need it:

context: https://redhat-internal.slack.com/archives/C01CQA76KMX/p1755784753878089?thread_ts=1755692105.699389&cid=C01CQA76KMX

We're pretty sure that the flakiness is caused by CPU pressure on the cluster, but it'll take some time to decide what to do (e.g. increase node sizes, increases the number of nodes, or reducing the parallelism).

This is a temporary measure to deflake the tests and once we fix the CPU pressure on the cluster, we can drop this change.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@bitoku: No Jira issue with key OCPNODE-60643 exists in the tracker at https://issues.redhat.com/.
Once a valid jira issue is referenced in the title of this pull request, request a refresh with /jira refresh.

In response to this:

What type of PR is this?

/kind flake

What this PR does / why we need it:

context: https://redhat-internal.slack.com/archives/C01CQA76KMX/p1755784753878089?thread_ts=1755692105.699389&cid=C01CQA76KMX

We're pretty sure that the flakiness is caused by CPU pressure on the cluster, but it'll take some time to decide what to do (e.g. increase node sizes, increases the number of nodes, or reducing the parallelism).

This is a temporary measure to deflake the tests and once we fix the CPU pressure on the cluster, we can drop this change.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@bitoku: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• bb58f5b|UPSTREAM: : temporary measure to deflake the test caused by CPU pressure: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[openshift-ci-robot]

@bitoku: This pull request references Jira Issue OCPBUGS-59735, which is invalid:

• expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-60643, which is invalid:

• expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

What type of PR is this?

/kind flake

What this PR does / why we need it:

context: https://redhat-internal.slack.com/archives/C01CQA76KMX/p1755784753878089?thread_ts=1755692105.699389&cid=C01CQA76KMX

We're pretty sure that the flakiness is caused by CPU pressure on the cluster, but it'll take some time to decide what to do (e.g. increase node sizes, increases the number of nodes, or reducing the parallelism).

This is a temporary measure to deflake the tests and once we fix the CPU pressure on the cluster, we can drop this change.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[bitoku]

/jira refresh

[openshift-ci-robot]

@bitoku: This pull request references Jira Issue OCPBUGS-59735, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.20.0) matches configured target version for branch (4.20.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

This pull request references Jira Issue OCPBUGS-60643, which is invalid:

• expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[bitoku]

/jira refresh

[openshift-ci-robot]

@bitoku: This pull request references Jira Issue OCPBUGS-59735, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.20.0) matches configured target version for branch (4.20.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

This pull request references Jira Issue OCPBUGS-60643, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.20.0) matches configured target version for branch (4.20.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[bitoku]

/retest-required

[haircommander]

/cc @bertinatto @jacobsee
/lgtm

@bitoku when this merges can you open a bug to track reverting it? I think it can be marked as 4.20 blocker for now

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bitoku, haircommander
Once this PR has been reviewed and has the lgtm label, please assign tkashem for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• DOWNSTREAM_OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@bitoku: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-serial	bb58f5b	link	true	/test e2e-aws-ovn-serial
ci/prow/e2e-gcp	bb58f5b	link	true	/test e2e-gcp
ci/prow/okd-scos-e2e-aws-ovn	bb58f5b	link	false	/test okd-scos-e2e-aws-ovn
ci/prow/e2e-aws-ovn-hypershift	bb58f5b	link	true	/test e2e-aws-ovn-hypershift
ci/prow/e2e-agnostic-ovn-cmd	bb58f5b	link	false	/test e2e-agnostic-ovn-cmd
ci/prow/k8s-e2e-conformance-aws	bb58f5b	link	true	/test k8s-e2e-conformance-aws
ci/prow/e2e-aws-ovn-techpreview-serial	bb58f5b	link	false	/test e2e-aws-ovn-techpreview-serial

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[bertinatto]

/hold

This should go to upstream first, and once it's merged there, we can backport to o/k with the PR number prefixed.

There are cases where flakes won't happen in upstream, but for those cases we should identify the reason and work with upstream to come up with a fix. Here is an example of a fix for a flake that was only happening in OpenShift CI, but we identified the root cause, worked with upstream, and came up with a fix that was later backported to o/k: kubernetes#132502

Feel free to use a PR against o/k to develop a fix, but once you are confident with the patch, work with upstream to get it merged there, and then backport it to o/k.

Please don't merge temporary <carry> patches in o/k, the rebase process is already difficult, this only makes things worse.

[bitoku]

@bertinatto
what makes it difficult is this is mere CPU resource problem, not race condition or timing problem. The test code itself is totally fine.
@haircommander what do you think? will we try raising a PR in upstream?

[bitoku]

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview 10

[openshift-ci]

@bitoku: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/2f97c510-81b3-11f0-8120-9979455fe8f1-0

[bitoku]

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview 10

[openshift-ci]

@bitoku: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/e447e180-8261-11f0-9a5d-4a95e26fd069-0

[bitoku]

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn 10

[openshift-ci]

@bitoku: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/b5e61d50-834e-11f0-8d78-f7d830590706-0