Defining SLOs for ArgoCD

[lukaszgyg]

This thread is going to collect ideas and solutions around the SLO definition for ArgoCD.
ArgoCD components are exposing metrics, which can give us some insides about the ArgoCD health. There is also the example prometheus dashboard definition which can be used by the problem analysis.
The next step for the ArgoCD monitoring is to define SLOs. We believe that having good SLOs provides the highest-quality indication for when we need to take action in order to improve the service quality.
Our current SLO list for ArgoCD consists of three checks:

• check if the availability of the ArgoCD web UI endpoint
• check the error ratio of 500 http requests
• check the ArgoCD reconcile performance

We identified metric argocd_app_reconcile as a good candidate for the service performance check. But, we are still trying to understand how the metric value change will influence the user experience.
Does anybody have experience in the SLO definition for ArgCD? Maybe you are using other/different metrics for your SLOs?
We are curious to hear how you defined SLO for ArgoCD. It could be a starting point for dedicated documentation or project contribution.

[lukepatrick]

Is this what we are hoping for?

https://www.youtube.com/watch?v=IKOMqRMhugw

https://sched.co/1u5d6