[Bug]: unable to use Azure Keyvault to manage secrets

[bastien-helloasso]

Problem

I faced issues while trying to use Azure Keyvault to manage secrets.

Warning FailedMount 4s (x7 over 36s) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : fetching NodePublishSecretRef azure-pipeline-agent/green-agent failed: kubernetes.io/csi: failed to find the secret blue-agent in the namespace azure-pipeline-agent with error: secrets "blue-agent" not found

I have open the following PR: #424

Impact

Unable to use the Azure Keyvault feature to manage secrets.

Short logs

Warning FailedMount 4s (x7 over 36s) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : fetching NodePublishSecretRef azure-pipeline-agent/green-agent failed: kubernetes.io/csi: failed to find the secret blue-agent in the namespace azure-pipeline-agent with error: secrets "blue-agent" not found

Full logs

No response

Deployment

No response

Version

12.0.1

[bastien-helloasso]

Well, while my PR is resolving container creation issue.
There is a core issue due to how Azure Secrets Store CSI manage the secret: the secret is created only while a pod use and mount the secret Provider Class which works fine with Pods... When there is no more job running, the secret is deleted but the Keda ScaleObject referencing the secret (also) to scale up/down according job queue is failing.

Not sure to have any workaround in mind unless keeping a pod running to which would keep the secret available :'(

│ Warning KEDAScalerFailed 30s (x7 over 89m) scale-handler error resolving secrets for ScaleTarget: error resolving secret name &SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:default,},Key:organization │ │ URL,Optional:nil,} for env AZP_URL in namespace azure-pipeline-agent │

[bastien-helloasso]

Checking the HAP.yaml file probably a fix required in the the azureKeyVault spec.
I'll dig more later :)