Make update config little more resilent to intermittent failures

[kvalliyurnatt]

Currently There are a few issues with the way we update config

1. We first disable the sandbox validator, wait for the pod to disappear and only then set up the defer function to restore the node labels, If the pod crashes after configuration is applied, then we can be stuck in a state where the sandbox pods are never re enabled.
2. if updateConfig function fails, we only log the error but we don't retry the operation. So a failed configuration apply needs to wait for another configuration update or a pod restart for the configuration to be re applied, which is not great user experience.

Restructring the code to make the workflow a little more resilient to such errors, this is in addition to the change in the GPU operator to help with NVIDIA/gpu-operator#2845

TODO:

• Test the changes against an actual machine with vGPUs