SecStorage not downloading templates

[ddegoede]

Fixes / issues

• Multiple records in store_ref cause trouble that only goes away after cleaning the entries
  • Maybe we can find a way to prevent them from being inserted
• Empty template.properties
  • Cause is in KvmDownloadProcessor.java that creates empty file on line 515 and uses it around 545. There is an exception that can be thrown in the middle
• Download seem not to continuie after 404 / error. Fixing url (so no error) and they all download

Fixes for script on secstor vm:
find ./ -type f -size 0
find ./ -type d -empty -delete

[ddegoede]

Another related issue: registering a new multizone template, possibly, removed zones/datacenters are not excluded.

[sanderv32]

This is fixed in PR #905

[ddegoede]

That PR might fix some things, but unfortunately not everything.

[sanderv32]

On destroy of a secondary storage VM will cleanup/evict all templates:
com/cloud/storage/StorageManagerImpl.java:469

Things to do/check:

• Templates downloaded from the past should not be evicted as they can give an 404 because they don't exists anymore. Remove evict from the cleanup method and see what happends.
• Create a lock file in the template directory inprogress.<vmid> so that only one secstoreVM can download an image. There should be a check on this file if it's older then 2 hours and if so then timeout and try again.