Fix delete snapshot policy expunged volume

[Damans227]

Description

Fixes: #12441

deleteSnapshotPolicies() uses findById() which fails for expunged volumes. Changed to findByIdIncludingRemoved() to allow deleting orphaned policies.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

• Manually verified fix on test environment
• Added unit tests for deleteSnapshotPolicies()

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.60%. Comparing base (2941b51) to head (e80ed4d).
⚠️ Report is 128 commits behind head on 4.22.

❗ There is a different number of reports uploaded between BASE (2941b51) and HEAD (e80ed4d). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (2941b51)	HEAD (e80ed4d)
unittests	1	0

Additional details and impacted files

@@              Coverage Diff              @@
##               4.22   #12474       +/-   ##
=============================================
- Coverage     17.56%    3.60%   -13.96%     
=============================================
  Files          5910      445     -5465     
  Lines        529128    37634   -491494     
  Branches      64636     6948    -57688     
=============================================
- Hits          92931     1356    -91575     
+ Misses       425740    36112   -389628     
+ Partials      10457      166    -10291     

Flag	Coverage Δ
uitests	3.60% <ø> (+0.01%)	⬆️
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sureshanaparti]

@Damans227 Check if it's possible to remove all snapshot policies of expunged / non-existing volumes during the upgrade (as one-time task as these are not applicable any more, and not relevant to show them in the UI). I think, it's better to remove all the snapshot policies associated with the volume during expunge volume itself, and refer them with findByIdIncluding Removed() wherever the snapshot policy is referred for the existing snapshots for the expunged volumes.

[Damans227]

@Damans227 Check if it's possible to remove all snapshot policies of expunged / non-existing volumes during the upgrade (as one-time task as these are not applicable any more, and not relevant to show them in the UI). I think, it's better to remove all the snapshot policies associated with the volume during expunge volume itself, and refer them with findByIdIncluding Removed() wherever the snapshot policy is referred for the existing snapshots for the expunged volumes.

@sureshanaparti So, the policies are already deleted during volume destroy - VolumeServiceImpl.destroyVolume() calls snapshotMgr.deletePoliciesForVolume().

The issue seems to me is when an orphaned policy exists due to an edge case (eg: failures, etc.). To reproduce the issue, I had to manually insert a record in DB to simulate an orphan policy. This fix allows handling those orphans created via UI/ API. Example:

An upgrade script to clean existing orphans is a good idea. Lemme know if I shall further investigate that?

[sureshanaparti]

@Damans227 destroy volume can be recovered, so better delete these policies while expunging the volume. also, check if you can keep the delete policies and remove volume in a transaction on destroy / expunge volume.

check if such orphans can be cleaned during ms start (maybe here -

cloudstack/framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java

Line 1210 in 036489b

private boolean cleanupVolume(final long volumeId) {

or other volume/snapshot service classes)

[Damans227]

@Damans227 destroy volume can be recovered, so better delete these policies while expunging the volume. also, check if you can keep the delete policies and remove volume in a transaction on destroy / expunge volume.

check if such orphans can be cleaned during ms start (maybe here -

cloudstack/framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java

Line 1210 in 036489b

private boolean cleanupVolume(final long volumeId) {

or other volume/snapshot service classes)

@sureshanaparti I see what you mean. I'll move policy deletion from destroy to expunge and add orphan cleanup on MS startup.

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16472

[blueorangutan]

[SF] Trillian Build Failed (tid-15241)

[blueorangutan]

[SF] Trillian Build Failed (tid-15246)

[Damans227]

[SF] Trillian Build Failed (tid-15246)

But env built successfully. That's odd. Jenkins needs investigation.

[Damans227]

[SF] Trillian Build Failed (tid-15246)

But env built successfully. That's odd. Jenkins needs investigation.

@sureshanaparti I tested the changes - created a VM with snapshot policy, destroyed it, then expunged it. Policy was deleted on expunge. The Jenkins env is available if you want to review/test the changes. The blueorangutan message above is misleading, so please ignore that.

[Pearl1594]

Tested the logic for deleting snapshot policy of an expunged volume - did so by

• Creating a snapshot policy for a root disk of a vm
• Expunged the VM
• Re-added the snapshot policy -to simulate orphaned policy
• attempted deletion - successfully deleted.

[Pearl1594]

code LGTM. Verified fix.

[DaanHoogland]

clgtm

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.