[Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM by nvazquez · Pull Request #10770 · apache/cloudstack

nvazquez · 2025-04-24T09:55:06Z

Description

This PR improves the listing of Vmware VMs for KVM migration tool to reduce the timeout observed while listing VMs on a datacenter. The improvement is performed in two stages:

First stage: Datacenter VMs listing -> use thin listing of VMs instead of obtaining all its disks and NICs (Invoke the listVmwareDcVms API setting the datacentername parameter only)
Second stage: When selecting a VM to import -> perform the full information retrieval of the selected VM including its disks and NICs information (Invoke the listVmwareDcVms API setting the datacentername, hostname and virtualmachinename parameter)

It was originally added by PR: #9925 but reverted as it caused unintended regression affecting the VM console access (#10634).

Fixes: #9782

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested against Vmware vCenter datacenter with 200+ VMs, listing times reduced significantly from 30 seconds to <1 second for the entire datacenter VMs

Same command time difference against the same vCenter: (time cmk list vmwaredcvms vcenter=XXXX datacentername=XXXX username=XXXX password=XXXX page=1 pagesize=10 filter=name)

Before:

real	0m35.755s
user	0m0.087s
sys	0m0.016s

After:

real	0m0.468s
user	0m0.076s
sys	0m0.013s

nvazquez · 2025-04-24T09:56:30Z

@blueorangutan package

blueorangutan · 2025-04-24T09:58:04Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-04-24T10:03:28Z

Codecov Report

Attention: Patch coverage is 4.14201% with 162 lines in your changes missing coverage. Please review.

Project coverage is 15.17%. Comparing base (55c8138) to head (76604e9).
Report is 22 commits behind head on 4.19.

Files with missing lines	Patch %	Lines
...in/java/com/cloud/hypervisor/vmware/mo/BaseMO.java	1.40%	70 Missing ⚠️
...d/hypervisor/vmware/manager/VmwareManagerImpl.java	0.00%	51 Missing ⚠️
.../org/apache/cloudstack/vm/UnmanagedInstanceTO.java	0.00%	12 Missing ⚠️
...ack/api/command/admin/zone/ListVmwareDcVmsCmd.java	0.00%	9 Missing ⚠️
...com/cloud/hypervisor/vmware/util/VmwareHelper.java	0.00%	9 Missing ⚠️
...dstack/api/response/UnmanagedInstanceResponse.java	33.33%	8 Missing ⚠️
...a/com/cloud/hypervisor/vmware/mo/DatacenterMO.java	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               4.19   #10770    +/-   ##
==========================================
  Coverage     15.16%   15.17%            
- Complexity    11332    11344    +12     
==========================================
  Files          5412     5415     +3     
  Lines        475033   475564   +531     
  Branches      57963    58033    +70     
==========================================
+ Hits          72048    72146    +98     
- Misses       394930   395353   +423     
- Partials       8055     8065    +10

Flag	Coverage Δ
uitests	`4.28% <ø> (-0.02%)`	⬇️
unittests	`15.89% <4.14%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

DaanHoogland

clgtm

blueorangutan · 2025-04-24T11:19:05Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13166

nvazquez · 2025-04-24T11:48:03Z

@blueorangutan test ol8 vmware-70u3

blueorangutan · 2025-04-24T11:50:03Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

.../vmware/src/main/java/org/apache/cloudstack/api/command/admin/zone/ListVmwareDcHostsCmd.java

blueorangutan · 2025-04-25T10:37:55Z

[SF] Trillian test result (tid-13127)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 79615 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10770-t13127-vmware-70u3.zip
Smoke tests completed. 127 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_role_account_acls	`Failure`	3.97	test_dynamicroles.py
test_role_account_acls_multiple_mgmt_servers	`Failure`	3.33	test_dynamicroles.py
test_01_events_resource	`Error`	355.18	test_events_resource.py
test_04_deploy_vm_for_other_user_and_test_vm_operations	`Error`	122.44	test_network_permissions.py
test_01_restore_vm	`Error`	3607.45	test_restore_vm.py
test_02_restore_vm_allocated_root	`Error`	10.30	test_restore_vm.py
test_12_destroy_cpvm	`Error`	6.36	test_ssvm.py
test_01_deploy_vm_on_specific_host	`Error`	19.06	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	3607.06	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	2.57	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	4.56	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	15.80	test_vm_deployment_planner.py

nvazquez · 2025-05-12T20:58:58Z

@blueorangutan package

blueorangutan · 2025-05-12T21:00:03Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-12T22:06:39Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13352

nvazquez · 2025-05-12T23:02:58Z

@blueorangutan test ol8 vmware-70u3

blueorangutan · 2025-05-12T23:04:03Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

nvazquez · 2025-05-13T10:08:20Z

@blueorangutan package

blueorangutan · 2025-05-13T10:10:04Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-13T11:06:35Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13363

DaanHoogland · 2025-05-13T11:39:16Z

@blueorangutan test ol8 vmware-70u3

blueorangutan · 2025-05-13T11:40:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

nvazquez · 2025-05-13T12:34:47Z

@blueorangutan package

blueorangutan · 2025-05-13T12:36:04Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-13T13:39:01Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13366

NuxRo

tested latest changes, lgtm

blueorangutan · 2025-05-13T20:53:26Z

[SF] Trillian test result (tid-13279)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 76286 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10770-t13279-vmware-70u3.zip
Smoke tests completed. 130 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestClusterDRS>:setup	`Error`	0.00	test_cluster_drs.py
test_01_restore_vm	`Error`	3604.61	test_restore_vm.py
test_02_restore_vm_allocated_root	`Error`	23.27	test_restore_vm.py
ContextSuite context=TestRestoreVM>:teardown	`Error`	31.73	test_restore_vm.py
test_01_deploy_vm_on_specific_host	`Error`	22.12	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	3603.82	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	4.58	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	22.20	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	3603.94	test_vm_deployment_planner.py

nvazquez · 2025-05-13T23:02:49Z

@blueorangutan test ol8 vmware-70u3

blueorangutan · 2025-05-13T23:04:04Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

blueorangutan · 2025-05-14T02:46:57Z

[SF] Trillian test result (tid-13288)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52017 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10770-t13288-vmware-70u3.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2025-05-14T20:47:05Z

[SF] Trillian test result (tid-13298)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 75725 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10770-t13298-vmware-70u3.zip
Smoke tests completed. 130 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_restore_vm	`Error`	3607.98	test_restore_vm.py
test_02_restore_vm_allocated_root	`Error`	8.99	test_restore_vm.py
test_01_deploy_vm_on_specific_host	`Error`	20.05	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	3603.42	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	2.47	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	2.52	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	4.71	test_vm_deployment_planner.py
test_09_expunge_vm	`Failure`	424.82	test_vm_life_cycle.py

apache#10770) Co-authored-by: dahn <daan.hoogland@gmail.com> Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

nvazquez added 3 commits April 24, 2025 06:49

Improve Vmware VMs retrieval

b770d27

Fix imports

8cac953

Fixes and improvements

16eab4d

nvazquez added type:bug component:vmware component:UI labels Apr 24, 2025

nvazquez added this to the 4.19.3 milestone Apr 24, 2025

nvazquez requested review from DaanHoogland, Pearl1594, borisstoyanov, harikrishna-patnala, kiranchavala, sureshanaparti, vladimirpetrov, weizhouapache and yadvr April 24, 2025 09:55

nvazquez linked an issue Apr 24, 2025 that may be closed by this pull request

[UI] Timeout listing Vmware Datacenter VMs for migration to KVM #9782

Closed

nvazquez mentioned this pull request Apr 24, 2025

[UI] Timeout listing Vmware Datacenter VMs for migration to KVM #9782

Closed

DaanHoogland approved these changes Apr 24, 2025

View reviewed changes

DaanHoogland added the status:needs-testing label Apr 24, 2025

Pearl1594 added this to ACS 4.20.1 Apr 24, 2025

Pearl1594 moved this to In Progress in ACS 4.20.1 Apr 24, 2025

sureshanaparti reviewed Apr 25, 2025

View reviewed changes

.../vmware/src/main/java/org/apache/cloudstack/api/command/admin/zone/ListVmwareDcHostsCmd.java Outdated Show resolved Hide resolved

DaanHoogland closed this May 13, 2025

github-project-automation bot moved this from In Progress to Done in ACS 4.20.1 May 13, 2025

DaanHoogland reopened this May 13, 2025

Restore hostname parameter

81a9771

Include boot mode and boot type on individual VM response

76604e9

DaanHoogland closed this May 13, 2025

DaanHoogland reopened this May 13, 2025

NuxRo approved these changes May 13, 2025

View reviewed changes

DaanHoogland removed the status:needs-testing label May 14, 2025

DaanHoogland merged commit 8d3ae3e into apache:4.19 May 14, 2025
45 of 49 checks passed

DaanHoogland deleted the 419-vmware-improve-afterrevert branch May 14, 2025 06:12

Conversation

nvazquez commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

nvazquez commented Apr 24, 2025

Uh oh!

blueorangutan commented Apr 24, 2025

Uh oh!

codecov bot commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Apr 24, 2025

Uh oh!

nvazquez commented Apr 24, 2025

Uh oh!

blueorangutan commented Apr 24, 2025

Uh oh!

Uh oh!

blueorangutan commented Apr 25, 2025

Uh oh!

nvazquez commented May 12, 2025

Uh oh!

blueorangutan commented May 12, 2025

Uh oh!

blueorangutan commented May 12, 2025

Uh oh!

nvazquez commented May 12, 2025

Uh oh!

blueorangutan commented May 12, 2025

Uh oh!

nvazquez commented May 13, 2025

Uh oh!

blueorangutan commented May 13, 2025

Uh oh!

blueorangutan commented May 13, 2025

Uh oh!

DaanHoogland commented May 13, 2025

Uh oh!

blueorangutan commented May 13, 2025

Uh oh!

nvazquez commented May 13, 2025

Uh oh!

blueorangutan commented May 13, 2025

Uh oh!

blueorangutan commented May 13, 2025

Uh oh!

NuxRo left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented May 13, 2025

Uh oh!

nvazquez commented May 13, 2025

Uh oh!

blueorangutan commented May 13, 2025

Uh oh!

blueorangutan commented May 14, 2025

Uh oh!

Uh oh!

blueorangutan commented May 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

nvazquez commented Apr 24, 2025 •

edited

Loading

codecov bot commented Apr 24, 2025 •

edited

Loading