vm cpu util in kvmtop unequal top inside vm

[MyungChi]

Hi.

I reinstall kvmtop latest release(2.1.3) in my system.
Total number of CPU is fixed.

But, There are issue usage of CPU usage of VM is not collect.
As you see below, VM name "vedge" is shown "52%" CPU usage.

So, I checked with top after login vedge VM. CPU usage is less than 1%.

kvmtop --cpu --mem

cpu                ram
cores    88        Total    19763962
curfreq  0.000000  Free     2562432
                   Availabl 11867460

                  cpu               ram
UUID     name     cores total steal total    used
256fc986 vedge    4     52    0     3880072  2278276
f62468f4 pike-osc 4     12    0     8173704  7493712

top in vedge vm

=====================================================

top - 02:54:01 up 14 days,  5:45,  2 users,  load average: 0.01, 0.03, 0.05
Tasks: 141 total,   2 running, 139 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us,  0.1 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.6 st
KiB Mem :  3880072 total,  1599800 free,  1643624 used,   636648 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1960016 avail Mem

[cha87de]

@MyungChi Can you please do some further tests to analyse this behavior?
On the host where kvmtop runs, please find the process ID of the VM (e.g. ps -ef | grep vedge) and then run top -p PROCESSID with the process id of the VM on the host. Are the "%CPU" of top and the "cpu total" of kvmtop equal?

[MyungChi]

@cha87de

I checked my server as you mentioned.
Pls refer below result.

kvmtop result

cpu                ram
cores    88        Total    19763962
curfreq  0.000000  Free     2867384
                   Availabl 11860298

                  cpu               ram
UUID     name     cores total steal total    used
256fc986 vedge    4     52    1     3880072  2283496

top in host

asks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.9 us,  2.3 sy,  0.0 ni, 96.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 19763961+total,  2907088 free, 77109040 used, 11762348+buff/cache
KiB Swap:  4194300 total,  2915824 free,  1278476 used. 11864074+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
80236 qemu      20   0 5235404 224424  13532 S 210.0  0.1  35231:59 qemu-kvm

[cha87de]

So you're seeing 210.0% CPU utilisation for this VM in top on the host, but no CPU utilisation inside the VM?

The 210% and 52% in top vs. kvmtop differences are probably because kvmtop computes the 100% relative to the amount of cores, hence I assume these two tools measure the same. Hence I'm not so sure yet that this is a bug of kvmtop.

[MyungChi]

My server has 88 core, It means 8,800%. 210% is just 2.3% of total cpu capability.

[cha87de]

What CPU architecture do you have? Can you name the Model of the CPU?

[MyungChi]

This is my server cpu info. server is "HPE DL380 G10"

processor : 87
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Gold 6152 CPU @ 2.10GHz
stepping : 4
microcode : 0x200004d
cpu MHz : 2100.000
cache size : 30976 KB
physical id : 1
siblings : 44
core id : 28
cpu cores : 22
apicid : 121
initial apicid : 121
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke spec_ctrl intel_stibp flush_l1d
bogomips : 4223.85
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

[MyungChi]

Hi.

As I know %CPU on top is core * usage in linux.
So, I checked top usage with with irix mode(shift + i), again. In irix mode, %CPU will show value for Max. 100%.

This is CPU usage when I type top -p $PID

top - 12:46:57 up 16 days, 20:25,  9 users,  load average: 2.79, 3.10, 3.24
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.9 us,  2.3 sy,  0.0 ni, 96.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 19763961+total,  2665308 free, 77067640 used, 11790667+buff/cache
KiB Swap:  4194300 total,  2955920 free,  1238380 used. 11839808+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
80236 qemu      20   0 5235404 224224  13532 S 222.3  0.1  40176:08 qemu-kvm

This is CPU usage after chage Irix module (shift + i).

Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.9 us,  2.3 sy,  0.0 ni, 96.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 19763961+total,  2683112 free, 77050064 used, 11790644+buff/cache
KiB Swap:  4194300 total,  2955920 free,  1238380 used. 11841642+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
80236 qemu      20   0 5243600 224224  13532 S  2.4  0.1  40178:34 qemu-kvm

[cha87de]

I measured the cpu utilisation inside a VM (Influx TICK Stack) and outside via kvmtop. When stressing the CPU with artificial load using the stress tool, the differences in measured cpu utilisation (4 repetitions) is as follows:

It is not exactly 0% diff, since the time for collecting cpu utilisation is not synchronized for the two collectors.

[MyungChi]

Hi.

Is there any update for this ?

[cha87de]

@MyungChi as stated in my previous comment, I cannot reproduce nor understand this issue. To fix it, please provide a more intense problem analysis and provide a sketch for a proper solution, which I can then integrate.

This is an open source project - so your support is as important as my own contribution.