Performance issues since v12.4

[phil-or]

Since version v12.4 (and also in v12.5+), check_nwc_health causes significantly higher load on the monitoring server (rhel9) when running the following modes on Cisco devices:
--mode cpu-load
--mode memory
--mode hardware-health

Before v12.4 (e.g. in v12.0–v12.3), the same checks ran quickly and with normal resource usage.
After updating to v12.4 load on the Icinga host increased noticeably. Also v12.9 caused very high load.

Version v12.0 does not show this problem, which indicates the regression occurred between v12.0 → v12.4.

The graphs below show the performance issues. The blue line represents the time of the update from v12.0 to v12.4.

[c-kr]

You could have a look at the cisco related changes between 12.0 and 12.4, revert them and check if the cpu load problem disappears:

12.0.2...12.4

Another approach would be to git bisect between these versions to find the problematic commit.

--- edit ---

I did some quick research and can confirm that my customer is also affected:

time check_nwc_health_old --hostname xxx --community xxx --mode hardware-health
OK - environmental hardware working fine | 'temp_1011'=34;56;;; 'temp_1012'=40;125;;; 'temp_1013'=56;125;;;

real    0m0.349s
user    0m0.288s
sys     0m0.047s

time check_nwc_health_new --hostname xxx --community xxx --mode hardware-health
OK - environmental hardware working fine | 'temp_1011'=34;56;;; 'temp_1012'=40;125;;; 'temp_1013'=56;125;;;

real    0m0.905s
user    0m0.772s
sys     0m0.108s

Both user and sys is doubled on average. I will do some more investigation and share my results here.

[c-kr]

So I did some more research and found 3 commits which add noticeable cpu time on average compared to the parent. This is somehow expected as functionality and complexity is added over time:

Commit	real	user	sys
(My-Customer-Prod) 9.0.1.1	0,35	0,29	0,05
(between)*	0,5	0,41	0,05
commit a3df73f23da545f92d5758160850e9293afaa33e	0,65	0,53	0,08
commit 9c548365350e9f28fdb4f645d7959c38ee5502af (tag: 12.8)	0,7	0,58	0,08
commit fcb86cdc835c8084a74e7137023f7a8718c3456f (tag: 12.10)	0,77	0,63	0,1

• I could not find a single commit for this additional runtime. I think we can assume that every commit in between here adds features / complexity (and cpu time).