timechart and trendline command

[VijaySrrie]

Hi All,

Need help with Timechart and trendline command for below query
Both timechart and trendline command are not working

index=_introspection sourcetype=splunk_resource_usage component=Hostwide
| eval total_cpu_usage=('data.cpu_system_pct' + 'data.cpu_user_pct')
| stats Perc90(total_cpu_usage) AS cpu_usage latest(_time) as _time by Env Tenant
| timechart span=12h values(cpu_usage) as CPU
| trendline sma2(CPU) AS trend

[VijaySrrie · Accepted solution]

index=_introspection sourcetype=splunk_resource_usage host IN ("hostname" ) component=Hostwide
| eval total_cpu_usage=('data.cpu_system_pct' + 'data.cpu_user_pct')
| eval Tenant=case(match(host,"name"),"Core",match(host,"name"),"Enterprise Security",match(host,"name"),"Critical Reports",match(host,"hostname"),"Mgmt",match(host,"hostname"),"IDX",match(host,"hostname"),"AWE",match(host,"hostname"),"ABC",1==1,host)
| eval Env=case(match(host,"hostname"),"Prod",match(host,"hostname"),"E2E",match(host,"hostname"),"ABC",1==1,splunk_server)
| fields host_zone Tenant _time total_cpu_usage
| table host_zone Tenant _time total_cpu_usage | search host_zone="pr" Tenant="Core"
| bin span=24h aligntime=@d
_time
| stats Perc90(total_cpu_usage) AS cpu_usage BY _time
| trendline sma2(cpu_usage) AS trend | fields * trend

[VijaySrrie]

I want to know how much CPU is utilized in our environment along with the trendline @ITWhisperer 

[ITWhisperer]

Please share some representative anonymised sample events in a code block

How often do you want to sample the cpu used?

Are Env and Tenant already extracted?

Do you want the stat broken down by Env and Tenant as well as time or some other dimensions?

[VijaySrrie]

@ITWhisperer Yes, env and tenant are already extracted, yes, we need stat broken by env and tenant as well as time

{"datetime":"08-19-2024 10:40:30.196 +1000","log_level":"INFO","component":"Hostwide","data":{"cpu_arch":"x86_64","os_name":"Linux","os_name_ext":"Linux","os_build":"#1 ABC Thu Apr 4 03:33:23 EDT 2024","os_version":"3.10.0-1160.118","instance_guid":"ABCDEFGH","splunk_version":"9.2.1","mem":"382641.051","mem_used":"41983.578","swap":"511.996","swap_used":"511.996","pg_paged_out":"50842005897","pg_swapped_out":"164124","forks":"00000600","cpu_count":"24","virtual_cpu_count":"48","runnable_process_count":"19","normalized_load_avg_1min":"1.14","cpu_user_pct":"45.35","cpu_system_pct":"10.68","cpu_idle_pct":"43.98"}}

[ITWhisperer]

You can't timechart by more than 2 dimensions and _time is one of those, try combining Env and Tenant

index=_introspection sourcetype=splunk_resource_usage component=Hostwide
| eval total_cpu_usage=('data.cpu_system_pct' + 'data.cpu_user_pct')
| eval EnvTenant=Env.":".Tenant
| timechart Perc90(total_cpu_usage) AS cpu_usage span=12h useother=f by EnvTenant

[ITWhisperer]

What is it you are trying to achieve?

At the moment, you are getting one stats result for each Env Tenant combination with the latest time stamp for that Env Tenant. This doesn't sound like something useful to timechart or trend.