I recreated the index, the %_Processor_Time table basically shows 100s and then bunch of 0s. And nothing in between, for now, I hope. I'll come back to it tomorrow maybe there will be some diversity. 0 1,718 81.229% 100 282 13.333% 0.15612418354205651 4 0.189% 0.15594301259076812 3 0.142% 0.15594309209218249 3 0.142% 0.15600581901974953 3 0.142% 0.15607107435167317 3 0.142% 0.15613357457951121 3 0.142% 0.1561871657303206 3 0.142% 0.16447270967157646 ... View more

Hi, I guess since I'm querying anything >90 I would get only the 100%s but what I am worried about, am I really getting the processes that runs for at least 15 seconds, or am I getting everything that at one point peaked at > 90 %. All I see is chrome at 100%, pages and pages of it. It might be true but I only use chrome on the server if I need to download something and for at least a few days I haven't even opened chrome there. My question though, does the polling interval matter? When Splunk UF forwards the data does it only forward data at the time of polling or the whole lot? Say interval is 30s, does the forwarded data contain all the data since last poll 29.xx seconds ago or only data at the time of the poll? Raw: 7/14/16 2:57:45.000 PM InetMgr 0 3504 23699456 %_Processor_Time = 0 ID_Process = 3504 Working_Set_-_Private = 23699456 host = mehost index = perfmon instance = InetMgr object = Process source = PerfmonMk:Process sourcetype = PerfmonMk:Process 7/14/16 2:57:45.000 PM LogonUI 0 748 7745536 %_Processor_Time = 0 ID_Process = 748 Working_Set_-_Private = 7745536 host = mehost index = perfmon instance = LogonUI object = Process source = PerfmonMk:Process sourcetype = PerfmonMk:Process 7/14/16 2:57:45.000 PM PRTG_Probe 0.15595768067475685 4104 21671936 %_Processor_Time = 0.15595768067475685 ID_Process = 4104 Working_Set_-_Private = 21671936 host = mehost index = perfmon instance = PRTG_Probe object = Process source = PerfmonMk:Process sourcetype = PerfmonMk:Process 7/14/16 2:57:45.000 PM Ssms 0 676 76500992 %_Processor_Time = 0 ID_Process = 676 Working_Set_-_Private = 76500992 host = mehost index = perfmon instance = Ssms object = Process source = PerfmonMk:Process sourcetype = PerfmonMk:Process 7/14/16 2:57:45.000 PM System 0.051985893558252283 4 73728 %_Processor_Time = 0.051985893558252283 ID_Process = 4 Working_Set_-_Private = 73728 host = mehost index = perfmon instance = System object = Process source = PerfmonMk:Process sourcetype = PerfmonMk:Process 7/14/16 2:57:45.000 PM chrome 100 56 68251648 %_Processor_Time = 100 ID_Process = 56 Working_Set_-_Private = 68251648 host = mehost index = perfmon instance = chrome object = Process source = PerfmonMk:Process sourcetype = PerfmonMk:Process 7/14/16 2:57:45.000 PM chrome#1 0 2256 937984 %_Processor_Time = 0 ID_Process = 2256 Working_Set_-_Private = 937984 host = mehost index = perfmon instance = chrome#1 object = Process source = PerfmonMk:Process sourcetype = PerfmonMk:Process 7/14/16 2:57:45.000 PM chrome#2 100 2196 128057344 %_Processor_Time = 100 ID_Process = 2196 Working_Set_-_Private = 128057344 host = mehost index = perfmon instance = chrome#2 object = Process source = PerfmonMk:Process sourcetype = PerfmonMk:Process 7/14/16 2:57:45.000 PM cmd 0 1668 610304 %_Processor_Time = 0 ID_Process = 1668 Working_Set_-_Private = 610304 host = mehost index = perfmon instance = cmd object = Process source = PerfmonMk:Process sourcetype = PerfmonMk:Process ... View more

Hi, You know why my avgCPU always shows 100%, that can't be right I think... index=perfmon sourcetype="PerfmonMk:Process" instance!="_Total" AND instance!="Idle" AND instance!="wmi*" | bucket _time span=15s | stats avg(%_Processor_Time) AS "avgCPU" avg(Working_Set_-_Private) AS "AvgMemory" BY host _time instance | where avgCPU > 90 | DEDUP _time In your query I'm not sure what "process" is so I changed it to instance (which are the names of the processes). Is this query giving me what I actually want? Or am I mangling data somewhere... Thanks! ... View more

Hi, thanks for replying. I can't seem to use this query. Is "cpu_util" a counter? is it supposed to be %_Processor_Time? Either wouldn't give me any result. Could you please explain what this does? stats avg(cpu_util) AS average_cpu_util by host _time process The query does not yield any result from this part on, which looks like the most important part. What counters should I have for this to work? Mine is basically like the ones on this page: [Link]http://blogs.splunk.com/2013/12/09/monitor-processes-per-user-on-microsoft-remote-desktop-services-session-host[1]. Thanks! ... View more

Thanks. Any way to point me to right query samples to do the task 1, 2, 3. host="AQS-INTEGRATION" sourcetype="Perfmon:CPU Load" counter="% Processor Time" earliest="-7d@d"| bucket _time span=15s | stats avg(Value) as avg_CPU by _time | where avg_CPU >= 90 | SORT BY _time DESC Does this query return the first part of 1)? I'm sort of on the fence with the bucket keyword. Do you know how to start forwarding Process data? In wmi.conf there's a sample stanza called "LocalProcess" and I put Disabled to 0, but I don't see any new Source/type, I even created the index that that stanza is referring to ("perfmon"). Do you know why I am not getting Windows Event Logs? Inputs.conf I have them Disabled=0, I created an Index called wineventslog (because Splunk was complaining it was missing), and still i don't see any new source/sourcetypes that should belong to windows event logs. For both parts I restarted both Splunk and the UFs. Splunk seems to have automatically parsed the dates on my log files, that's good. Going back can you give me a sample of multiple query searches in one result view? I want to see how difficult it is to understand. Thanks! ... View more

That looks about right but I can't seem to make it work. On each servers's universal forwarder I added this to \etc\system\local\wmi.conf: [WMI:SessionProcess] interval = 10 disabled = 0 index = perfmon_index wql = Select ProcessId, SessionId From Win32_Process And to \etc\system\local\inputs conf [perfmon://Process] interval = 10 object = Process counters = % Processor Time; ID Process; Working Set - Private; IO Read Operations/sec; IO Write Operations/sec instances = * index= perfmon_index disabled = 0 mode = multikv And restarted the UniversalForwarder service. It doesn't seem to forward any processes. Also in the samples inside the default wmi.conf and index.conf, the index is usually "perfmon" and not "perfmon_index". Which one should it be? Any ideas? Thanks! ... View more

Hi, I have this: : sourcetype="Perfmon:CPU Load" counter="% Processor Time" earliest="-7d@d"| bucket _time span=20s | stats avg(Value) as avgCPU by _time | where avgCPU >80 Seems to work, had to lower the avgCPU to test. Reading your link though I have yet to find a way to include the top processes whenever the CPU goes above the threshold. I don't think perfmon keeps that data. I was hoping the Universal Forwarder, seeing it had the option to monitor CPU on setup (I suppose using Perfmon...), maybe has the ability to run scripts on a threshold and append the output of the scripts into the events sent to Splunk. I mean it is a lot of documentation, and I was admittedly skimming based on the names of those commands . Did I miss anything? Can I get some direction. Thanks! ... View more