I have a problem with Splunk App for Unix and Linux: it gets CPU data from forwarder and indexes it, but it doesn't show it in the dashboard. I know that there already is a similar question (this one: https://answers.splunk.com/answers/618426/splunk-app-for-unix-and-linux-why-is-the-app-not-g.html ), but I don't know if my question is a duplicate: that other question doesn't provide a lot of details.
I have Splunk Universal Forwarder installed on one machine and Splunk Enterprise installed on another machine. On forwarder I installed Splunk Add-on for Unix and Linux. On Splunk Enterprise I installed Splunk App for Unix and Linux. I configured the forwarder to send data to Splunk Enterprise. I verified that Splunk Enterprise receives CPU related data from the forwarder by doing search "source=cpu".
In the search result I see that Splunk Enterprise receives this data, but two things are suspicious for me:
- I see that events contain multiple lines: I would rather expect each line to be an event on its own
- I see that events do not contain fields I would expect them to have, like "CPU", "pctUser", "pctNice", "pctSystem" etc
Then I went to "splunk app for Unix" -> "hosts". There I see a message:
CPU: unknown - is cpu.sh enabled?
In order to investigate the problem I clicked "CPU: No results found. Inspect ...". This is what I saw:
It seems that the problem is caused by the fact that this search does not return data. In order to investigate it I try to find out what part of the search causes the problem. I found that this search returns results:
but this does not:
index=main sourcetype=cpu CPU="all"
Well, so the problem is that in the sourcetype called "cpu" we don't have a field called "CPU". So it seems that the source of the problem is the fact that when Splunk Enterprise receives CPU data from the forwarder, it does not extract - as I noticed in the beginning - interesting fields from it.
But why? I think "Splunk App for Unix and Linux" should take care of it?
You need to install Splunk Add-on for Unix and Linux on Search Head and Indexers (This does not require Inputs enabled if you do not want to monitor those stats on Search Head and Indexers), after that it will start breaking all required fields for dashboards in Splunk App for Unix and Linux