Splunk Add-on for Unix and Linux coming up empty

[redc]

I'm in the process of setting up the Splunk App for Unix and Linux and the Splunk Add-on for Unix and Linux.

I've installed and configured the App via Splunk Web (which runs on a Windows box) using default settings. I've installed the Add-on on one of my Linux boxes and enabled all of the default inputs using default settings. I've got data flowing into the "os" index.

But...all of the App dashboards are coming up empty/"No results found."

Here's a screenshot of the Hosts dashboard, showing the information for the one Linux host I've configured:

Using the "Process Status" as an example (since it's easy to inspect), I get:
This search has completed and found 5 matching events. However, the transforming commands in the highlighted portion of the following search:

search index=os sourcetype=top host=my-host-name | stats max(pctCPU) as pctCPU max(pctMEM) as pctMEM last(cpuTIME) as cpuTIME by COMMAND, USER | eval CMD=COMMAND | fields CMD, USER, pctCPU, pctMEM, cpuTIME

generated no results.

If I run the search command portion (excluding the stats command and everything after it), I get events that look like this (screenshot #2); I assume this format normal:

Argh! So, what am I missing?

[araitz · Accepted solution]

Well I think you've found your own problem. Doesn't sound like the TA-nix is installed on your search head. Not sure how that can be? Here's the stanza from Splunk_TA_nix/default/props.conf:

 [top]
 SHOULD_LINEMERGE=false
 LINE_BREAKER=(^$|[\r\n]+[\r\n]+)
 TRUNCATE=1000000
 DATETIME_CONFIG = CURRENT
 KV_MODE=multi

[araitz]

See above comment - answers won't let me move it 🙂

[redc]

The bottom screenshot is the data that should be filling the "Process Status" portion of the first screenshot, NOT the cpu.sh, vmstat.sh, and df.sh portions in the "Specification" and "System Status" portions. Those three inputs are also sending data to the "os" index but getting the same "transforming commands" generated no results error as above.

Yes, I performed the actions in that document, up to the alerts portion (I'm not ready yet to start alerts flowing), including deleting the auto-created "all_hosts" and "default" category/group in order to configure my own.

[lguinn2]

I don't know what to tell you about the app in general, but I know what is wrong with this particular search!

index=os sourcetype=top host=my-host-name  
| multikv
| stats  max(pctCPU) as pctCPU max(pctMEM) as pctMEM last(cpuTIME) as cpuTIME by COMMAND, USER 
 | eval  CMD=COMMAND  
| fields  CMD, USER, pctCPU, pctMEM, cpuTIME

My only other suggestion is that you check the versions of the app and the add-on - there may be older and newer versions, and you should be sure to use versions of the two that work together...

There is a manual for the app at Splunk App for Linux and Unix

[redc]

Ah-ha. I'd installed it, but it was disabled. Enabling the Splunk_TA_nix on the search head solved it.

[redc]

Just for kicks, I tried this by running the search manually and it does generate output where before it doesn't. Sorry @araitz.

However, I find it odd that the app doesn't do this "out of the box" since I would expect there to always be multiple commands/users running on any given server (or at least, for that to be the case more often than not).

But is this the solution? Since I haven't seen what this dashboard should look like when it's working correctly, I'm not sure if this produces the correct results.

[araitz]

@lguinn - the nix TA runs KV_MODE=MULTI automatically, so running multikv explicitly is not required.

[araitz]

Did you follow the prompt from the home page to set up the app? Did you read through the docs on first time configuration?

http://docs.splunk.com/Documentation/UnixApp/latest/User/First-timeconfiguration