Splunk Metrics via Splunk Add on for *nix

[Tohrment]

So, we aren't in an environment where we can just deploy apps (a la collectd) but want to get metrics in from Linux boxes. Has anyone figured out how to accomplish this with the Splunk Add-on for *nix? I am terrible with regex(trying to change that) so I have not been able to figure out the proper transforms to get the data in a format fit for metrics index ingestion but needing to get it into our lab for future presentations on the matter(in the hopes of getting a license bump lol). Anyway, any help would be most appreciated!

[lukeh · Accepted solution]

Hi Tohrment,

I have recently released a new add-on to collect Linux metrics without collectd 🙂

Metrics Add-on for Infrastructure:
https://splunkbase.splunk.com/app/4856/

All feedback is welcome 🙂

L.

[Arijit1]

Hello  Lukeh,

Thanks  for solution Metrics Add-on for Infrastructure.  it is  working  perfectly  for linux.  

Now I am struggling  with HPUX  ,solaris and AIX.  

It would be great  if you have solution on those OS  as  well.  

Thanks  in Advance 

Arijit Chowdhury 

[satyenshah]

One frustration with the Linux TA (v6.0.2) is that the output contains so much whitespace. Ingesting that data as a metric spares your license, but you still lose on storage and speed.

For example, I enabled vmstat.sh, bandwidth.sh, df.sh, and cpu.sh on one host. Those four inputs generate 64,552 bytes / hour of raw events. If I dedup the whitespace (replace \s+ with \s) then that shrinks to 36,465 bytes, which is 44% reduction.

In other words, metrics from the Linux TA are 51% whitespace. Anybody try to fix this? The whitespace originates from the awk/printf commands in the bash scripts. The commands format the output into pretty printed tables, which doesn't make sense for machine data.

[fferozbasha]

Hi, I have recently worked on a wrapper add-on to work on top of the Splunk NIX Add-On to tap in the output and convert them in to a metric event and transport it to indexers (either via Splunk TCP in csv file format, if the forwarder is of version more than 7 OR send via HTTP Event collector, if the forwarder is version less than 7). I am still trying to do some performance test around that before I can package the same as an add-on and publish it.

[Tohrment]

That would be amazing my friend. If you need help testing let me know. We have a small lab specifically for testing new versions of splunk and add ons. Keep us posted if you would be so kind. Thanks!

[fferozbasha]

Thanks much Tohrment. I will surely share the add-on with you for further testing. I will publish the same in splunkbase and share you the link.

[jherring_splunk]

Did you ever manage to get this setup working? I have a need to do a logs-to-metrics for *nix TA output and any shortcuts would be appreciated.

[gjanders]

Thanks for replying, FYI 7.2 does have the logs to metrics conversion but what Ferroz is describing covers older versions!

[Tohrment]

I have not found anything regarding using logs to metrics to utilize the output of the Splunk Add-on for *nix. If you have a link specifically for that and not the generalized article for Logs2Metrics I would more than be happy to look over it and attempt it.

[gjanders]

Nothing except the generalised article which is why this was just a comment and not an answer 🙂