topic Re: Splunk Metrics via Splunk Add on for *nix in Deployment Architecture

Splunk Metrics via Splunk Add on for *nix

Tohrment — Fri, 26 Oct 2018 12:24:41 GMT

So, we aren't in an environment where we can just deploy apps (a la collectd) but want to get metrics in from Linux boxes. Has anyone figured out how to accomplish this with the Splunk Add-on for *nix? I am terrible with regex(trying to change that) so I have not been able to figure out the proper transforms to get the data in a format fit for metrics index ingestion but needing to get it into our lab for future presentations on the matter(in the hopes of getting a license bump lol). Anyway, any help would be most appreciated!

Re: Splunk Metrics via Splunk Add on for *nix

fferozbasha — Fri, 09 Nov 2018 04:40:44 GMT

Hi, I have recently worked on a wrapper add-on to work on top of the Splunk NIX Add-On to tap in the output and convert them in to a metric event and transport it to indexers (either via Splunk TCP in csv file format, if the forwarder is of version more than 7 OR send via HTTP Event collector, if the forwarder is version less than 7). I am still trying to do some performance test around that before I can package the same as an add-on and publish it.

Re: Splunk Metrics via Splunk Add on for *nix

Tohrment — Fri, 09 Nov 2018 12:52:16 GMT

That would be amazing my friend. If you need help testing let me know. We have a small lab specifically for testing new versions of splunk and add ons. Keep us posted if you would be so kind. Thanks!

Re: Splunk Metrics via Splunk Add on for *nix

gjanders — Fri, 09 Nov 2018 20:55:49 GMT

Thanks for replying, FYI 7.2 does have the logs to metrics conversion but what Ferroz is describing covers older versions!

Re: Splunk Metrics via Splunk Add on for *nix

Tohrment — Fri, 09 Nov 2018 21:01:19 GMT

I have not found anything regarding using logs to metrics to utilize the output of the Splunk Add-on for *nix. If you have a link specifically for that and not the generalized article for Logs2Metrics I would more than be happy to look over it and attempt it.

Re: Splunk Metrics via Splunk Add on for *nix

gjanders — Fri, 09 Nov 2018 21:07:17 GMT

Nothing except the generalised article which is why this was just a comment and not an answer 🙂

Re: Splunk Metrics via Splunk Add on for *nix

fferozbasha — Sat, 10 Nov 2018 05:53:07 GMT

Thanks much Tohrment. I will surely share the add-on with you for further testing. I will publish the same in splunkbase and share you the link.

Re: Splunk Metrics via Splunk Add on for *nix

jherring_splunk — Mon, 19 Aug 2019 20:41:27 GMT

Did you ever manage to get this setup working? I have a need to do a logs-to-metrics for *nix TA output and any shortcuts would be appreciated.

Re: Splunk Metrics via Splunk Add on for *nix

satyenshah — Wed, 11 Dec 2019 17:15:42 GMT

One frustration with the Linux TA (v6.0.2) is that the output contains so much whitespace. Ingesting that data as a metric spares your license, but you still lose on storage and speed.

For example, I enabled vmstat.sh, bandwidth.sh, df.sh, and cpu.sh on one host. Those four inputs generate 64,552 bytes / hour of raw events. If I dedup the whitespace (replace \s+ with \s) then that shrinks to 36,465 bytes, which is 44% reduction.

In other words, metrics from the Linux TA are 51% whitespace. Anybody try to fix this? The whitespace originates from the awk/printf commands in the bash scripts. The commands format the output into pretty printed tables, which doesn't make sense for machine data.

Re: Splunk Metrics via Splunk Add on for *nix

lukeh — Thu, 30 Jan 2020 10:33:20 GMT

Hi Tohrment,

I have recently released a new add-on to collect Linux metrics without collectd 🙂

Metrics Add-on for Infrastructure:
https://splunkbase.splunk.com/app/4856/

All feedback is welcome 🙂

Re: Splunk Metrics via Splunk Add on for *nix

Arijit1 — Fri, 03 Jul 2020 17:57:36 GMT

Hello Lukeh,

Thanks for solution Metrics Add-on for Infrastructure. it is working perfectly for linux.

Now I am struggling with HPUX ,solaris and AIX.

It would be great if you have solution on those OS as well.

Thanks in Advance

Arijit Chowdhury