Getting Data In

Is there a way to break lines in props.conf on search head?

wkupersa
Path Finder

I have the ability to configure a search head but not the indexers. I am wondering if I can break multi-line netstat events into multiple events on the search head using props.conf. I realize that I can use multikv to break things during the search, but I also need field extraction happening after line breaking and do it in the search creates really long searches.

0 Karma

arkadyz1
Builder

How are you using your data? Are the results of those long searches going to be used in dashboards?

If yes, you are better off not extracting those fields during search time, but pre-fetching them into a summary index with a scheduled search and then building the dashboards off that summary index instead.

In addition to being much more efficient overall (you extract the new fields just once and reuse them as many times as necessary), this way does not add to your license usage - you are paying for the original data, but not for the transformed by "collect" or summary searches. The downside will be your inability to run the dashboards in real time - the last data point will be from the most recent run of that scheduled summary search.

wkupersa
Path Finder

An example of how I would use this netstat data would be to look for a new LISTENING port open on a host..... so I might do something like (in rough psuedocode) search earliest=-2w latest=now netstat.data| stats first(event) by port,host | where first > -1d

Thank you for your suggestion. That was my initial approach to the problem. Then when an admin installed the Splunk_TA_nix and splunk_app_for_nix, other events like ps started getting broken up, so I figured that I could reverse engineer what it was doing and apply it to netstat. No luck.

The only concern with summary indexes is the additional space being used.

0 Karma

arkadyz1
Builder

If your concern is the additional space needed for summary indexes, you can cap the original, uprocessed data at a very low size and have the bulk of the data in the summary. We came to this eventually. We even have two types of dashboards - "static" and "live". The "live" ones offer a much simpler data processing, work off the raw data and have the real-time default timers - rt-5m to rt, for example. The "static" dashboards feed off the summary data, run historic searches and have more bells and whistles - at the expense of not being able to track data as they come in.

Your method of usage seems to be tolerant to the fact that the data might not be available immediately but will be collected by a summary search at some near point in the future - unless the user of the dashboard wants to see the new listening ports as soon as they are opened.

somesoni2
Revered Legend

If longer searches due to field extractions is your problem, create search macros to for each rex command that you have.

0 Karma

wkupersa
Path Finder

Update to my own post: I am able to get close by using kv_mode=multi in my props.conf. However I find that although this works for 99.8% of the hosts, there are some hosts that seem to not get parsed by multikv.

I am able to duplicate this by removing the kv_mode line from props.conf and running the netstat output manually through multikv. It works for almost all the hosts but only works for 100% if I run it with "noheader=true"

It's driving me nuts because the output of the netstat that parses and the netstat that doesn't parse looks exactly the same. I am wishing that I could get a hex dump of _raw so I could look for differences, but I don't know of a way to do so.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi wkupersa,

No, this is not possible. Talk to the guys that can configure the indexers and let them do the changes for you on the indexer.

If this is not possible for what ever reason, create another scripted input that returns events in the format you need them and run this scripted input on the search head.

Hope this helps ...

cheers, MuS

Get Updates on the Splunk Community!

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...