topic Re: Can Splunk do filtering based on the index name rather than source or sourcetype? in Getting Data In

Can Splunk do filtering based on the index name rather than source or sourcetype?

babcolee — Mon, 29 Feb 2016 16:31:38 GMT

We have a condition where we need to filter out data based on the byte count in the log. We have collapsed the source and sourcetype names coming from different servers and we need to be specific based on the index name.

Instead of:
props.conf

[source::///var/log/paloalto/palo.log]
TRANSFORMS-null = setnull

Can we use for the props.conf configuration:

[index::plvpalo]
TRANSFORMS-null = setnull

[source::///var/log/paloalto/palo.log]
index = plvpalo
TRANSFORMS-null = setnull

Re: Can Splunk do filtering based on the index name rather than source or sourcetype?

somesoni2 — Mon, 29 Feb 2016 16:48:27 GMT

I would say No, you can filter logs based on Index name in the way you can use source/sourcetype/host to filter logs, but you can filter out data for a source/sourcetype/host based on the index name. Try something like this

props.conf

[source::///var/log/paloalto/palo.log]
TRANSFORMS-null = setnull

transforms.conf

[setnull]
SOURCE_KEY = _MetaData:Index
REGEX = plvpalo
DEST_KEY = queue
FORMAT = nullQueue

Re: Can Splunk do filtering based on the index name rather than source or sourcetype?

babcolee — Mon, 29 Feb 2016 17:46:46 GMT

I am already using a REGEX statement in the transforms.conf file to filter out any less than 1400 bytes. Would the transforms.conf work as follows:

[setnull]
SOURCE_KEY = _MetaData:Index
REGEX = plvpalo
REGEX = ^(?:[^,]*?,){31}(\d{1,3}|1[0-3]\d{2}|1400),
DEST_KEY = queue
FORMAT = nullQueue

Re: Can Splunk do filtering based on the index name rather than source or sourcetype?

somesoni2 — Mon, 29 Feb 2016 18:04:45 GMT

The above one will not work as the byte size was been check from SOURCE_KEY=_raw (default) and index name will be checked from SOURCE_KEY = _MetaData:Index. You can create two transforms.conf stanzas and call them both.

props.conf

 [source::///var/log/paloalto/palo.log]
 TRANSFORMS-null = setnull,setnullindex

transforms.conf

[setnull]
..keep the current setting that you have...

 [setnullindex]
 SOURCE_KEY = _MetaData:Index
 REGEX = plvpalo
 DEST_KEY = queue
 FORMAT = nullQueue

Re: Can Splunk do filtering based on the index name rather than source or sourcetype?

babcolee — Mon, 29 Feb 2016 20:23:42 GMT

It is not performing as expected. Here is what we are trying to accomplish. The log file is a csv and we need to filter out all the events / data that is under 1400 bytes which is found in field 31.

Sample log:

2016/02/25 19:14:20,010401000240,TRAFFIC,start,1,2016/02/25 19:14:20,0.1.2.3,4.5.6.7,8.9.10.11,12.13.14.15,Outbound Services,,,dns,vsys1,TRUST,UNTRUST,ethernet1/18.80,ethernet1/17.1000,All Syslog Servers -Includes VZ,2016/02/25 19:14:20,133312,1,63869,53,60901,53,0x400000,udp,allow,96,96,0,1,2016/02/25 19:14:21,0,any,0,13810046794,0x0,255.255.0.0-255.255.255.255,US,0,1,0,n/a

The current configuration is:

props.conf

[source::///var/log/proxy/paloalto/palo.log]
TRANSFORMS-null = setnull,setnullindex

transforms.conf

[setnull]
REGEX = ^(?:[^,]*?,){31}(\d{1,3}|1[0-3]\d{2}|1400)
DEST_KEY = queue
FORMAT = nullQueue

[setnullindex]
SOURCE_KEY = _MetaData:Index
REGEX = plvpalo
DEST_KEY = queue
FORMAT = nullQueue