Getting Data In

Can Splunk do filtering based on the index name rather than source or sourcetype?

babcolee
Path Finder

We have a condition where we need to filter out data based on the byte count in the log. We have collapsed the source and sourcetype names coming from different servers and we need to be specific based on the index name.

Instead of:
props.conf

[source::///var/log/paloalto/palo.log]
TRANSFORMS-null = setnull

Can we use for the props.conf configuration:

[index::plvpalo]
TRANSFORMS-null = setnull

Or

[source::///var/log/paloalto/palo.log]
index = plvpalo
TRANSFORMS-null = setnull
0 Karma

somesoni2
SplunkTrust
SplunkTrust

I would say No, you can filter logs based on Index name in the way you can use source/sourcetype/host to filter logs, but you can filter out data for a source/sourcetype/host based on the index name. Try something like this

props.conf

[source::///var/log/paloalto/palo.log]
TRANSFORMS-null = setnull

transforms.conf

[setnull]
SOURCE_KEY = _MetaData:Index
REGEX = plvpalo
DEST_KEY = queue
FORMAT = nullQueue
0 Karma

babcolee
Path Finder

I am already using a REGEX statement in the transforms.conf file to filter out any less than 1400 bytes. Would the transforms.conf work as follows:

[setnull]
SOURCE_KEY = _MetaData:Index
REGEX = plvpalo
REGEX = ^(?:[^,]*?,){31}(\d{1,3}|1[0-3]\d{2}|1400),
DEST_KEY = queue
FORMAT = nullQueue
0 Karma

somesoni2
SplunkTrust
SplunkTrust

The above one will not work as the byte size was been check from SOURCE_KEY=_raw (default) and index name will be checked from SOURCE_KEY = _MetaData:Index. You can create two transforms.conf stanzas and call them both.

props.conf

 [source::///var/log/paloalto/palo.log]
 TRANSFORMS-null = setnull,setnullindex

transforms.conf

[setnull]
..keep the current setting that you have...

 [setnullindex]
 SOURCE_KEY = _MetaData:Index
 REGEX = plvpalo
 DEST_KEY = queue
 FORMAT = nullQueue
0 Karma

babcolee
Path Finder

It is not performing as expected. Here is what we are trying to accomplish. The log file is a csv and we need to filter out all the events / data that is under 1400 bytes which is found in field 31.

Sample log:

2016/02/25 19:14:20,010401000240,TRAFFIC,start,1,2016/02/25 19:14:20,0.1.2.3,4.5.6.7,8.9.10.11,12.13.14.15,Outbound Services,,,dns,vsys1,TRUST,UNTRUST,ethernet1/18.80,ethernet1/17.1000,All Syslog Servers -Includes VZ,2016/02/25 19:14:20,133312,1,63869,53,60901,53,0x400000,udp,allow,96,96,0,1,2016/02/25 19:14:21,0,any,0,13810046794,0x0,255.255.0.0-255.255.255.255,US,0,1,0,n/a

The current configuration is:

props.conf

[source::///var/log/proxy/paloalto/palo.log]
TRANSFORMS-null = setnull,setnullindex

transforms.conf

[setnull]
REGEX = ^(?:[^,]*?,){31}(\d{1,3}|1[0-3]\d{2}|1400)
DEST_KEY = queue
FORMAT = nullQueue

[setnullindex]
SOURCE_KEY = _MetaData:Index
REGEX = plvpalo
DEST_KEY = queue
FORMAT = nullQueue
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...