We have a condition where we need to filter out data based on the byte count in the log. We have collapsed the source and sourcetype names coming from different servers and we need to be specific based on the index name.
Instead of:
props.conf
[source::///var/log/paloalto/palo.log]
TRANSFORMS-null = setnull
Can we use for the props.conf configuration:
[index::plvpalo]
TRANSFORMS-null = setnull
Or
[source::///var/log/paloalto/palo.log]
index = plvpalo
TRANSFORMS-null = setnull
I would say No, you can filter logs based on Index name in the way you can use source/sourcetype/host to filter logs, but you can filter out data for a source/sourcetype/host based on the index name. Try something like this
props.conf
[source::///var/log/paloalto/palo.log]
TRANSFORMS-null = setnull
transforms.conf
[setnull]
SOURCE_KEY = _MetaData:Index
REGEX = plvpalo
DEST_KEY = queue
FORMAT = nullQueue
I am already using a REGEX statement in the transforms.conf file to filter out any less than 1400 bytes. Would the transforms.conf work as follows:
[setnull]
SOURCE_KEY = _MetaData:Index
REGEX = plvpalo
REGEX = ^(?:[^,]*?,){31}(\d{1,3}|1[0-3]\d{2}|1400),
DEST_KEY = queue
FORMAT = nullQueue
The above one will not work as the byte size was been check from SOURCE_KEY=_raw
(default) and index name will be checked from SOURCE_KEY = _MetaData:Index
. You can create two transforms.conf stanzas and call them both.
props.conf
[source::///var/log/paloalto/palo.log]
TRANSFORMS-null = setnull,setnullindex
transforms.conf
[setnull]
..keep the current setting that you have...
[setnullindex]
SOURCE_KEY = _MetaData:Index
REGEX = plvpalo
DEST_KEY = queue
FORMAT = nullQueue
It is not performing as expected. Here is what we are trying to accomplish. The log file is a csv and we need to filter out all the events / data that is under 1400 bytes which is found in field 31.
Sample log:
2016/02/25 19:14:20,010401000240,TRAFFIC,start,1,2016/02/25 19:14:20,0.1.2.3,4.5.6.7,8.9.10.11,12.13.14.15,Outbound Services,,,dns,vsys1,TRUST,UNTRUST,ethernet1/18.80,ethernet1/17.1000,All Syslog Servers -Includes VZ,2016/02/25 19:14:20,133312,1,63869,53,60901,53,0x400000,udp,allow,96,96,0,1,2016/02/25 19:14:21,0,any,0,13810046794,0x0,255.255.0.0-255.255.255.255,US,0,1,0,n/a
The current configuration is:
props.conf
[source::///var/log/proxy/paloalto/palo.log]
TRANSFORMS-null = setnull,setnullindex
transforms.conf
[setnull]
REGEX = ^(?:[^,]*?,){31}(\d{1,3}|1[0-3]\d{2}|1400)
DEST_KEY = queue
FORMAT = nullQueue
[setnullindex]
SOURCE_KEY = _MetaData:Index
REGEX = plvpalo
DEST_KEY = queue
FORMAT = nullQueue