All Apps and Add-ons
Highlighted

How to remove the header details with null value in it and at same time how to retain the header details with values in splunk console?

Motivator

Hi Splunkers, I got a requirement from the cyber security user to remove the events with header displaying empty and retain those events with header details having some values into it.

Header Details: USERNAME LINE HOSTNAME TIME

We are using Splunk Add-on for Unix and Linux version 5.2.4 to capture the UNIX details in splunk and the who.sh scripted input are used to capture the who logged on to the system details and both the Linux, Solaris OS details are successfully ingested into splunk via scripted inputs but there are many events with zero values in it and only Header details are shown as a event which we wanted to remove it.

Attached snap shot for your references.alt text

. dirname $0/common.sh

CMD='who -H'
HEADER='USERNAME LINE HOSTNAME TIME'
HEADERIZE='{NR == 1 && $0 = header}'
FORMAT='{length(hostname) || hostname=$NF; gsub("[)(]", "",hostname); time=$3; for (i=4; i<=lastTimeColumn; i++) time = time " " $i}'
PRINTF='{if (NR == 1) {print $0} else {printf "%-14s %-10s %-40.40s %-s\n", $1,$2,hostname,time}}'

if [ "x$KERNEL" = "xLinux" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 5) {hostname = ""; lastTimeColumn = NF}}'

elif [ "x$KERNEL" = "xSunOS" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = ""; lastTimeColumn = NF}}'

elif [ "x$KERNEL" = "xAIX" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = ""; lastTimeColumn = NF}}'

elif [ "x$KERNEL" = "xDarwin" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = ""; lastTimeColumn = NF}}'

elif [ "x$KERNEL" = "xFreeBSD" ] ; then
FILL_BLANKS='{hostname = ""; lastTimeColumn = NF-1; if (NF < 6) {hostname = ""; lastTimeColumn = NF}}'
fi

assertHaveCommand $CMD
$CMD | tee $TEEDEST | $AWK "$HEADERIZE $FILLBLANKS $FORMAT $PRINTF" header="$HEADER"
echo "Cmd = [$CMD]; | $AWK '$HEADERIZE $FILLBLANKS $FORMAT $PRINTF' header=\"$HEADER\"" >> $TEEDEST

Note: we have used Line_Breaker stanza in props.conf to break the multiple events into single events as per the requirement.

[who]
SHOULDLINEMERGE=false
LINE
BREAKER=([\r\n]+)
TRUNCATE=1000000
DATETIME_CONFIG = CURRENT

Kindly guide us on how to do this customization.

Highlighted

Re: How to remove the header details with null value in it and at same time how to retain the header details with values in splunk console?

Motivator

Can anyone throw me some lights on this issues ?

0 Karma
Highlighted

Re: How to remove the header details with null value in it and at same time how to retain the header details with values in splunk console?

Motivator

You will have to write props.conf and transforms.conf to remove unwatered rows from your logs.

If you have HF,

Add the below props.conf in your search app local folder.

[yoursourcetype]
TRANSFORMS-set = setnull

And create transforms.conf in the same folder
And add below lines to it

[setnull]
REGEX = USERNAME
DEST_KEY = queue
FORMAT = nullQueue

If you dont have heavy forwarder add this in indexer and restart splunkd.

0 Karma
Highlighted

Re: How to remove the header details with null value in it and at same time how to retain the header details with values in splunk console?

Motivator

Hi Thambisetty, thanks for your support, I think I have tried to this but it did not work out, yes we have HF for parsing the data before indexing the data.

Props:
[yoursourcetype]
TRANSFORMS-set= setnull,setparsing

Transforms.conf :

Who

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

basically keeping anything that has header and at least one data row

[setparsing]
REGEX = ^(USERNAME).+[\r\n]+\w+
DEST_KEY = queue
FORMAT = indexQueue

Anyway let me try it once again.

0 Karma
Highlighted

Re: How to remove the header details with null value in it and at same time how to retain the header details with values in splunk console?

Ultra Champion

Make sure the addon is also deployed on the indexers, search heads, and any heavy forwards in the env. I once experienced this and it was showing up merely because I forgot to follow the documentation regarding where to deploy the TA and so the sourcetypes were not kicking in correctly.

0 Karma