parsing specific events in logs at heavy forwarder

[funlearning321]

Hello,

I am new to splunk and learning it . I am trying the parse the events with specific keyword will dropping the other events from the logs at heavy forwarder. For example, below are the sample logs .

2018-02-21T18:00:13.119575+00:00 apachefront audispd: node=abc.corp.com type=PATH msg=audit(1550772013.107:10434531): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=786685 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL 2018-02-21T18:00:13.1154665+00:00 apachefront audispd: node=apachefront type=EOE msg=audit(1550772013.107:10434531): 2018-02-21T18:00:13.120488+00:00 apachefront audispd: node=apachefront type=SYSCALL msg=audit(155054653.115:103534532): arch=c000003e syscall=59 success=yes exit=0 a0=1053420 a1=10534e0 a2=1050980 a3=7ffe6956c490 items=2 ppid=39078 pid=39084 auid=708926886 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=15253 comm="ps" exe="/bin/ps" key="root" 2018-02-21T18:00:13.12561541+00:00 apachefront audispd: node=apachefront type=EXECVE msg=audit(155564013.115:104455432): a0="ps" a1="-eT" 2018-02-21T18:00:13.121049+00:00 apachefront audispd: node=apachefront type=CWD msg=audit(16872013.115:1062): cwd="/" 2018-02-21T18:00:13.121241+00:00 apachefront audispd: node=apachefront type=PATH msg=audit(1550772013.115:10434532): name="/usr/bin/ps" inode=1995646 dev=fe:02 mode=0100755 ouid=0 rdev=00:00 nametype=NORMAL 2018-02-21T18:00:13.156434+00:00 apachefront audispd: node=apachefront type=PATH msg=audit(1550765463.115:10434532): item=1 name="/lproc/ld-linux-x86-32" inode=7865644 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL

In the logs, i am trying to make heavy forwarder to send the events that have type=SYSCALL and type= EXECEVE while dropping the others. Below is my transforms.conf, however heavy forwarder is dropping the all the events. Any help would be appreciated.

[set_SYSCALL]
REGEX =  \,\d{3}\s*\w+\s*\[type=SYSCALL]
DEST_KEY = queue
FORMAT = nullQueue

[set_EXECVE]
REGEX =  \,\d{3}\s*\w+\s*\[type=EXECVE]
DEST_KEY = queue
FORMAT = nullQueue

Thank you

[woodcock]

Try this:
In props.conf:

[YourSourcetypeHere]
TRANSFORMS-set = setnull, setkeep

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setkeep]
REGEX =  \,\d{3}\s*\w+\s*\[type=(?:SYSCALL|EXECVE)]
DEST_KEY = queue
FORMAT = indexQueue

[pkeenan87]

It looks like you are sending both stanzas to the nullQueue which is why it is dropping all of the data. You will need to send only those groups to the TCP out queue and drop all of the other data by setting defaultgroup to nothing in outputs.conf

*** Props.conf
[your_sourcetype_here]
TRANSFORMS-syscall = set_SYSCALL
TRANSFORMS-execve = set_EXECVE

*** Transforms.conf
[set_SYSCALL]
REGEX = \,\d{3}\s*\w+\s*[type=SYSCALL]
DEST_KEY = _TCP_ROUTING
FORMAT = splunk

[set_EXECVE]
REGEX = \,\d{3}\s*\w+\s*[type=EXECVE]
DEST_KEY = _TCP_ROUTING
FORMAT = splunk

*** outputs.conf
[tcpout]
defaultGroup = nothing

[tcpout:splunk]
server= your_splunk_ip_here:9997

This documentation will be a good reference point for you on this: https://docs.splunk.com/Documentation/Splunk/7.2.4/Forwarding/Forwarddatatothird-partysystemsd

[chrisyounger]

Try this

[apachefront-setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[apachefront-setparsing]
REGEX = \stype=(?:SYSCALL|EXECVE)\s
DEST_KEY = queue
FORMAT = indexQueue

And in props.conf call it like so:

[sourcetype_goes_here]
TRANSFORMS-set= apachefront-setnull,apachefront-setparsing

Good luck

[funlearning321]

Thank you for the reply. i have tried the configuration that you provided and changes the regex to match only the type that i am looking for. However, splunk is still processing all the events.

REGEX = type=(SYSCALL|EXECVE)