Getting Data In
Highlighted

parsing specific events in logs at heavy forwarder

New Member

Hello,

I am new to splunk and learning it . I am trying the parse the events with specific keyword will dropping the other events from the logs at heavy forwarder. For example, below are the sample logs .

2018-02-21T18:00:13.119575+00:00 apachefront audispd: node=abc.corp.com type=PATH msg=audit(1550772013.107:10434531): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=786685 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL 2018-02-21T18:00:13.1154665+00:00 apachefront audispd: node=apachefront type=EOE msg=audit(1550772013.107:10434531): 2018-02-21T18:00:13.120488+00:00 apachefront audispd: node=apachefront type=SYSCALL msg=audit(155054653.115:103534532): arch=c000003e syscall=59 success=yes exit=0 a0=1053420 a1=10534e0 a2=1050980 a3=7ffe6956c490 items=2 ppid=39078 pid=39084 auid=708926886 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=15253 comm="ps" exe="/bin/ps" key="root" 2018-02-21T18:00:13.12561541+00:00 apachefront audispd: node=apachefront type=EXECVE msg=audit(155564013.115:104455432): a0="ps" a1="-eT" 2018-02-21T18:00:13.121049+00:00 apachefront audispd: node=apachefront type=CWD msg=audit(16872013.115:1062): cwd="/" 2018-02-21T18:00:13.121241+00:00 apachefront audispd: node=apachefront type=PATH msg=audit(1550772013.115:10434532): name="/usr/bin/ps" inode=1995646 dev=fe:02 mode=0100755 ouid=0 rdev=00:00 nametype=NORMAL 2018-02-21T18:00:13.156434+00:00 apachefront audispd: node=apachefront type=PATH msg=audit(1550765463.115:10434532): item=1 name="/lproc/ld-linux-x86-32" inode=7865644 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL

In the logs, i am trying to make heavy forwarder to send the events that have type=SYSCALL and type= EXECEVE while dropping the others. Below is my transforms.conf, however heavy forwarder is dropping the all the events. Any help would be appreciated.

[set_SYSCALL]
REGEX =  \,\d{3}\s*\w+\s*\[type=SYSCALL]
DEST_KEY = queue
FORMAT = nullQueue

[set_EXECVE]
REGEX =  \,\d{3}\s*\w+\s*\[type=EXECVE]
DEST_KEY = queue
FORMAT = nullQueue

Thank you

0 Karma
Highlighted

Re: parsing specific events in logs at heavy forwarder

SplunkTrust
SplunkTrust

Try this

[apachefront-setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[apachefront-setparsing]
REGEX = \stype=(?:SYSCALL|EXECVE)\s
DEST_KEY = queue
FORMAT = indexQueue

And in props.conf call it like so:

[sourcetype_goes_here]
TRANSFORMS-set= apachefront-setnull,apachefront-setparsing

Good luck

0 Karma
Highlighted

Re: parsing specific events in logs at heavy forwarder

New Member

Thank you for the reply. i have tried the configuration that you provided and changes the regex to match only the type that i am looking for. However, splunk is still processing all the events.

REGEX = type=(SYSCALL|EXECVE)

0 Karma
Highlighted

Re: parsing specific events in logs at heavy forwarder

Communicator

It looks like you are sending both stanzas to the nullQueue which is why it is dropping all of the data. You will need to send only those groups to the TCP out queue and drop all of the other data by setting defaultgroup to nothing in outputs.conf

*** Props.conf
[yoursourcetypehere]
TRANSFORMS-syscall = setSYSCALL
TRANSFORMS-execve = set
EXECVE

*** Transforms.conf
[setSYSCALL]
REGEX = \,\d{3}\s\w+\s[type=SYSCALL]
DEST
KEY = TCPROUTING
FORMAT = splunk

[setEXECVE]
REGEX = \,\d{3}\s\w+\s[type=EXECVE]
DEST
KEY = TCPROUTING
FORMAT = splunk

*** outputs.conf
[tcpout]
defaultGroup = nothing

[tcpout:splunk]
server= yoursplunkip_here:9997

This documentation will be a good reference point for you on this: https://docs.splunk.com/Documentation/Splunk/7.2.4/Forwarding/Forwarddatatothird-partysystemsd

0 Karma
Highlighted

Re: parsing specific events in logs at heavy forwarder

Esteemed Legend

Try this:
In props.conf:

[YourSourcetypeHere]
TRANSFORMS-set = setnull, setkeep

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setkeep]
REGEX =  \,\d{3}\s*\w+\s*\[type=(?:SYSCALL|EXECVE)]
DEST_KEY = queue
FORMAT = indexQueue
0 Karma