Getting Data In

On a heavy forwarder, can I forward a subset of data to syslog and drop everything else?

reswob4
Builder

Here is my situation: I have a Windows HF that is collecting a lot of different data. Some via powershell scripts, some via WMI, some via file monitoring locally and over UNC paths. All of that data is being forwarded to two indexes.

A few weeks ago I configured one of the file monitoring inputs to send a copy of the data it collected to a syslog server.

I now need to send that data (collected via file monitoring) to the syslog server and NOT to the indexers. IOW, I want all data collected by this HF to go to the indexers, EXCEPT this data which should be sent to the syslog server ONLY.

How do I do that?

I've read through this which helped me get the current configuration:
http://docs.splunk.com/Documentation/SplunkCloud/6.6.1/Forwarding/Forwarddatatothird-partysystemsd

Here are my config files:

.../etc/apps/myapp/local/props.conf:

[WinDNS]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 1000
EXTRACT-Domain = (?i) .*? \.(?P<Domain>[-a-zA-Z0-9@:%_\+.~#?;//=]{2,256}\.[a-z]{2,6})
EXTRACT-src = (?i) [Rcv|Snd] (?P<source_address>\d+\.\d+\.\d+\.\d+)
EXTRACT-Threat_ID,Context,Int_packet_ID,proto,mode,Xid,type,Opcode,Flags_Hex,char_code,ResponseCode,question_type = .+?[AM|PM]\s+(?<Threat_ID>\w+)\s+(?<Context>\w+)\s+(?<Int_packet_ID>\w+)\s+(?<proto>\w+)\s+(?<mode>\w+)\s+\d+\.\d+\.\d+\.\d+\s+(?<Xid>\w+)\s(?<type>(?:R)?)\s+(?<Opcode>\w+)\s+\[(?<Flags_Hex>\w+)\s(?<char_codes>.+?)(?<ResponseCode>[A-Z]+)\]\s+(?<question_type>\w+)\s
EXTRACT-Authoritative_Answer,TrunCation,Recursion_Desired,Recursion_Available = (?m) .+?Message:\W.+\W.+\W.+\W.+\W.+AA\s+(?<Authoritative_Answer>\d)\W.+TC\s+(?<TrunCation>\d)\W.+RD\s+(?<Recursion_Desired>\d)\W.+RA\s+(?<Recursion_Available>\d)
TRANSFORMS-droplocal2 = droplocal2
TRANSFORMS-dropbach = dropbach
#TRANSFORMS-dropall = dropall
SEDCMD-win_dns = s/\(\d+\)/./g
TRANSFORMS-dns = send_to_syslog

.../etc/apps/myapp/local/transforms.conf

[dropbach]
REGEX = \[.+?\]\s+\w+\s+.+?BACH
DEST_KEY=queue
FORMAT=nullQueue

[droplocal2]
REGEX = \[.+?\]\s+\w+\s+.+?local
DEST_KEY=queue
FORMAT=nullQueue

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group

#[dropall]
#REGEX = .
#DEST_KEY=queue
#FORMAT=nullQueue

.../etc/system/local/outputs.conf

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0

[tcpout-server://splunk-01:9997]

[tcpout:default-autolb-group]
disabled = false
server = splunk-01:9997,splunk-02:9997

[tcpout-server://splunk-02:9997]  # not sure why this is here....

[syslog:my_syslog_group]
server = 1.1.1.5:514

As you can tell, I tried to add a 'dropall', but that just dropped everything without sending a copy to the syslog server first.

I then found this forum post:

https://answers.splunk.com/answers/4083/can-i-route-some-data-as-syslog-output-to-multiple-destinati...

which seems to imply that to do what I want to do, I need to modify the outputs.conf so the defaultgroup=nothing and then modify all the props.conf and transforms.conf files for all my inputs to point to the "default-autolb-group" in outputs.conf that sends to the indexers, and then for this app have the ONLY output reference pointing to the "my_syslog_group" in outputs.conf.

Is that correct or something else?

0 Karma

reswob4
Builder
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!