Hello,
I am trying to set up a heavy forwarder that filters the received Events from a universal forwarder and other sources, send them all to another logging solution per syslog, and only a subset of logs should be sent for further use to a Splunk indexer.
For now I am only experimenting with Windows event logs to get a hang of proper filtering, but it simply doesn't work.
Configs on the heavy forwarder in %Splunk_Home%\etc\system\local
Inputs.conf
#no code here
Because I defined the input port for the Splunk forwarder in the Splunk Web, but it would be nice to now in what .conf file Splunk saves this port.
Outputs.conf
[tcpout]
defaultGroup = nothing
[tcpout:OutSplunk]
server = <server>:<port>
[syslog:OutLogger]
server=<server>:<port>
type=tcp
props.conf
[WinEventLog:Security]
TRANSFORMS-route=WinSecEvent-Splunk,Everything
[WinEventLog:Setup]
TRANSFORMS-route=AllWinEvent-Splunk,Everything
[WinEventLog:System]
TRANSFORMS-route=AllWinEvent-Splunk,Everything
[WinEventLog:Application]
TRANSFORMS-route=AllWinEvent-Splunk,Everything
transforms.conf
[WinSecEvent-Splunk]
REGEX=(?msi)(.*Keywords=(?!Audit\sSuccess).*)
DEST_KEY=_TCP_ROUTING
FORMAT=OutSplunk
[AllWinEvent-Splunk]
REGEX=(?msi)(.*Type=(Error|Warning).*)
DEST_KEY=_TCP_ROUTING
FORMAT=OutSplunk
[Everything]
REGEX=.
DEST_KEY=_SYSLOG_ROUTING
FORMAT=OutLogger
The syslog output somehow works, but the filtering for the Windows events not. The indexer gets all events.
To check my regular expressions I use https://regex101.com . Since splunk also uses perl compatible regular expressions (pcre), it should work in theory, but in practice the events don't get filtered.
Update 03.08.2015
11:00 AM
Instead of creating a new question I will ask it here
according to this site
http://docs.splunk.com/Documentation/Splunk/6.2.4/Forwarding/Forwarddatatothird-partysystemsd#Syslog...
the syslog processor of splunk creates rfc 3164 compliant output even with windows event logs but my third-party logging solution receives every line of a win event as a syslog message and not the the whole event as one message.
I will try to contact the support / a splunk partner about it but maybe someone here knows the answer
And by the way the filtering still doesn't work even after playing around a bit with the regex
@maciep
you were right, I found the config for the splunk input set in the web gui in an app
it is in %Splunk_Home%\etc\apps\search\local\input.conf
01:00 PM
@MuS
thanks for the answer that solved the problem with the filtering
I thought that you define a default group for events and messages where you don't have routing rules, so that everything that doesn't have a stanza in props.conf and transforms.conf still gets routed to an default indexer.
By the way all data has to be routed to the third party system and a copy of a subset to splunk
Ok now I still have the problem with the syslog output that is s clearly not rfc 3164 conform
I tested it now with two different third party systems and in both every line of an windows event is received as one syslog message.
Update 04.08.2015
9:00 AM
@maciep
that works by adding this line to every stanza of the windows events
SEDCMD-rmlines=s/[\n\r\t]/ /g
I get the events as one liner to the third party system but now I also get every thing as one line to splunk
is there maybe a way to remove the line breaks in the Everything stanza of transforms.conf or do I really need a another splunk instance to convert windows event logs to syslog before forwarding the message because the syslog processor doesn't create proper output
thanks in advance for the help
Ludwig_mdc
We got this from a vendor who I think got it from the comment section here: http://blogs.splunk.com/2013/10/14/windows-event-logs-in-splunk-6/
In our props on the heavy forwarder for this sourcetype, we have the following. It seems to format the data in a way that the target syslog server expects.
SHOULD_LINEMERGE = true
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none
TRANSFORMS-FIELDS = strip-winevt-linebreaker
SEDCMD-win=s/(?mis)(Token Elevation Type indicates|This event is generated|Application Id=).*$//g
SEDCMD-rmlines=s/[\n\r\t]/ /g
I do not see an answer here. Why was it closed?
We got this from a vendor who I think got it from the comment section here: http://blogs.splunk.com/2013/10/14/windows-event-logs-in-splunk-6/
In our props on the heavy forwarder for this sourcetype, we have the following. It seems to format the data in a way that the target syslog server expects.
SHOULD_LINEMERGE = true
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none
TRANSFORMS-FIELDS = strip-winevt-linebreaker
SEDCMD-win=s/(?mis)(Token Elevation Type indicates|This event is generated|Application Id=).*$//g
SEDCMD-rmlines=s/[\n\r\t]/ /g
it works but you will also get single line windows events in the splunk indexer
therefore I am a bit disappointed that splunk an enterprise logging solution that claims in its documentation to be able to create rfc 3164 compliant syslog output even with event logs
´http://docs.splunk.com/Documentation/Splunk/6.2.4/Forwarding/Forwarddatatothird-partysystemsd#Syslog...
or does that only work if you have the heavy forwarder on every system instead of an universal forwarder
It is unnecessary (maybe even problematic) to update your question's subject-line to indicate a solution. Just add the solution as an "Answer" (not a "Comment") and then click "Accept" on your Answer to close out the question and let others know that there is a validated solution.
Hi Ludwig_MDC,
Your Splunk sends everything to the third party server because you use the defaultGroup
in your outputs.conf
see the docs for details
defaultGroup = <target_group>, <target_group>, ...
* The forwarder sends all data to the specified groups.
cheers, MuS
If you're asking about the receiving port for the heavy forwarder, check inputs.conf. But since you set it in the GUI, it will likely be in one of your apps - like launcher or search etc. It won't be in system/local.
For your config, I would think everything would only be getting sent to the syslog server, so I'm obviously no help there. I'm working on a similar project and have been struggling with routing from a heavy forwarder, so a lot to learn yet there for me. But I do have a ticket open with Splunk so if I gain a better understanding and can apply here as well, I'll follow at that time.