How to edit my configuration to collect Windows event logs with a universal forwarder to send to a syslog collector?

[oliverj]

Yes, this question has been asked a hundred times. I have looked at all of the examples, but my grasp of the different conf files and their interactions is lacking.

First:
I have a Windows device.
It has the Universal forwarder installed. (Version 6.3)
My destination device is a syslog server (TIBCO Loglogic, accepts standard syslog)

My config files are as follows: (This is the entire config, not snippits)
inputs.conf

[default]
host = $decideOnStartup
connection_host = "ip"

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 1

[WinEventLog://Security]
index = winevt
disabled = 0
current_only = 0

transforms.conf

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group

props.conf

[host::10*]
TRANSFORMS-mine = send_to_syslog

[source:*]
SEDCMD-rmlines=s/[\n\r\t]/ /g

outputs.conf

[tcpout:group1]
server=172.17.1.12:514
sendCookedData = false

[syslog:my_syslog_group]
server = 172.17.1.12:514
type = tcp
timestampformat = %b %e %H:%M:%S

The problems I am having:

1. I was hoping for something much more simple. Just something in the outputs.conf: Winevent in, syslog out.
2. I get a lot of junk information (it looks like splunk application info) with "INFO" or "WARN" that has nothing to do with Windows events.
3. Most importantly: My Windows logs are broken into newlines! A single winevent takes 15 or so lines. My transforms.conf seems to do nothing, nor any of the other examples I have seen.

So yes, I am getting Windows logs as syslog, but the data is not usable to the end user due to the newlines.

Any help would be greatly appreciated!

[oliverj · Accepted solution]

Final update:
Use a heavy forwarder, not the universal forwarder.
UDP works out of the box, but I got TCP working with a props.conf edit.
You may need to expand the default message size limit for this.
output.conf

[tcpout]
 defaultGroup = default-autolb-group
 indexAndForward = 0
 useACK = true

 [tcpout:default-autolb-group]
 server = indexer.myindexer.com:9989

 [tcpout-server://indexer.myindexer.com:9989]

 [syslog]
 defaultGroup = syslogGroup

 [syslog:syslogGroup]
 server = indexer.myindexer.com:514
 type = udp
 maxEventSize = 8000

props.conf

[default]
SEDCMD-rmlines=s/[\n\r\t]/ /g
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256

I now have windows event logs coming from universal forwarder -> heavy forwarder -> syslog server as 1 entry per log, using TCP.

Thank you for your help, javiergn

[javiergn]

Why don't you use the Windows Snare agent instead of the universal forwarder? It'll read your event logs and send them over in Syslog format to your Syslog server. There's a free version by the way.

See the following links:

https://www.intersectalliance.com/our-product/snare-agent/

https://answers.splunk.com/search.html?f=&redirect=search%2Fsearch&sort=relevance&q=Snare&type=quest...

Hope that helps.

Thanks,
J

[oliverj]

Found it! Mostly.

First: Although Universal Forwarder will output to Syslog, it is breaking up my windows logs into individual lines.
I tried the heavy forwarder instead, with this output.conf file (All others are default, nothing special in props or transforms):

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0
useACK = true

[tcpout:default-autolb-group]
server = indexer.myindexer.com:9989

[tcpout-server://indexer.myindexer.com:9989]

[syslog]
defaultGroup = syslogGroup

[syslog:syslogGroup]
server = indexer.myindexer.com:514
type = udp
maxEventSize = 8000

When I do this, all events sent to my heavy forwarder are now forwarded out to my syslog server, and they are 1 event per line. The maxEventSize was manually set because windows logs are over the default of (1000?). Careful here, because UDP packets are limited by MTU size. If your MTU isn't large enough to fit the message, it will most likely be truncated.

So, I hate UDP. But when I switch to type = TCP, everything switched back to one event per line!
Can anyone offer insight on that? Leaving everything else the same, just switching type = tcp causes this.

[javiergn]

First of all, it is probably a good practise and will save you from a lot of pain to truncate Windows Event Logs and remove the annoying description that starts with "This event is generated ...". Take a look at the following two answers:

https://answers.splunk.com/answers/4752/disabling-or-removing-extra-description-text-in-windows-2008...

https://answers.splunk.com/answers/4752/disabling-or-removing-extra-description-text-in-windows-2008...

With regards to the the TCP sending 1 event per line, take a look at this:

https://answers.splunk.com/answers/110836/long-json-got-truncated-even-though-truncate-is-set-to-0.h...
https://answers.splunk.com/answers/325823/universal-forwarder-is-truncatingadding-extra-line.html#an...
https://answers.splunk.com/answers/172844/very-long-log-events-coming-over-syslog-514udp-are.html

And of course, check the knowledge base:

http://docs.splunk.com/Documentation/Splunk/6.4.0/Data/Configureeventlinebreaking

Hope that helps.

[oliverj]

That is an excellent plan, but in this environment getting new software approved takes months.
We are migrating off of LogLogic and on to SPLUNK -- the Windows EVT -> Syslog is only an interim solution, while we set up the SPLUNK servers and start migrating over.
Keeping the same forwarding software will make things much easier for all parties involved. Something like Snare would do the trick, but would be adding a 3rd piece of software into the mix, when I am trying to eliminate new applications and simplify deployments for the end users.