Splunk Search

How to modify Windows Servers syslogs on indexer from a multiline format to a single line format before forwarding events to a third party system?

Path Finder

Hello,

We have about 900 Windows servers which are being indexed by our single splunk enterprise instance. We are then forwarding these server logs in a standard syslog format to a 3rd party system. The 3rd party system perceives the logs in a multiline format. We need to convert them to single line because they do not support multiline.

Here is an event example from the 3rd party system:

Dec 29 07:47:18 172.25.32.44 12/29/2014 02:47:17 AM
Dec 29 07:47:18 172.25.32.44 LogName=Security
Dec 29 07:47:18 172.25.32.44 SourceName=Microsoft Windows security auditing.
Dec 29 07:47:18 172.25.32.44 EventCode=4689
Dec 29 07:47:18 172.25.32.44 EventType=0
Dec 29 07:47:18 172.25.32.44 Type=Information
Dec 29 07:47:18 172.25.32.44 ComputerName=MYSERVER.dev.ad
Dec 29 07:47:18 172.25.32.44 TaskCategory=Process Termination
Dec 29 07:47:18 172.25.32.44 OpCode=Info
Dec 29 07:47:18 172.25.32.44 RecordNumber=9663387
Dec 29 07:47:18 172.25.32.44 Keywords=Audit Success
Dec 29 07:47:18 172.25.32.44 Message=A process has exited.
Dec 29 07:47:18 172.25.32.44 Subject:
Dec 29 07:47:18 172.25.32.44    Security ID:            NT AUTHORITY\LOCAL SERVICE
Dec 29 07:47:18 172.25.32.44    Account Name:           LOCAL SERVICE
Dec 29 07:47:18 172.25.32.44    Account Domain:         NT AUTHORITY
Dec 29 07:47:18 172.25.32.44    Logon ID:               0x3e5
Dec 29 07:47:18 172.25.32.44 Process Information:
Dec 29 07:47:18 172.25.32.44    Process ID:     0xa84
Dec 29 07:47:18 172.25.32.44    Process Name:   D:\Program Files (x86)\Citrix\HealthMon\Tests\Citrix\RequestTicket.exe
Dec 29 07:47:18 172.25.32.44    Exit Status:    0x0

How can we go about modifying the feed/events from a standard multiline format to a single line format?

I was told perhaps using regex to change the line breaks to some other delimiter could work.

Please advise as to how we can mod the multiline format to single line format?

0 Karma
1 Solution

Path Finder

The regex below did the trick for us. We modded the props.conf file on the indexer. The SED scripts contain verbose trunking variables as well..its in the first line. The others simply give us the final format. (The following was the standalone stanza for WinEventLog*)

[source::WinEventLog*]
SEDCMD-win=s/(?mis)(Token Elevation Type indicates|This event is generated|Application Id=|Advanced help for this problem).*$//g
SEDCMD-rmlines=s/[\n\r]/ /g
SEDCMD-symtc1=s/([a-zA-Z]+ [a-zA-Z]+ [a-zA-Z]+|[a-zA-Z]+ [a-zA-Z]+|[a-zA-Z]+)=/;\1=/g
SEDCMD-symtc2=s/([a-zA-Z]+ [a-zA-Z]+ [a-zA-Z]+|[a-zA-Z]+ [a-zA-Z]+|[a-zA-Z]+):\s/;\1=/g
SEDCMD-symtc3=s/\s+;/;/g
SEDCMD-symtc4=s/=\s+/=/g

--

Now the events show up like this.

Jan 29 11:51:02 172.25.32.44 HOSTNAME: 01/29/2015 11:50:42 AM;LogName=Security;SourceName=Microsoft Windows security auditing.;EventCode=4689;EventType=0;Type=Information;ComputerName=HOSTNAME;TaskCategory=Process Termination;OpCode=Info;RecordNumber=17576732;Keywords=Audit Success;Message=A process has exited.;Subject=;Security ID=NT AUTHORITY\SYSTEM;Account Name=HOSTNAME$;Account Domain=DEV;Logon ID=0x3e7;Process Information=;Process ID=0x14c8;Process Name=C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe;Exit Status=0x1

--

Thank you Chris P

R

View solution in original post

Path Finder

The regex below did the trick for us. We modded the props.conf file on the indexer. The SED scripts contain verbose trunking variables as well..its in the first line. The others simply give us the final format. (The following was the standalone stanza for WinEventLog*)

[source::WinEventLog*]
SEDCMD-win=s/(?mis)(Token Elevation Type indicates|This event is generated|Application Id=|Advanced help for this problem).*$//g
SEDCMD-rmlines=s/[\n\r]/ /g
SEDCMD-symtc1=s/([a-zA-Z]+ [a-zA-Z]+ [a-zA-Z]+|[a-zA-Z]+ [a-zA-Z]+|[a-zA-Z]+)=/;\1=/g
SEDCMD-symtc2=s/([a-zA-Z]+ [a-zA-Z]+ [a-zA-Z]+|[a-zA-Z]+ [a-zA-Z]+|[a-zA-Z]+):\s/;\1=/g
SEDCMD-symtc3=s/\s+;/;/g
SEDCMD-symtc4=s/=\s+/=/g

--

Now the events show up like this.

Jan 29 11:51:02 172.25.32.44 HOSTNAME: 01/29/2015 11:50:42 AM;LogName=Security;SourceName=Microsoft Windows security auditing.;EventCode=4689;EventType=0;Type=Information;ComputerName=HOSTNAME;TaskCategory=Process Termination;OpCode=Info;RecordNumber=17576732;Keywords=Audit Success;Message=A process has exited.;Subject=;Security ID=NT AUTHORITY\SYSTEM;Account Name=HOSTNAME$;Account Domain=DEV;Logon ID=0x3e7;Process Information=;Process ID=0x14c8;Process Name=C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe;Exit Status=0x1

--

Thank you Chris P

R

View solution in original post

New Member

Where in the props.conf do you add those lines?

Thanks,
Tom

0 Karma

Splunk Employee
Splunk Employee

You can use the Splunk App for CEF to format/transform/filter the data to desired state, before outputting results to the 3rd party system in CEF format. See https://apps.splunk.com/app/1847/#/overview

Path Finder

our 3rd party appliance only accepts syslog format (RFC3164)

We were told it's possible to tweak the UF's themselves by modding the props.conf WinEventlog stanza

This stanza and attributes are responsible for sending to the index in a multiline format - from what we hear.

this is default stanza.

[source::WinEventLog...]
SHOULDLINEMERGE = false
MAX
TIMESTAMPLOOKAHEAD=30
LINE
BREAKER = (\r\n)
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none

TRANSFORMS-FIELDS = strip-winevt-linebreaker

I changed the should_linemerge attribute to "true" and not much changed on the indexer. Yet logs from our linux servers show up as the following which is what we need:

2015-01-22 14:52:48,747 DEBUG fr.test.servlet.filter.XForwardedFilter Incoming request /web/my/home/releaseNotes with originalRemoteAddr '172.22.102.70', originalRemoteHost='172.25.80.70', originalSecure='false', originalScheme='http', original[X-Forwarded-For]='172.25.244.67, original[x-forwarded-proto]='null' will be seen as newRemoteAddr='172.25.246.67', newRemoteHost='172.25.246.200', newScheme='http', newSecure='false', new[X-Forwarded-For]='null, new[X-Forwarded-By]='null'

0 Karma

Path Finder

it turns out that the universal forwarder is not configurable in this way.

all we want to do is remove the date time stamp from the line breaks

0 Karma

Path Finder

Is it possible to do pre-processing on the forwarders ?

0 Karma