topic Re: Help parsing incoming data in Getting Data In

Help parsing incoming data

secphilomath1 — Fri, 22 Dec 2023 15:23:10 GMT

We have data coming in that we need to alert on, however because of the formatting of the data, this is very hard to do. The data is coming in as key value pairs but the values are not encapsulated in quotes and is being truncated. For example

_Raw - filepath=c:\program files\abc123\

What we end up getting is

Parsed - filepath=c:\program

Everything after the space is ignored.

If I wanted to find all occurrences where the path was c:\program files\abc123, I can't.

We are sending the data via syslog to the splunk servers

Thanks in advance!

Re: Help parsing incoming data

gcusello — Fri, 22 Dec 2023 16:23:47 GMT

Hi @secphilomath1 ,

what technology are you using for these data?

if they are standard, you can use the related add-on that gives you al the parsing rules.

If it's custom, you have t omanually parse it.

Ciao.

Giuseppe

Re: Help parsing incoming data

richgalloway — Fri, 22 Dec 2023 16:34:42 GMT

Please share the props and transforms for that sourcetype as well as a couple of sanitized sample events.

Re: Help parsing incoming data

secphilomath1 — Fri, 22 Dec 2023 16:55:43 GMT

@richgalloway

The data is coming from a FIM product called Tripwire. Here is the raw data;

Dec 22 02:30:34 10.62.32.10 1 2023-12-22T10:30:34.771Z servernameTW_ES - - - CEF:0|Tripwire|Enterprise|5.5|6|Audit Event|1|UserName=NT AUTHORITY\NETWORK SERVICE UserNameLabel=User Name ElementName=null ElementNameLabel=Element Name VersionTimeStamp=null VersionTimeStampLabel=Version Timestamp Message='C:\Windows\System32\Tasks\Microsoft\Windows\SoftwareProtectionPlatform\SvcRestartTask' accessed by 'NT AUTHORITY\NETWORK SERVICE'. Type 'Set Security'. Application: 'C:\Windows\System32\svchost.exe' Details: DACL Category=Audit Event CategoryLabel=Category rt=12/22/23 2:25 AM Level=Information LevelLabel=level dhost=trip.cs.ad.domain.com

I don't have any props or transforms yet because I am not sure where to start with this.

Thanks

Re: Help parsing incoming data

richgalloway — Thu, 28 Dec 2023 19:24:33 GMT

Everything ingested by Splunk should have props.conf settings. Start with the "Great 8": LINE_BREAKER, SHOULD_LINEMERGE, TIME_PREFIX, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD, TRUNCATE, EVENT_BREAKER_ENABLE, and EVENT_BREAKER.

Field extraction from events like this are tricky because the field delimiter is also an allowed character within a field. It means using lookahead to determine if the current character is part of a field name or field value. As it turns out, Splunk is not great with lookahead. Try these settings to see if they work for you.

Props.conf:

[mysourcetype] SHOULD_LINEMERGE=true LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true TIME_PREFIX=\s\d\s TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N%Z TRANSFORMS-extract = tripwire_fields TRUNCATE = 10000 EVENT_BREAKER_ENABLE = true EVENT_BREAKER = ([\r\n]+)

Transforms.conf:

[tripwire_fields] REGEX = (\w+)=(.*?)(?=\s\w+=) FORMAT = $1::$2

Re: Help parsing incoming data

secphilomath1 — Wed, 27 Dec 2023 19:22:43 GMT

Thank you, I think this solved it!

Re: Help parsing incoming data

richgalloway — Thu, 28 Dec 2023 19:25:15 GMT

Please note: I added a missing "2" at the end of the transforms.conf code.

If your problem is resolved, then please click the "Accept as Solution" button to help future readers.