Splunk Search

Why is my data not parsing correctly?

mhouse3
Path Finder

I am trying to make sure I know how to configure an environment to ingest weblogs that are correctly parsed and I am running into trouble in that I am only getting 1 single event. I have used feedback provided to similar questions to build out my configurations. Note that the original intent of this exercise was to see what would the different effect be with two different props.conf.

My weblog source is this on both forwarders:

  '<photo id="123" title="Birthday" format="jpg">
       <owner id="1111">Jason</owner>
    <CreationDate>2009-11-06T02:22:37.063</CreationDate>
       <comments>
           <comment ownerid="112">Good pic!</comment>
           <comment ownerif="223">Happy birthday</comment>
       <comments>
   </photo>


  <photo id="123" title="Birthday" format="jpg">
       <owner id="1111">Jason</owner>
    <CreationDate>2009-11-06T02:22:37.063</CreationDate>
       <comments>
           <comment ownerid="112">Good pic!</comment>
           <comment ownerif="223">Happy birthday</comment>
       <comments>
   </photo>


  <photo id="123" title="Birthday" format="jpg">
       <owner id="1111">Jason</owner>
    <CreationDate>2009-11-06T02:22:37.063</CreationDate>
       <comments>
           <comment ownerid="112">Good pic!</comment>
           <comment ownerif="223">Happy birthday</comment>
       <comments>
   </photo>'

My inputs.conf on FW1is this:
'[monitor:///home/labuser/xmldata/]
index=web
sourcetype=xml
disabled=false'

My inputs.conf on FW2 is this so that I could figure out which props.conf works:
'[monitor:///home/labuser/xmldata/]
index=web2
sourcetype=xml2
disabled=false'

My props.conf on FW1 is this:
'KV_MODE = xml
LINE_BREAKER = ()
MUST_BREAK_AFTER = \
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE = 0
TIME_PREFIX = \
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N'

My props.conf on FW2 is this:
'KV_MODE = xml
LINE_BREAKER = ([\r\n]+)()
MUST_BREAK_AFTER = \
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE = 0
TIME_PREFIX = \
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N'

All the data in both web index and web2 index looks identical in Splunk that both index=web1 or index=web2 produce identical results in that I only get a single event back instead of multiple events. What am I doing wrong?

0 Karma

manjunathmeti
Champion

For LINE_BREAKER, regex captured group should contain everything between end of previous event and beginning of current event. Try this:

[xml]
KV_MODE = xml
LINE_BREAKER = \<\/photo\>([\r\n\s]+)\<photo
NO_BINARY_CHECK = 1
TRUNCATE = 0
TIME_PREFIX = \<CreationDate\>
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
0 Karma

mhouse3
Path Finder

@manjunathmeti I just realized that my questions got cut off.

I was actually using:
LINE_BREAKER = ()

and later tried using this:
LINE_BREAKER = ([\r\n]+)()

In both cases all the data came in as a single event. These should have worked right?

0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.3.2411?

Hey Splunky People! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2411. This release ...

Buttercup Games: Further Dashboarding Techniques (Part 6)

This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the ...

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...