Trying to create a line break on source type in the UI for xml import

[jlarousse]

An example of the file is below. I want to break on <Object> and I tried (\<Object>\) and (\<Object\s) with no success. Can someone offer some advice or something to try?

<Objects>

<Object>
<id><\id>
<mac><\mac>
<ip><\ip>
<ip6><\ip6>
<description><\description>
<firstSeen><\firstSeen>
<lastSeen><\lastSeen>
<manufacturer><\manufacturer>
<os><\os>
<user><\user>
<vlan><\vlan>
<wirelessCapabilities><\wirelessCapabilities>
<smInstalled><\smInstalled>
<recentDeviceMac><\recentDeviceMac>
<clientVpnConnections><\clientVpnConnections>
<lldp><\lldp>
<cdp><\cdp>
<Name><\Name>
<Network><\Network>
<NetID><\NetID>
<MXSerial><\MXSerial>
<OrgID><\OrgID>
<OrgName><\OrgName>
<PolicyName><\PolicyName>
<Status><\Status>
<PolicyId><\PolicyId>
<BlockedDate><\BlockedDate>
<\Object>
<Object>
<id><\id>
<mac><\mac>
<ip><\ip>
<ip6><\ip6>
<description><\description>
<firstSeen><\firstSeen>
<lastSeen><\lastSeen>
<manufacturer><\manufacturer>
<os><\os>
<user><\user>
<vlan><\vlan>
<wirelessCapabilities><\wirelessCapabilities>
<smInstalled><\smInstalled>
<recentDeviceMac><\recentDeviceMac>
<clientVpnConnections><\clientVpnConnections>
<lldp><\lldp>
<cdp><\cdp>
<Name><\Name>
<Network><\Network>
<NetID><\NetID>
<MXSerial><\MXSerial>
<OrgID><\OrgID>
<OrgName><\OrgName>
<PolicyName><\PolicyName>
<Status><\Status>
<PolicyId><\PolicyId>
<BlockedDate><\BlockedDate>
<\Object>
<\Objects>

[Vardhan]

Hi,

did u use the custom sourcetype? And the props.conf did you placed in indexers?

I don't find any issue in the props.conf settings.

[jlarousse · Accepted solution]

This issue has been resolved. Long story short, we had to use a workaround. We created a file watcher that imported the data, however, we kept the input script that moved the file after 5 minutes to a new directory so the same data wasn't imported more than once.

The xml data from the script kept being received out of order by Splunk and that is what was causing the parsing issue. Saving the data from the script to file was in order, though. Doesn't make sense why it kept doing this, but you fix it and move on.

[jlarousse]

Yes, I created a custom source type with  no success of getting the same result as you. I even had support on with me and they had me try a few things with no success. They are going through the diag data to figure out why it's still breaking on each line. 

I'm running the splunk btool for my custom source and I get the following.

C:\Program Files\Splunk\bin>splunk btool props list --debug MySourceType
C:\Program Files\Splunk\etc\apps\search\local\props.conf [MySourceType]
C:\Program Files\Splunk\etc\system\default\props.conf ADD_EXTRA_TIME_FIELDS = True
C:\Program Files\Splunk\etc\system\default\props.conf ANNOTATE_PUNCT = True
C:\Program Files\Splunk\etc\apps\search\local\props.conf AUTO_KV_JSON = false
C:\Program Files\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE =
C:\Program Files\Splunk\etc\apps\search\local\props.conf BREAK_ONLY_BEFORE_DATE =
C:\Program Files\Splunk\etc\system\default\props.conf CHARSET = AUTO
C:\Program Files\Splunk\etc\apps\search\local\props.conf DATETIME_CONFIG = CURRENT
C:\Program Files\Splunk\etc\system\default\props.conf DEPTH_LIMIT = 1000
C:\Program Files\Splunk\etc\system\default\props.conf HEADER_MODE =
C:\Program Files\Splunk\etc\apps\search\local\props.conf KV_MODE = xml
C:\Program Files\Splunk\etc\system\default\props.conf LEARN_MODEL = true
C:\Program Files\Splunk\etc\system\default\props.conf LEARN_SOURCETYPE = true
C:\Program Files\Splunk\etc\apps\search\local\props.conf LINE_BREAKER = ([\r\n]+)\<Object>
C:\Program Files\Splunk\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100
C:\Program Files\Splunk\etc\system\default\props.conf MATCH_LIMIT = 100000
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_AGO = 2000
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_HENCE = 2
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800
C:\Program Files\Splunk\etc\system\default\props.conf MAX_EVENTS = 256
C:\Program Files\Splunk\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
C:\Program Files\Splunk\etc\system\default\props.conf MUST_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE =
C:\Program Files\Splunk\etc\apps\search\local\props.conf NO_BINARY_CHECK =
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION = indexing
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-all = full
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-inner = inner
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-outer = outer
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-raw = none
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-standard = standard
C:\Program Files\Splunk\etc\apps\search\local\props.conf SHOULD_LINEMERGE = false
C:\Program Files\Splunk\etc\system\default\props.conf TRANSFORMS =
C:\Program Files\Splunk\etc\system\default\props.conf TRUNCATE = 10000
C:\Program Files\Splunk\etc\apps\search\local\props.conf category = Custom
C:\Program Files\Splunk\etc\system\default\props.conf detect_trailing_nulls = auto
C:\Program Files\Splunk\etc\apps\search\local\props.conf disabled = false
C:\Program Files\Splunk\etc\system\default\props.conf maxDist = 100
C:\Program Files\Splunk\etc\system\default\props.conf priority =
C:\Program Files\Splunk\etc\apps\search\local\props.conf pulldown_type =
C:\Program Files\Splunk\etc\system\default\props.conf sourcetype =

[jlarousse]

I opened a ticket with support, the line breakers I have entered from suggestions here and other places worked for other Splunk customers. Support says it must be something with the way the sourcetype is presenting the data. We have a zoom call setup for today. I will update after we have a fix in hopes it will help someone else.

Thanks for all the suggestions, I appreciate your time and knowledge.

[richgalloway]

Try

LINE_BREAKER = ([\r\n]+)\<Object>

The capture group is critical - Splunk won't break lines without it.

[jlarousse]

I just tried it and it is still breaking at each line.  The strange thing in the search on the Patterns tab it shows <Object>.

This is what I have in the props.conf

KV_MODE = xml
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\<Object>
DATETIME_CONFIG = CURRENT

[richgalloway]

See if this works any better.

LINE_BREAKER = ()\<Object>

Of course, make sure the sourcetype (or whatever is used in the props.conf stanza name matches the data.

[jlarousse]

That didn't work either. I also tried ([\r\n]*)(\<Object\>) and it didn't work. I verified my stanza is correct and notice the Patterns tabe showed all the tags after I did ([\r\n]*)(\<Object\>).