LINE_BREAKER is being ignored

[dolj]

Hi Community,

I have a JSON data source that I am trying to get into Splunk via a heavy Forwarder using a custom built app that uses an API call. For some reason my LINE_BREAKER seems to be getting ignored

every line ends and starts as follows. 

myemail@this-that-theother.co"},{"specialnumber":"number" 

the line break is the comma between the open and close curly braces..... IOW ,{

this is the line I am using in my props.conf

LINE_BREAKER = (\,)\{\"

for some reason the data continues to come in, in one big blob of multiple events. 

This is my props.conf

KV_MODE = json
SHOULD_LINEMERGE = 0
category = something
pulldown_type = 1
TZ = UTC
TIME_PREFIX=\"time\"\:\"
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT =%Y-%m-%dT%H:%M:SZ
TRUNCATE = 999999
LINE_BREAKER = (\,)\{\"
EVENT_BREAKER_ENABLE = false

Time comes in as such "time":"2025-03-25T19:36:35Z"

Am I missing something? 

[dolj]

As mentioned above the events are coming in as one big blob not broken into separate events based on the line breaker above. 

[PickleRick]

Yes. I understand. The question is how your data is being ingested.

You said that you use a custom app querying an API endpoint. I assume therefore that said app has some modular input which produces data for the forwarder. But said data can be streamed to Splunk process in three ways. https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/custominputs/modinputsscript#Stre...

If your data is streamed as XML and is being incorrectly (not) split into separate events by the modular input since it bypasses the line breaking part of ingestion pipeline completely your LINE_BREAKER settings don't matter.

Of course this is based on my assumption from what little you wrote about your custom ingestion method.

[dolj]

Sorry for being vague I am trying to build the app using the Splunk Add-On-Builder using a rest api call. The problem I am having is the logs are coming in, in one big blob and I have tried multiple line_breaker options and tested them in regex101. 

With respect to the streaming mode. I checked all the .py files associated with the app and could not find any instances of 

<streaming_mode>xml</streaming_mode> 

or 

<streaming_mode>simple</streaming_mode> 

 in any of them. is it one of the cases where i have to add it?  Does Splunk default to XML?

[PickleRick]

By default it's supposed to be simple mode. But (and that's a big but), AOB might default to XML (and might not even be able to do it differently).

You can check it like this (an example from my home lab):

# /opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/TA-api-test/test_input_1.py --scheme

<scheme>
<title>test_input_1</title>
<description>Go to the add-on's configuration UI and configure modular inputs under the Inputs menu.</description>
<use_external_validation>true</use_external_validation>
<streaming_mode>xml</streaming_mode>
<use_single_instance>false</use_single_instance>
<endpoint>
<args>
<arg name="name">
<title>test_input_1 Data Input Name</title>
</arg>

<arg name="placeholder">
<title>placeholder</title>
<required_on_create>0</required_on_create>
<required_on_edit>0</required_on_edit>
</arg>

</args>
</endpoint>
</scheme>

As you can see - it's XML mode. And I'm not sure you can change that. At least I didn't see any option in AOB to change that. You might be able to fiddle with the input definition in AOB to see if it can explicitly break the REST results into separate events.

[dolj]

Very helpful 

it is set to stream XML so I guess that is the issue and I need to either find a way to deal with it or modify the setting which as you mentioned looks easier said than done. 

[PickleRick]

If your input sends broken events in xml mode they are not broken again.