Why is my Event line breaking not working properly?

[msichani]

Hi, I've reviewed almost all the question about event line breaking but still have some inconsistency with data ingesting to my Splunk Enterprise. Is there any sort of debugging/logging system for data input and the method which Splunk use to handle multiline events. My data looks like this:

{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
Here's what I've tried in props.conf:

[SOURCETYPE_NAME]
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+){\"timestam_ns\"
TIME_FORMAT = %s%9N
TIME_PREFIX = "timestamp_ns":
For SHOULD_LINEMERGE = false, I've tried other LINE_BREAKER as well, like ^\{, ([\r\n]+)\{, etc but no luck.

[SOURCETYPE_NAME]
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^\{
TIME_FORMAT = %s%9N
TIME_PREFIX = "timestamp_ns":

Just for your information, I've tried manual Add Data feature and both configs works fine there. And I'm testing these stuff on a Splunk Developer Personal License before applying the changes against the actual Enterprise version.

[horsefez]

Hi msichani,

try the following stanza

[SOURCETYPE_NAME]
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = \{\"timestamp\_ns\"
TIME_FORMAT = %s%9N
TIME_PREFIX = "timestamp_ns":

[msichani]

The problem was with HF in between, your stanaza should definitely work as well but with proper routing. Thanks anyway.

[somesoni2 · Accepted solution]

I would give this a try. Also ensure that you kept this config in right place (Indexer/heavy forwarder whichever comes first in flow)

[SOURCETYPE_NAME]
 SHOULD_LINEMERGE = false
 LINE_BREAKER = ([\r\n]+)(?=\s*\{\s*\"timestam_ns\")
 TIME_FORMAT = %s%9N
 TIME_PREFIX = ^\s*\{\s*\"timestam_ns\"
 MAX_TIMESTAMP_LOOKAHEAD = 20

[msichani]

I think the trick was the right place, it was going through heavy forwarder, Added _TCP_ROUTING and it looks fine now. Thanks for the hint @somesoni2