Hi, I've reviewed almost all the question about event line breaking but still have some inconsistency with data ingesting to my Splunk Enterprise. Is there any sort of debugging/logging system for data input and the method which Splunk use to handle multiline events. My data looks like this:
{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
{"timestamp_ns":1497623896051426216,"timestamp":1497623896,"measurements":{"response_send_time_us":147,"walltime_us":1003,"xxxxx_walltime_us":493,"xxxxx_walltime_us":510,"xxxxx_time_us":159,"xxxxx_time_us":120,"xxxxx_time_us":82},"application":"xxxxx","type":"xxxxx_query_request","metadata":{"request_type":2,"xx_id":1,"request_timestamp_ns":1497623896050422653,"request_id":"1234567890123456789"}}
Here's what I've tried in props.conf:
[SOURCETYPE_NAME]
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+){\"timestam_ns\"
TIME_FORMAT = %s%9N
TIME_PREFIX = "timestamp_ns":
For SHOULD_LINEMERGE = false , I've tried other LINE_BREAKER as well, like ^\{ , ([\r\n]+)\{ , etc but no luck.
[SOURCETYPE_NAME]
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^\{
TIME_FORMAT = %s%9N
TIME_PREFIX = "timestamp_ns":
Just for your information, I've tried manual Add Data feature and both configs works fine there. And I'm testing these stuff on a Splunk Developer Personal License before applying the changes against the actual Enterprise version.
... View more