How to find events that are not splitting correctly

[freern]

I'm trying to determine which of my companies application logs aren't being split correctly but I'm having a hard time getting a regex search that only returns results that fit the scenario.

The problem part of the raw log is [SKY_LOG_END]\n[SKY_LOG_START]

Any help would be appreciated.

[jkat54]

Will the linecount field help here? In verbose mode you should be able to search for linecount>1 and see all the events with greater than one line.

[jkat54]

If you have access to _internal index, there is a component=linebreak* you can search for too.

[richgalloway]

The "Heath Check" dashboard in the Monitoring Console will report on possible event breaking problems.

[freern]

Thanks for your response. I don't have access to the monitoring console as that side of things is managed by our system administrators. Is there any way to find this information as part of a splunk search using the information i've given?

[richgalloway]

This is the MC search for event processing errors.

index=_internal splunk_server=local search_group=dmc_group_indexer earliest=-60m (source=*splunkd.log* (component=AggregatorMiningProcessor OR component= LineBreakingProcessor OR component=DateParserVerbose) (log_level=WARN OR log_level=ERROR)) OR (source=*metrics.log* group=thruput name=index_thruput) 
| stats sum(eval(round(ev,0))) AS event_count count(eval(component=="AggregatorMiningProcessor")) AS aggregation_issues count(eval(component=="LineBreakingProcessor")) AS line_breaking_issues count(eval(component=="DateParserVerbose")) AS date_parsing_issues by host 
| eval crap_score = round((aggregation_issues + line_breaking_issues + date_parsing_issues) / event_count * 1000, 3) 
| eval severity_level = case(crap_score == 0, 0, crap_score > 0 AND crap_score < 1, 1, True(), 2) 
| rename host AS instance 
| fields - crap_score

[to4kawa]

@freern
please provide sample logs with masking sensitive data like XXXX.