Why are events not being split for each date for one heavy forwarder?

[athorat]

Events should be split for each date, which is not happening for one of the forwarders:

The following is the part of a single event:

[6/28/16 11:28:37:500 MST] 00000028 Resource      W org.richfaces.resource.ResourceHandlerImpl getResourcePathFromRequest 
[6/28/16 11:28:37:657 MST] 00000028 HtmlLabelRend W   Attribute 'for' of label component with id chgLanguageForm:j_id_1o_2u is not 
[6/28/16 11:28:37:658 MST] 00000028 HtmlLabelRend W   Attribute 'for' of label component with id chgLanguageForm:j_id_1o_2x is not 
[6/28/16 11:28:37:790 MST] 00000028 HtmlLabelRend W   Attribute 'for' of label component with id tasksForm:j_id_1o_9y_2_1 is not 
[6/28/16 11:28:37:791 MST] 00000028 JbpmPageableD I com.exigen.ipb.base.dataload.impl.PageableDataProviderService fetchDataSize 
[6/28/16 11:28:37:792 MST] 00000028 JbpmPageableD I com.exigen.ipb.base.dataload.impl.PageableDataProviderService fetchDataSize 

Props.conf on the heavy forwarder:

[PASAPP:PP1:SYSTEMOUT]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\s*Date

Anything which I am missing here?

Anil.

[aljohnson_splun]

Its interesting you are indexing data so far in the future ! So I had to add MAX_DAYS_HENCE. Unless I'm parsing your date wrong (likely).

I copied your sample, put it in the data preview. This is what I got from this props.conf:

SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%m/%d/%y %H:%M:%S:%3N %Z
TIME_PREFIX=^\s+\[
MAX_DAYS_HENCE=10950

It seems to me that you don't need to specify the LINE_BREAKER - and can accept the default, which is :

* Defaults to ([\r\n]+), meaning data is broken into an event for each line,
  delimited by any number of carriage return or newline characters.

[athorat]

@aljohnson_splunk yes thats because the server clock changes as part of the testing process

So you suggesting to use???
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%m/%d/%y %H:%M:%S:%3N %Z
TIME_PREFIX=^s+[

[aljohnson_splun]

When I tested on the small amount of data you gave, the settings above seemed to work for me, but in any scenario, you should test all of your input settings on a testing environment, or if that is not available, a testing index of course.

[jkat54]

Yeah I didnt see the date. I think you'll need to use the MAX_DAYS_HENCE too.

I'm telling you to use a LINE_BREAKER because it seems maybe these are all coming in as one line. But if there are indeed good carriage returns / line feeds... then you dont need the line breaker.

[athorat]

@aljohnson_splunk

after updating the settings on universal forwarder
the events started splitting up, but the error messages , as well are split on every line.
so for example the following error message is split as each event.

at org.jbpm.graph.def.Transition.take(Transition.java:151)
at org.jbpm.graph.def.Node.enter(Node.java:319)
at org.jbpm.graph.node.TaskNode.execute(TaskNode.java:168)
at org.jbpm.taskmgmt.exe.TaskMgmtInstance.createTaskInstance(TaskMgmtInstance.java:154)
at org.jbpm.taskmgmt.exe.TaskInstance.assign(TaskInstance.java:204)
at org.jbpm.taskmgmt.exe.TaskMgmtInstance.performAssignment(TaskMgmtInstance.java:216)
Caused by: org.jbpm.graph.def.DelegationException: Policy must have an agent!

[aljohnson_splun]

Oh, so you're saying that these are for sure NOT single line log files ? In that case, we definitely should have SHOULD_LINEMERGE=true (default). What I did to test was add your trace at the end of the events you added above - this basically lets the default value for BREAK_ONLY_BEFORE_DATE=true (default) do its job.

After I changed SHOULD_LINEMERGE to true, it looks like it is parsing again normally. But these are all just approximations when I obviously can't look at all the data. I'd suggest looking through the props.conf section here on SHOULD_LINEMERGE=true and seeing if there are other settings that you might need. You got this !

[athorat]

Thanks for the update @jkat54 that did not work though.
props.conf is updated on the heavy forwarder only.

[jkat54]

Please try this:

 [PASAPP:PP1:SYSTEMOUT]
 SHOULD_LINEMERGE = false
 LINE_BREAKER = \[\d+\/\d+\/\d+\s\d{2}:\d{2}:\d{2}:\d{3}\s\w\w\w]

[athorat]

@jkat54

Thanks for the reply will try this and update you if it works.