topic Why are events not being split for each date for one heavy forwarder? in Getting Data In

Why are events not being split for each date for one heavy forwarder?

athorat — Wed, 06 Jan 2016 20:04:34 GMT

Events should be split for each date, which is not happening for one of the forwarders:

The following is the part of a single event:

[6/28/16 11:28:37:500 MST] 00000028 Resource      W org.richfaces.resource.ResourceHandlerImpl getResourcePathFromRequest 
[6/28/16 11:28:37:657 MST] 00000028 HtmlLabelRend W   Attribute 'for' of label component with id chgLanguageForm:j_id_1o_2u is not 
[6/28/16 11:28:37:658 MST] 00000028 HtmlLabelRend W   Attribute 'for' of label component with id chgLanguageForm:j_id_1o_2x is not 
[6/28/16 11:28:37:790 MST] 00000028 HtmlLabelRend W   Attribute 'for' of label component with id tasksForm:j_id_1o_9y_2_1 is not 
[6/28/16 11:28:37:791 MST] 00000028 JbpmPageableD I com.exigen.ipb.base.dataload.impl.PageableDataProviderService fetchDataSize 
[6/28/16 11:28:37:792 MST] 00000028 JbpmPageableD I com.exigen.ipb.base.dataload.impl.PageableDataProviderService fetchDataSize

Props.conf on the heavy forwarder:

[PASAPP:PP1:SYSTEMOUT]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\s*Date

Anything which I am missing here?

Anil.

Re: Why are events not being split for each date for one heavy forwarder?

jkat54 — Wed, 06 Jan 2016 20:26:23 GMT

Please try this:

 [PASAPP:PP1:SYSTEMOUT]
 SHOULD_LINEMERGE = false
 LINE_BREAKER = \[\d+\/\d+\/\d+\s\d{2}:\d{2}:\d{2}:\d{3}\s\w\w\w]

Re: Why are events not being split for each date for one heavy forwarder?

aljohnson_splun — Tue, 29 Sep 2020 08:20:19 GMT

Its interesting you are indexing data so far in the future ! So I had to add MAX_DAYS_HENCE. Unless I'm parsing your date wrong (likely).

I copied your sample, put it in the data preview. This is what I got from this props.conf:

SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%m/%d/%y %H:%M:%S:%3N %Z
TIME_PREFIX=^\s+\[
MAX_DAYS_HENCE=10950

It seems to me that you don't need to specify the LINE_BREAKER - and can accept the default, which is :

* Defaults to ([\r\n]+), meaning data is broken into an event for each line,
  delimited by any number of carriage return or newline characters.

Re: Why are events not being split for each date for one heavy forwarder?

athorat — Tue, 29 Sep 2020 08:20:22 GMT

@aljohnson_splunk yes thats because the server clock changes as part of the testing process

So you suggesting to use???
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%m/%d/%y %H:%M:%S:%3N %Z
TIME_PREFIX=^s+[

Re: Why are events not being split for each date for one heavy forwarder?

athorat — Wed, 06 Jan 2016 21:41:49 GMT

@jkat54

Thanks for the reply will try this and update you if it works.

Re: Why are events not being split for each date for one heavy forwarder?

aljohnson_splun — Thu, 07 Jan 2016 01:18:39 GMT

When I tested on the small amount of data you gave, the settings above seemed to work for me, but in any scenario, you should test all of your input settings on a testing environment, or if that is not available, a testing index of course.

Re: Why are events not being split for each date for one heavy forwarder?

jkat54 — Tue, 29 Sep 2020 08:19:24 GMT

Yeah I didnt see the date. I think you'll need to use the MAX_DAYS_HENCE too.

I'm telling you to use a LINE_BREAKER because it seems maybe these are all coming in as one line. But if there are indeed good carriage returns / line feeds... then you dont need the line breaker.

Re: Why are events not being split for each date for one heavy forwarder?

athorat — Thu, 07 Jan 2016 17:30:54 GMT

Thanks for the update @jkat54 that did not work though.
props.conf is updated on the heavy forwarder only.

Re: Why are events not being split for each date for one heavy forwarder?

athorat — Fri, 08 Jan 2016 17:20:16 GMT

@aljohnson_splunk

after updating the settings on universal forwarder
the events started splitting up, but the error messages , as well are split on every line.
so for example the following error message is split as each event.

at org.jbpm.graph.def.Transition.take(Transition.java:151)
at org.jbpm.graph.def.Node.enter(Node.java:319)
at org.jbpm.graph.node.TaskNode.execute(TaskNode.java:168)
at org.jbpm.taskmgmt.exe.TaskMgmtInstance.createTaskInstance(TaskMgmtInstance.java:154)
at org.jbpm.taskmgmt.exe.TaskInstance.assign(TaskInstance.java:204)
at org.jbpm.taskmgmt.exe.TaskMgmtInstance.performAssignment(TaskMgmtInstance.java:216)
Caused by: org.jbpm.graph.def.DelegationException: Policy must have an agent!

Re: Why are events not being split for each date for one heavy forwarder?

aljohnson_splun — Fri, 08 Jan 2016 21:04:32 GMT

Oh, so you're saying that these are for sure NOT single line log files ? In that case, we definitely should have SHOULD_LINEMERGE=true (default). What I did to test was add your trace at the end of the events you added above - this basically lets the default value for BREAK_ONLY_BEFORE_DATE=true (default) do its job.

After I changed SHOULD_LINEMERGE to true, it looks like it is parsing again normally. But these are all just approximations when I obviously can't look at all the data. I'd suggest looking through the props.conf section here on SHOULD_LINEMERGE=true and seeing if there are other settings that you might need. You got this !