Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

[alacercogitatus]

So here's a new one. I have an input (OpenLDAP Audit Logs). Each event (from #modify to #end modify) is generated at the same time, but on multiple lines, which splunk indexes. In props.conf, I set BREAK_ONLY_BEFORE_DATE = False. However when I do the searches, the events are not being parsed correctly. Below is an example of an event, and then the event split. It is acting as if BREAK_ONLY_BEFORE_DATE = True!

The results from a btool debug can be found. I used splunk cmd btool --debug props list. Did I miss something?

props.conf

[ldap:open:audit]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = ^#\s+\w+\s+\d+\s+.*$
BREAK_ONLY_BEFORE_DATE = False
REPORT-MultiValueAudit = loa-MultiValueAudit
REPORT-AuditUser = loa-audituser

transforms.conf

[loa-MultiValueAudit]
DELIMS="\n", ":"
MV_ADD=true

[loa-audituser]
REGEX = ^# modify (\d+) (dc=[\w,=]+) (\w+=([\w\s]*),([,=\w]*))
FORMAT = modifiedTimestamp::$1 suffix::$2 ModifiedBy::$4 ModifierBase::$5 fullModifierName::$3

====== EVENT TO PARSE (sourcetype=ldap:open:audit) =========

# modify 1317923089 dc=ycp,dc=edu cn=Manager,dc=contoso,dc=com  conn=-1
dn: uid=USER,ou=People,dc=contoso,dc=com
changetype: modify
delete: pwdFailureTime
-
replace: entryCSN
entryCSN: 20111006174450.024605Z#000000#004#000000
-
replace: modifiersName
modifiersName: cn=Manager,dc=contoso,dc=com
-
replace: modifyTimestamp
modifyTimestamp: 20111006174450Z
-
# end modify 1317923089

==== End Event ====

==== Results of Event Parse ====

===event1===

# modify 1317923089 dc=ycp,dc=edu cn=Manager,dc=contoso,dc=com  conn=-1
dn: uid=USER,ou=People,dc=contoso,dc=com
changetype: modify
delete: pwdFailureTime
-
replace: entryCSN

===end event1===

===event2===

entryCSN: 20111006174450.024605Z#000000#004#000000
-
replace: modifiersName
modifiersName: cn=Manager,dc=contoso,dc=com
-
replace: modifyTimestamp

===end event2===

===event3===

modifyTimestamp: 20111006174450Z

===end event3===

===event4===

# end modify 1317923089

===end event4===

==== End Results of Event Parse ====

[alacercogitatus · Accepted solution]

I believe I found the issue. The logs were being splunked to indexers that weren't doing the index time configurations from the main indexer. Only the search-time extractions were being performed. So, I added the configs from the main indexer to the secondary indexers and it seems to be working. Thanks for all the recommendations!

[youngsuh]

@alacercogitatus Did you create app or modified the local\props.conf on the indexer for the solution?

[Lowell]

Is any of the stuff on this page helpful? http://splunk-base.splunk.com/answers/4075/whats-the-best-way-to-track-down-propsconf-problems

[_d_]

Hey alacercogitatus, Give this stanza a try:

[ldap:open:audit] 
TIME_PREFIX = ^#\s+modify\s
TIME_FORMAT= %s
MAX_TIMESTAMP_LOOKAHEAD = 11
LINE_BREAKER = ([\r\n]+)(?=^#\s+modify\s\d{10}\s+)
SHOULD_LINEMERGE = false

[youngsuh]

LINE_BREAKER=([\r\n]+)time: worked for me.

[Lowell]

I updated the formatting in this answer and perhaps tweaked regex as well. There seemed to be some extra "\s" in there which sometimes happens with formatting on this site. I'd double check what you have and maybe try this again. This approach should take all event-breaking logic out of the picture by forcing splunk to use custom line-breaking logic instead of event-combining logic which is where you were having issues. (This should also be faster. The downside is that it's more rigid and could cause issues if your log file varies much.)

[alacercogitatus]

nope, it didn't work. It seems no matter what I do, it splits at the date.

[Lowell]

I'm not 100% sure, but here's something to try:

 BREAK_ONLY_BEFORE = ^#\s+\w+\s+\d+\s+

There really isn't any value in trying to match the entire line. And on top of that, trying to use $ could be your problem. In regex terms, depending on the mode, $ can mean end-of-line or end-of-string, and I'm really not sure if this regex is applied line-by-line or not. I just know that I've gotten into trouble before with assumptions about ^ and $, and that could be having an effect here too. In this case, you certainly don't need the extra ".*$" on the end, so I'd drop it in any case.

If that doesn't work, try also replacing the ^ (for the same reason).

 BREAK_ONLY_BEFORE = [\r\n]+#\s+\w+\s+\d+\s+

BTW, is there any reason not to put the word "modify" in there instead of \w+?

Also, I assume you realize that only newly indexed events will be effected by this index-time change.

It wouldn't hurt to add a TIME_PREFIX and TIME_FORMAT in there too, since there seem to be multiple unix epoch time sprinkled in your example.

[Lowell]

BTW, %+ gives a timestamp like: Sat Oct 8 06:41:52 EDT 2011, so %s is certainly what you want here. (Just a side note, I was curious about this. This certainly doesn't solve your problem, unfortunately.)

[alacercogitatus]

I switched it to %s. I found it here. But it's still acting as if BREAK_ONLY_BEFORE_DATE = True. So odd, especially since it used to work......

[Lowell]

I think your TIME_FORMAT should be "%s" and not "%+"; I've never heard of "%+" before, but it's possible I'm missing something. Of course, that doesn't solve your breaking issue.

[alacercogitatus]

Thanks for the reply, but still not working. Here's what I used. I assumed that %+ was a unix timestamp.

[ldap:open:audit]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = [\r\n]+#\s+modify\s+\d+\s+
BREAK_ONLY_BEFORE_DATE = False
TIME_PREFIX = #\s+modify\s+
TIME_FORMAT = %+
REPORT-MultiValueAudit = loa-MultiValueAudit
REPORT-AuditUser = loa-audituser
priority = 100