topic Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE in Getting Data In

Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

alacercogitatus — Mon, 28 Sep 2020 09:57:50 GMT

So here's a new one. I have an input (OpenLDAP Audit Logs). Each event (from #modify to #end modify) is generated at the same time, but on multiple lines, which splunk indexes. In props.conf, I set BREAK_ONLY_BEFORE_DATE = False. However when I do the searches, the events are not being parsed correctly. Below is an example of an event, and then the event split. It is acting as if BREAK_ONLY_BEFORE_DATE = True!

The results from a btool debug can be found. I used splunk cmd btool --debug props list. Did I miss something?

props.conf

[ldap:open:audit]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = ^#\s+\w+\s+\d+\s+.*$
BREAK_ONLY_BEFORE_DATE = False
REPORT-MultiValueAudit = loa-MultiValueAudit
REPORT-AuditUser = loa-audituser

transforms.conf

[loa-MultiValueAudit]
DELIMS="\n", ":"
MV_ADD=true

[loa-audituser]
REGEX = ^# modify (\d+) (dc=[\w,=]+) (\w+=([\w\s]*),([,=\w]*))
FORMAT = modifiedTimestamp::$1 suffix::$2 ModifiedBy::$4 ModifierBase::$5 fullModifierName::$3

====== EVENT TO PARSE (sourcetype=ldap:open:audit) =========

# modify 1317923089 dc=ycp,dc=edu cn=Manager,dc=contoso,dc=com  conn=-1
dn: uid=USER,ou=People,dc=contoso,dc=com
changetype: modify
delete: pwdFailureTime
-
replace: entryCSN
entryCSN: 20111006174450.024605Z#000000#004#000000
-
replace: modifiersName
modifiersName: cn=Manager,dc=contoso,dc=com
-
replace: modifyTimestamp
modifyTimestamp: 20111006174450Z
-
# end modify 1317923089

==== End Event ====

==== Results of Event Parse ====

===event1===

# modify 1317923089 dc=ycp,dc=edu cn=Manager,dc=contoso,dc=com  conn=-1
dn: uid=USER,ou=People,dc=contoso,dc=com
changetype: modify
delete: pwdFailureTime
-
replace: entryCSN

===end event1===

===event2===

entryCSN: 20111006174450.024605Z#000000#004#000000
-
replace: modifiersName
modifiersName: cn=Manager,dc=contoso,dc=com
-
replace: modifyTimestamp

===end event2===

===event3===

modifyTimestamp: 20111006174450Z

===end event3===

===event4===

# end modify 1317923089

===end event4===

==== End Results of Event Parse ====

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

Lowell — Fri, 07 Oct 2011 18:08:42 GMT

I'm not 100% sure, but here's something to try:

 BREAK_ONLY_BEFORE = ^#\s+\w+\s+\d+\s+

There really isn't any value in trying to match the entire line. And on top of that, trying to use $ could be your problem. In regex terms, depending on the mode, $ can mean end-of-line or end-of-string, and I'm really not sure if this regex is applied line-by-line or not. I just know that I've gotten into trouble before with assumptions about ^ and $, and that could be having an effect here too. In this case, you certainly don't need the extra ".*$" on the end, so I'd drop it in any case.

If that doesn't work, try also replacing the ^ (for the same reason).

 BREAK_ONLY_BEFORE = [\r\n]+#\s+\w+\s+\d+\s+

BTW, is there any reason not to put the word "modify" in there instead of \w+?

Also, I assume you realize that only newly indexed events will be effected by this index-time change.

It wouldn't hurt to add a TIME_PREFIX and TIME_FORMAT in there too, since there seem to be multiple unix epoch time sprinkled in your example.

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

alacercogitatus — Mon, 28 Sep 2020 09:57:53 GMT

Thanks for the reply, but still not working. Here's what I used. I assumed that %+ was a unix timestamp.

[ldap:open:audit]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = [\r\n]+#\s+modify\s+\d+\s+
BREAK_ONLY_BEFORE_DATE = False
TIME_PREFIX = #\s+modify\s+
TIME_FORMAT = %+
REPORT-MultiValueAudit = loa-MultiValueAudit
REPORT-AuditUser = loa-audituser
priority = 100

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

Lowell — Fri, 07 Oct 2011 19:42:30 GMT

I think your TIME_FORMAT should be "%s" and not "%+"; I've never heard of "%+" before, but it's possible I'm missing something. Of course, that doesn't solve your breaking issue.

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

alacercogitatus — Mon, 28 Sep 2020 09:57:56 GMT

I switched it to %s. I found it here. But it's still acting as if BREAK_ONLY_BEFORE_DATE = True. So odd, especially since it used to work......

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

_d_ — Mon, 10 Oct 2011 21:42:48 GMT

Hey alacercogitatus, Give this stanza a try:

[ldap:open:audit] 
TIME_PREFIX = ^#\s+modify\s
TIME_FORMAT= %s
MAX_TIMESTAMP_LOOKAHEAD = 11
LINE_BREAKER = ([\r\n]+)(?=^#\s+modify\s\d{10}\s+)
SHOULD_LINEMERGE = false

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

alacercogitatus — Tue, 11 Oct 2011 12:57:57 GMT

nope, it didn't work. It seems no matter what I do, it splits at the date.

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

Lowell — Tue, 11 Oct 2011 15:20:00 GMT

BTW, %+ gives a timestamp like: Sat Oct 8 06:41:52 EDT 2011, so %s is certainly what you want here. (Just a side note, I was curious about this. This certainly doesn't solve your problem, unfortunately.)

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

Lowell — Tue, 11 Oct 2011 15:28:43 GMT

I updated the formatting in this answer and perhaps tweaked regex as well. There seemed to be some extra "\s" in there which sometimes happens with formatting on this site. I'd double check what you have and maybe try this again. This approach should take all event-breaking logic out of the picture by forcing splunk to use custom line-breaking logic instead of event-combining logic which is where you were having issues. (This should also be faster. The downside is that it's more rigid and could cause issues if your log file varies much.)

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

Lowell — Tue, 11 Oct 2011 15:46:00 GMT

Is any of the stuff on this page helpful? http://splunk-base.splunk.com/answers/4075/whats-the-best-way-to-track-down-propsconf-problems

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

alacercogitatus — Tue, 11 Oct 2011 17:22:59 GMT

I believe I found the issue. The logs were being splunked to indexers that weren't doing the index time configurations from the main indexer. Only the search-time extractions were being performed. So, I added the configs from the main indexer to the secondary indexers and it seems to be working. Thanks for all the recommendations!

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

youngsuh — Fri, 24 Apr 2020 15:31:59 GMT

@alacercogitatus Did you create app or modified the local\props.conf on the indexer for the solution?

Re: Linebreaking Failure - BREAK_ONLY_BEFORE_DATE

youngsuh — Tue, 05 May 2020 20:16:07 GMT

LINE_BREAKER=([\r\n]+)time: worked for me.