topic Re: Events breaking on dates instead of the must break only on param in Getting Data In

Events breaking on dates instead of the must break only on param

Cuyose — Tue, 29 Sep 2020 10:20:33 GMT

from btools prop list run on search head.
The events still break on dates within the events rather than the "---------" so we have a bunch of partial events being indexed.
[sourcetypes]
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE = ----------
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG =
HEADER_MODE =
KVMODE =
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 10000
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 1000000
category = Custom
detect_trailing_nulls = false
disabled = false
maxDist = 100
priority =
pulldown_type = true
sourcetype =

Re: Events breaking on dates instead of the must break only on param

somesoni2 — Thu, 21 Jul 2016 16:25:20 GMT

Sample events please.

Re: Events breaking on dates instead of the must break only on param

Cuyose — Thu, 21 Jul 2016 16:42:00 GMT

here is a sample. it will break on the context create time and sometimes the log time, sometimes the date=, but never the ------ explicitly declared

----------
Date=2016-06-20 15:54:20Z
LogLevel=INFO
Logger=XXXXXX.ContactCenter.NGAT.Web.Infrastructure.MultiCacheProviderFactory
Thread=161
LogContext=UIGeneralLog
Message=MultiCacheProvider: Operation succeeded.
MachineName=XXXXXXXX
MethodName=MultiCacheProviderOperationCompletedHandler
SourcePath=E:\jenkins\workspace\XXXXX\web\Infrastructure\MultiCacheProviderFactory.cs
SourceLine=218
ContextCreateTime=2016-06-20 15:54:20Z
LogTime=2016-06-20 15:54:20Z
SessionId=
InteractionIdStringId=f809f344-d728-4d12-9301-889ba406d86f
CallSessionIdStringId=8c5109ad-2ed1-4ed3-8e33-de5c34be4625
ActivityIdStringId=9d49138f-e264-4799-9279-b3c616bf9927
AgentUserName=
UserAgent=Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/7.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+.NET4.0C;+.NET4.0E;+InfoPath.3)
RequestPath=https://XXXXX/
Referrer=https://XXXXXX/?wa=wsignin1.0&wtrealm=https:%2f%XXXXXX%2f&wctx=rm%3d0%26id%3dpassive%26ru%3d%252f&wct=2016-06-20T15:54:20Z
SessionIsNew=
SessionKeys=
Controller=
Action=
MC.RequestId=98052227
MC.Operation=AddOrGetExisting
MC.Result=Success
MC.TotalTimeMillis=15.5998
MC.CacheKey=SessionSecurityToken-/;urn:uuid:cc6794c2-65b8-46f4-a36a-eb015d92a50c;
MC.CacheHit=False

Re: Events breaking on dates instead of the must break only on param

hardikJsheth — Tue, 29 Sep 2020 10:20:41 GMT

Try setting BREAK_ONLY_BEFORE_DATE = false in your props.conf.

Re: Events breaking on dates instead of the must break only on param

Cuyose — Tue, 29 Sep 2020 10:17:27 GMT

still doesnt work
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE = ----------
BREAK_ONLY_BEFORE_DATE = false
CHARSET = UTF-8
DATETIME_CONFIG =
HEADER_MODE =
KVMODE =
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
LOOKUP-dropdowns = dropdownsLookup host OUTPUT unix_category unix_group
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 10000
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 1000000
category = Custom
detect_trailing_nulls = false
disabled = false
maxDist = 100
priority =
pulldown_type = true
sourcetype =

Re: Events breaking on dates instead of the must break only on param

somesoni2 — Thu, 21 Jul 2016 16:55:30 GMT

I would try like this for your sourcetype definition (props.conf on indexer/heavy forwarder)

[sourcetypes]
LINE_BREAKER =([\r\n]+)(----------)
SHOULD_LINEMERGE = false
TIME_PREFIX = Date=
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
TRUNCATE = 1000000

Re: Events breaking on dates instead of the must break only on param

Cuyose — Thu, 21 Jul 2016 18:11:59 GMT

These pushed out to the search head and forwarders and still doesn't work

Re: Events breaking on dates instead of the must break only on param

somesoni2 — Thu, 21 Jul 2016 18:37:41 GMT

As mentioned it should be in indexer or Heavy forwarder. A restart of SPlunk is required after you push the change.

Re: Events breaking on dates instead of the must break only on param

hardikJsheth — Thu, 21 Jul 2016 18:50:43 GMT

Refer to following link. Most of the parameters related to line breaking are required on Indexer and not on forwarder.
https://wiki.splunk.com/Community:HowIndexingWorks

Re: Events breaking on dates instead of the must break only on param

woodcock — Sun, 24 Jul 2016 06:47:23 GMT

Do you have a working solution (this is clicked Accepted)?