Getting Data In

Events breaking on dates instead of the must break only on param

Cuyose
Builder

from btools prop list run on search head.
The events still break on dates within the events rather than the "---------" so we have a bunch of partial events being indexed.
[sourcetypes]
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE = ----------
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG =
HEADER_MODE =
KVMODE =
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 10000
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 1000000
category = Custom
detect_trailing_nulls = false
disabled = false
maxDist = 100
priority =
pulldown_type = true
sourcetype =

Tags (2)
0 Karma
1 Solution

somesoni2
Revered Legend

I would try like this for your sourcetype definition (props.conf on indexer/heavy forwarder)

[sourcetypes]
LINE_BREAKER =([\r\n]+)(----------)
SHOULD_LINEMERGE = false
TIME_PREFIX = Date=
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
TRUNCATE = 1000000

View solution in original post

0 Karma

somesoni2
Revered Legend

I would try like this for your sourcetype definition (props.conf on indexer/heavy forwarder)

[sourcetypes]
LINE_BREAKER =([\r\n]+)(----------)
SHOULD_LINEMERGE = false
TIME_PREFIX = Date=
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
TRUNCATE = 1000000

View solution in original post

0 Karma

Cuyose
Builder

These pushed out to the search head and forwarders and still doesn't work

0 Karma

woodcock
Esteemed Legend

Do you have a working solution (this is clicked Accepted)?

0 Karma

somesoni2
Revered Legend

As mentioned it should be in indexer or Heavy forwarder. A restart of SPlunk is required after you push the change.

0 Karma

hardikJsheth
Motivator

Refer to following link. Most of the parameters related to line breaking are required on Indexer and not on forwarder.
https://wiki.splunk.com/Community:HowIndexingWorks

0 Karma

hardikJsheth
Motivator

Try setting BREAK_ONLY_BEFORE_DATE = false in your props.conf.

0 Karma

Cuyose
Builder

still doesnt work
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE = ----------
BREAK_ONLY_BEFORE_DATE = false
CHARSET = UTF-8
DATETIME_CONFIG =
HEADER_MODE =
KVMODE =
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
LOOKUP-dropdowns = dropdownsLookup host OUTPUT unix_category unix_group
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 10000
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 1000000
category = Custom
detect_trailing_nulls = false
disabled = false
maxDist = 100
priority =
pulldown_type = true
sourcetype =

0 Karma

somesoni2
Revered Legend

Sample events please.

0 Karma

Cuyose
Builder

here is a sample. it will break on the context create time and sometimes the log time, sometimes the date=, but never the ------ explicitly declared

----------
Date=2016-06-20 15:54:20Z
LogLevel=INFO
Logger=XXXXXX.ContactCenter.NGAT.Web.Infrastructure.MultiCacheProviderFactory
Thread=161
LogContext=UIGeneralLog
Message=MultiCacheProvider: Operation succeeded.
MachineName=XXXXXXXX
MethodName=MultiCacheProviderOperationCompletedHandler
SourcePath=E:\jenkins\workspace\XXXXX\web\Infrastructure\MultiCacheProviderFactory.cs
SourceLine=218
ContextCreateTime=2016-06-20 15:54:20Z
LogTime=2016-06-20 15:54:20Z
SessionId=
InteractionIdStringId=f809f344-d728-4d12-9301-889ba406d86f
CallSessionIdStringId=8c5109ad-2ed1-4ed3-8e33-de5c34be4625
ActivityIdStringId=9d49138f-e264-4799-9279-b3c616bf9927
AgentUserName=
UserAgent=Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/7.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+.NET4.0C;+.NET4.0E;+InfoPath.3)
RequestPath=https://XXXXX/
Referrer=https://XXXXXX/?wa=wsignin1.0&wtrealm=https:%2f%XXXXXX%2f&wctx=rm%3d0%26id%3dpassive%26ru%3d%252f&wct=2016-06-20T15:54:20Z
SessionIsNew=
SessionKeys=
Controller=
Action=
MC.RequestId=98052227
MC.Operation=AddOrGetExisting
MC.Result=Success
MC.TotalTimeMillis=15.5998
MC.CacheKey=SessionSecurityToken-/;urn:uuid:cc6794c2-65b8-46f4-a36a-eb015d92a50c;
MC.CacheHit=False
0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.