topic Re: How to set the BREAK_ONLY_BEFORE? in Splunk Search

How to set the BREAK_ONLY_BEFORE?

neha22 — Tue, 05 Apr 2022 16:42:02 GMT

I am not sure of how to set the BREAK_ONLY_BEFORE I have tried the below setting.. all my logs are of log4j format and starts at [2022-04-05 11:18:23,839] format

BREAK_ONLY_BEFORE: date

My logs are which are send to splunk through fluentd in as different events:

[2022-04-05 11:18:23,839] WARN Error while loading: connectors-versions.properties (com.amadeus.scp.kafka.connect.utils.Version)
java.lang.NullPointerException
    at java.util.Properties$LineReader.readLine(Properties.java:434)
    at java.util.Properties.load0(Properties.java:353)
    at java.util.Properties.load(Properties.java:341)
    at com.amadeus.scp.kafka.connect.utils.Version.<clinit>(Version.java:47)
    at com.amadeus.scp.kafka.connect.connectors.kafka.source.router.K2KRouterSourceConnector.version(K2KRouterSourceConnector.java:62)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:380)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:385)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:355)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:328)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:261)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:253)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:222)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:199)
    at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:60)
    at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)

Re: Handling multiline formats in splunk

gcusello — Tue, 05 Apr 2022 14:10:40 GMT

Hi @neha22,

I'd define the Timestamp format and position and I'd use it for event breaking,something like this:

[your_sourcetype] TIME_PREFIX = ^\[ TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N SHOULD_LINEMERGE = True

Ciao.

Giuseppe

Re: Handling multiline formats in splunk

neha22 — Tue, 05 Apr 2022 15:49:51 GMT


TIME_PREFIX:  "^[",
TIME_FORMAT: "([%Y-%m-%d %H:%M:%S,%3N]+)",

i tried as above but the logs still displayed as different events not a single one.

Re: How to set the BREAK_ONLY_BEFORE?

VatsalJagani — Wed, 06 Apr 2022 06:37:05 GMT

@neha22 - I wouldn't suggest SHOULD_LINEMERGE=true as it is not performance effective. Try the below configuration with LINE_ BREAKER.

Just FYI, this will only apply to new events coming into Splunk, not to existing events.

Put this configuration on forwarders and indexers level.

LINE_BREAKER = ([\n\r]+)\[\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2},\d+\]\s+ SHOULD_LINEMERGE = false TIME_PREFIX = ^[ TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N MAX_TIMESTAMP_LOOKAHEAD = 25

I hope this helps!! Consider upvoting if it does!!!

Re: How to set the BREAK_ONLY_BEFORE?

neha22 — Wed, 06 Apr 2022 10:59:26 GMT

I am using the HEC method

Re: How to set the BREAK_ONLY_BEFORE?

VatsalJagani — Wed, 06 Apr 2022 12:05:59 GMT

If you are using the HEC endpoint (/services/collector/event ) then Parsing (LINE_BREAKING), Merging (SHOULD_LINEMERGE, timestamp extraction) will not work.

Use HEC this HEC endpoint instead - /services/collector/raw

Reference - https://docs.splunk.com/Documentation/Splunk/8.2.5/Data/HECRESTendpoints