Getting Data In

Why is line breaking not occurring as specified in props.conf?

feng_zhang
New Member

Hi Guys

I have an issue with line breaking. I used data preview in Splunk Web and it breaks line as what I wanted. But it doesn't do the trick when it deploys to props.conf in heavy forwarder.

The props.conf is

SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
BREAK_ONLY_BEFORE = \[requestID
BREAK_ONLY_BEFORE_DATE = false
CHARSET = UTF-8
MAX_TIMESTAMP_LOOKAHEAD = 12
disabled = false
TIME_FORMAT = %H:%M:%S,%3Q
TIME_PREFIX = X-Forwarded-For\=([^\.]+\.){3}\d{1,3}\]\s
#TZ_ALIAS = EST=AEST
DATETIME_CONFIG =
MAX_EVENTS = 20
pulldown_type = true
category = Application

The log sample lists below.

[requestID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
UA=Apache-HttpClient/android/SM-G900I
rcid=NA
referer=NA
node.no=1
SESSIONID=-xxxxxxxxxx
REMOTEADDRESS=xxx.xxx.xxx.xxx
X-Forwarded-For=xx.xxx.xx.xx] 15:05:31,599 DEBUG utilities.MiddlewareUtils - returning content-type = text/xml
[requestID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
UA=Apache-HttpClient/android/SM-G900I
rcid=NA
referer=NA
node.no=1
SESSIONID=-1067442995
REMOTEADDRESS=168.xxx.xxx.40
X-Forwarded-For=49.xxx.xx.73] 15:05:31,599 DEBUG connectors.ConnectorUtils - isNtlm Authentication Mode false

line count stats in search head.

Top 10 Values   Count   %    
1               520,045  99.516%    
8               2,086   0.399%  
257           328     0.063%    
9               28     0.005%   
5               23     0.004%   
4               8       0.002%  
2               6       0.001%  
255           6     0.001%  
199           5     0.001%  
177           3     0%

I used btool to check props.conf. This is what it shows.

ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE = \[requestID
BREAK_ONLY_BEFORE_DATE = false
CHARSET = UTF-8
DATETIME_CONFIG =
HEADER_MODE =
LEARN_MODEL = true
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 20
MAX_TIMESTAMP_LOOKAHEAD = 12
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = true
TIME_FORMAT = %H:%M:%S,%3Q
TIME_PREFIX = X-Forwarded-For\=([^\.]+\.){3}\d{1,3}\]\s
TRANSFORMS =
TRUNCATE = 10000
category = Application
detect_trailing_nulls = false
disabled = false
maxDist = 100
priority =
pulldown_type = true
sourcetype =

I tried to change SHOULD_LINEMERGE from true to false. Splunk doesn't count each line as individual events and still parses the log in the same way. I also tried to change the sourcetype into new one. It is still the same. Please help.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Id also look at updating the following:

BREAK_ONLY_BEFORE =^\[requestID
MAX_TIMESTAMP_LOOKAHEAD=300

Add the '^' to the breakonly before to signify its the start of a new line..

You should increase the MAX_TIMESTAMP_LOOKAHEAD to the number of characters in your event where the time stamp is at. This being set to 12 may have an adverse effect since your timestamp is much further down in your event.

0 Karma

feng_zhang
New Member

Hi esix

I tried what you suggested. It seems like it doesn't affect how line is breaking. I tried to remove the stanza in props.conf. Line breaking keeps the same way. Then I tried to move the stanze from heavy forwarder to indexer. It is still the same. I am running in circle now.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Did you have linemerge true on this?

Can you a paste from your log files to http://pasted.co/ or similar site and share. I can test on this side and see.

0 Karma

feng_zhang
New Member

yes, I tried with linemerge as true. I tried log in data preview in one of the search head. It works fine. Line breaking perfectedly. But not when I applied to props.conf in Heavy forwarder or Indexer. I will try to sanitize the log and share the log

0 Karma

mpreddy
Communicator

add these 2 properties and give a try:

LINE_BREAKER= ([\r\n]+)
MAX_EVENTS =
* Specifies the maximum number of input lines to add to any event.
* Splunk breaks after the specified number of lines are read.
* Defaults to 256 (lines).

0 Karma

feng_zhang
New Member

Then I need to change SHOULD_LINEMERGE = false. Otherwise Line_breaker won't be working. Am I right?

0 Karma

feng_zhang
New Member

Then I need to change SHOULD_LINEMERGE = false. Otherwise Line_breaker won't be working. Am I right?

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

You are correct. With SHOULD_LINEMERGE=false, you can use line_breaker.

https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Propsconf

And here is a good doc for using linemerge=false

http://docs.splunk.com/Documentation/Splunk/latest/Data/indexmulti-lineevents

0 Karma

feng_zhang
New Member

I tried what mpreddy suggested. It still shows the same result. Not breaking line correctly.
Top 10 Values Count %

1 92,915 99.78%

8 149 0.16%

257 42 0.045%

255 2 0.002%

5 2 0.002%

9 2 0.002%

179 1 0.001%

183 1 0.001%

185 1 0.001%

3 1 0.001%

0 Karma
Get Updates on the Splunk Community!

Set Up More Secure Configurations in Splunk Enterprise With Config Assist

This blog post is part 3 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...