topic Re: Event line breaker to index multi-line events into single event in Getting Data In

Event line breaker to index multi-line events into single event

ssamant007 — Tue, 30 Nov 2021 22:35:24 GMT

My current log monitoring splunk forwarder is indexing events in group (like sometimes more than 1 events together) but I wanted to have each event (which is own datetime at the start) to be indexed separately. Only the starting of event is same for each line (event) and rest of the string varies. I tried configuring the props.conf file using the following formats:

LINE_BREAKER = ([\r\n]+) (though its by default but seems not working as my events are separated by newline or \r in the source log file)

and then I tried as below:

BREAK_ONLY_BEFORE = ^\d+\s*$

Currently it is being indexed as shown below:

However, I wanted to have each entry indexed as a separate event.

Entries in source file (example)

2021-Dec-01 Wed 08:50:06.914 INFO [Thread-3] - org.eclipse.jetty.server.session - {} - doStart(DefaultSessionIdManager.java:334) - DefaultSessionIdManager workerName=node0
2021-Dec-01 Wed 08:50:06.915 INFO [Thread-3] - org.eclipse.jetty.server.session - {} - doStart(DefaultSessionIdManager.java:339) - No SessionScavenger set, using defaults
2021-Dec-01 Wed 08:50:06.917 INFO [Thread-3] - org.eclipse.jetty.server.session - {} - startScavenging(HouseKeeper.java:132) - node0 Scavenging every 660000ms
2021-Dec-01 Wed 08:50:06.956 INFO [Thread-3] - org.eclipse.jetty.server.AbstractConnector - {} - doStart(AbstractConnector.java:331) - Started ServerConnector@5e283ab9{HTTP/1.1, (http/1.1)}{127.0.0.1:22113}
2021-Dec-01 Wed 08:50:06.956 INFO [Thread-3] - org.eclipse.jetty.server.Server - {} - doStart(Server.java:415) - Started @6850ms
2021-Dec-01 Wed 08:50:24.331 INFO [pool-6-thread-1] - com.automationanywhere.nodemanager.service.impl.WindowsEventServiceImpl - {} - onMachineLogon(WindowsEventServiceImpl.java:226) - Machine Logon: 1
2021-Dec-01 Wed 08:58:35.372 INFO [pool-6-thread-1] - com.automationanywhere.nodemanager.service.impl.WindowsEventServiceImpl - {} - onMachineLocked(WindowsEventServiceImpl.java:204) - Machine Locked: 1
2021-Dec-01 Wed 09:17:38.934 INFO [pool-6-thread-1] - com.automationanywhere.nodemanager.service.impl.WindowsEventServiceImpl - {} - onMachineUnlocked(WindowsEventServiceImpl.java:214) - Machine Unlocked: 1
2021-Dec-01 Wed 09:17:38.937 INFO [pool-6-thread-1] - com.automationanywhere.nodemanager.service.impl.WindowsEventServiceImpl - {} - onMachineUnlocked(WindowsEventServiceImpl.java:216) - Session id 1 removed from tracking on machine unlock.

I would appreciate any help in configuring the props.conf file to index events as a single entry.

TIA.

Re: Event line breaker to index multi-line events into single event

PickleRick — Tue, 30 Nov 2021 22:42:18 GMT

Well, if your line breaker is indeed the default ([\r\n]+), then there must be something wrong with your log because both a single \r or a single \n or any combination of those two characters constitutes a linebreak.

Re: Event line breaker to index multi-line events into single event

ssamant007 — Tue, 30 Nov 2021 22:53:37 GMT

Yes, technically it should work but upon checking the end of line character in the log file it shows CRLF character for each line.

You can see in the image that EOL character in log file entries has \r\n for each line.

Re: Event line breaker to index multi-line events into single event

PickleRick — Wed, 01 Dec 2021 20:42:27 GMT

Do you have other settings affecting event breaking?

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Configureeventlinebreaking

Re: Event line breaker to index multi-line events into single event

ssamant007 — Wed, 01 Dec 2021 22:43:05 GMT

Hi yes, I have gone through the documentation as well, and I have configured the props.conf file inside the $splunk_home$\etc\system\local\ as follows:

[mysource-type]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
DATETIME_CONFIG = CURRENT

..and the default values of props.conf file in the ..\system\default\ folder are as follows:

[default]
CHARSET = AUTO
LINE_BREAKER_LOOKBEHIND = 100
TRUNCATE = 10000
LB_CHUNK_BREAKER_TRUNCATE = 2000000
DATETIME_CONFIG = \etc\datetime.xml
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
HEADER_MODE =
MATCH_LIMIT = 100000
DEPTH_LIMIT = 1000
MAX_DAYS_HENCE=2
MAX_DAYS_AGO=2000
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_TIMESTAMP_LOOKAHEAD = 128
DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
TRANSFORMS =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
LEARN_SOURCETYPE = true
LEARN_MODEL = true
termFrequencyWeightedDist = false
maxDist = 100
AUTO_KV_JSON = true
detect_trailing_nulls = auto
sourcetype =
priority =

Do you think I missed on the some other configurations?? I double check that in the source log file each line are separated with CRLF charatcter...

Re: Event line breaker to index multi-line events into single event

ssamant007 — Thu, 02 Dec 2021 08:27:26 GMT

apparently, it worked after selecting the sourcetype as CSV.