Getting Data In

Event line breaker to index multi-line events into single event

ssamant007
Explorer

My current log monitoring splunk forwarder is indexing events in group (like sometimes more than 1 events together) but I wanted to have each event (which is own datetime at the start) to be indexed separately. Only the starting of event is same for each line (event) and rest of the string varies. I tried configuring the props.conf file using the following formats:

LINE_BREAKER = ([\r\n]+) (though its by default but seems not working as my events are separated by newline or \r in the source log file)

and then I tried as below:

BREAK_ONLY_BEFORE = ^\d+\s*$

 Currently it is being indexed as shown below:

ssamant007_0-1638311294280.png

However, I wanted to have each entry indexed as a separate event. 

Entries in source file (example)

2021-Dec-01 Wed 08:50:06.914 INFO [Thread-3] - org.eclipse.jetty.server.session - {} - doStart(DefaultSessionIdManager.java:334) - DefaultSessionIdManager workerName=node0
2021-Dec-01 Wed 08:50:06.915 INFO [Thread-3] - org.eclipse.jetty.server.session - {} - doStart(DefaultSessionIdManager.java:339) - No SessionScavenger set, using defaults
2021-Dec-01 Wed 08:50:06.917 INFO [Thread-3] - org.eclipse.jetty.server.session - {} - startScavenging(HouseKeeper.java:132) - node0 Scavenging every 660000ms
2021-Dec-01 Wed 08:50:06.956 INFO [Thread-3] - org.eclipse.jetty.server.AbstractConnector - {} - doStart(AbstractConnector.java:331) - Started ServerConnector@5e283ab9{HTTP/1.1, (http/1.1)}{127.0.0.1:22113}
2021-Dec-01 Wed 08:50:06.956 INFO [Thread-3] - org.eclipse.jetty.server.Server - {} - doStart(Server.java:415) - Started @6850ms
2021-Dec-01 Wed 08:50:24.331 INFO [pool-6-thread-1] - com.automationanywhere.nodemanager.service.impl.WindowsEventServiceImpl - {} - onMachineLogon(WindowsEventServiceImpl.java:226) - Machine Logon: 1
2021-Dec-01 Wed 08:58:35.372 INFO [pool-6-thread-1] - com.automationanywhere.nodemanager.service.impl.WindowsEventServiceImpl - {} - onMachineLocked(WindowsEventServiceImpl.java:204) - Machine Locked: 1
2021-Dec-01 Wed 09:17:38.934 INFO [pool-6-thread-1] - com.automationanywhere.nodemanager.service.impl.WindowsEventServiceImpl - {} - onMachineUnlocked(WindowsEventServiceImpl.java:214) - Machine Unlocked: 1
2021-Dec-01 Wed 09:17:38.937 INFO [pool-6-thread-1] - com.automationanywhere.nodemanager.service.impl.WindowsEventServiceImpl - {} - onMachineUnlocked(WindowsEventServiceImpl.java:216) - Session id 1 removed from tracking on machine unlock.

I  would appreciate any help in configuring the props.conf file to index events  as a single entry.

TIA.

0 Karma

ssamant007
Explorer

apparently, it worked after selecting the sourcetype as CSV.

0 Karma

PickleRick
Ultra Champion

Well, if your line breaker is indeed the default ([\r\n]+), then there must be something wrong with your log because both a single \r or a single \n or any combination of those two characters constitutes a linebreak.

0 Karma

ssamant007
Explorer

Yes, technically it should work but upon checking the end of line character in the log file it shows CRLF character for each line.

ssamant007_0-1638312693026.png

You can see in the image that  EOL character in log file entries has \r\n for each line.

0 Karma

PickleRick
Ultra Champion
0 Karma

ssamant007
Explorer

Hi yes, I have gone through the documentation as well, and I have configured the props.conf file inside the $splunk_home$\etc\system\local\ as follows:

[mysource-type]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
DATETIME_CONFIG = CURRENT

..and the default values of props.conf file in the ..\system\default\ folder are as follows:

[default]
CHARSET = AUTO
LINE_BREAKER_LOOKBEHIND = 100
TRUNCATE = 10000
LB_CHUNK_BREAKER_TRUNCATE = 2000000
DATETIME_CONFIG = \etc\datetime.xml
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
HEADER_MODE =
MATCH_LIMIT = 100000
DEPTH_LIMIT = 1000
MAX_DAYS_HENCE=2
MAX_DAYS_AGO=2000
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_TIMESTAMP_LOOKAHEAD = 128
DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
TRANSFORMS =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
LEARN_SOURCETYPE = true
LEARN_MODEL = true
termFrequencyWeightedDist = false
maxDist = 100
AUTO_KV_JSON = true
detect_trailing_nulls = auto
sourcetype =
priority =

Do you think I missed on the some other configurations?? I double check that in the source log file each line are separated with CRLF charatcter... 

 

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...