topic Re: How do i stop a file from being segmented? in Getting Data In

How do i stop a file from being segmented?

MikeBertelsen — Tue, 29 Sep 2020 06:59:16 GMT

This is the beginning of the file, line numbers for clarity:
1. Log File for: BatchJobOutput_20150801-0139_13516_MonthlyBatchJob_SAMM191.log
2. Started: Sat Aug 1 01:39:22 CDT 2015
3. Using path to access.properties: /opt/WebSphere/AppServer/lib/app
4. --------------------------
5. /usr/java64/jdk1.6.0_43/bin:/bin:/usr/bin:/opt/gnome/bin:/usr/X11/bin:/home/cd7543/scripts

This is the end of the file:
159. Ended: Sat Aug 1 01:40:55 CDT 2015
160.

There are many date references between these two sections and Splunk takes this one file and splits it up which is then displayed to the end user segmented, in reverse order.

How do I get Splunk to index this as one contiguous file?

Re: How do i stop a file from being segmented?

woodcock — Wed, 12 Aug 2015 21:51:02 GMT

Add this to your props.conf:

TIME_PREFIX = ^Started:\s+

This won't work if the other timestamps start with the same RegEx.

Re: How do i stop a file from being segmented?

MikeBertelsen — Thu, 13 Aug 2015 15:16:22 GMT

That helps a lot. the file is now broken up into two segments. the first line and the second which contains the timestamp.
I can make do with this but if needed how do I get the previous line included?

Re: How do i stop a file from being segmented?

somesoni2 — Thu, 13 Aug 2015 15:37:19 GMT

You want all 160 lines to appear into one big single event with timestamp being taken from line 2?

Re: How do i stop a file from being segmented?

MikeBertelsen — Thu, 13 Aug 2015 15:42:15 GMT

Yes that is correct.

Re: How do i stop a file from being segmented?

somesoni2 — Thu, 13 Aug 2015 15:58:14 GMT

Try something like this for your props.conf

[YourSourceType]
BREAK_ONLY_BEFORE=^Log\s*File\s*for:
MAX_TIMESTAMP_LOOKAHEAD=150
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_PREFIX=Started:\s*

Re: How do i stop a file from being segmented?

woodcock — Thu, 13 Aug 2015 16:12:45 GMT

Add these, too:

 BREAK_ONLY_BEFORE_DATE = false
 BREAK_ONLY_BEFORE = ^Log File for:\s+
 SHOULD_LINEMERGE = true

Re: How do i stop a file from being segmented?

MikeBertelsen — Thu, 13 Aug 2015 17:29:00 GMT

Of the solutions offered, this one works the best.
But I am exceeding the 10000 byte limit and the file is being truncated.
Suggestions?

Re: How do i stop a file from being segmented?

somesoni2 — Thu, 13 Aug 2015 17:50:22 GMT

10000 bytes limits in the preview screen? If you see data truncated in Preview, don't worry actual data will not get truncated.
If you see the actual data getting truncated, add following to props.conf stanza.

TRUNCATE = 0

Re: How do i stop a file from being segmented?

MikeBertelsen — Thu, 13 Aug 2015 17:52:17 GMT

yes in the preview screen

Re: How do i stop a file from being segmented?

martin_mueller — Tue, 18 Aug 2015 19:11:32 GMT

Do not set settings such as TRUNCATE to zero, ever.

Pick a value that seems outlandishly large, sure - but not zero. If something breaks at your source, Splunk will attempt to build infinitely large events...