topic Re: json -> event -> Truncate=0 in Getting Data In

json -> event -> Truncate=0

lpolo — Thu, 27 Sep 2012 15:36:13 GMT

I have a sourcetype that the events are in json format. Each json event could be more the 2000 lines. I have the following configuration in props.conf:

[sourcetype_name]
LINE_BREAKER = ([\n]+).*query
SHOULD_LINEMERGE = true
TRUNCATE = 0

It works fine if the number of lines is less than 500. In events where the number of lines is more than ~500, the event is not indexed completed.

what else can i do in order to fix this?

Regards,
Lp

what can I do to tune up this up?

Re: json -> event -> Truncate=0

RicoSuave — Thu, 27 Sep 2012 16:47:33 GMT

And how exactly have you proven that the events are not getting indexed? If you are basing this solely off of what the event viewer is showing you, this is expected behavior. By default the number of lines per event that it's going to show you is limited to 500 lines. This is to prevent your browser from crashing due to increased memory usage. This is adjustable by editing a parameter for the events viewer module for your particular view. You can also edit the default flashtimeline view, though i recommend creating a new view solely for this particular sourcetype.

http://docs.splunk.com/Documentation/Splunk/latest/Developer/ModuleReference#EventsViewer

Take a look at the maxlinesconstraint parameter, but be warned that adjusting this too high, given your large events, might cause the browser to crash. I suggest revisiting how this json data is being generated and reduce its line count.

So you can edit your flashtimeline.xml to read like this:

<module name="EventsViewer" layoutPanel="resultsAreaLeft">
<param name="segmentation">full</param>
<param name=”maxLinesConstraint”>10000</param>
<param name="reportFieldLink">report_builder_format_report</param>
</module>

This has also been discussed here: http://splunk-base.splunk.com/answers/559/how-do-you-tweak-splunk-to-display-an-event-that-is-more-than-500-lines-long

Re: json -> event -> Truncate=0

gkanapathy — Thu, 27 Sep 2012 17:21:39 GMT

A few things:

Are you sure the event is getting truncated? Note the the UI will only display up to 500 lines. I don't know of a way to increase that. but if you perform other operations, or use the API or CLI, you may have the entire event.
I believe you should use SHOULD_LINEMERGE = false. I don't know what your data looks like, but if you leave that on, events get merged based on Splunk finding a timestamp in the line. This may get you the same results (depending on your data) but I don't know, and in any case is less efficient.
If you really do mean to merge lines using timestamps, then you would also want to increase MAX_EVENTS. Again, without knowing what your events look like, I can't say for certain.

Re: json -> event -> Truncate=0

lpolo — Mon, 28 Sep 2020 12:31:18 GMT

1) Yes I am sure. Also, I am aware of the UI settings in order to exceed the limit of lines per event in UI.
2) My regular expression should define the line breaker. LINE_BREAKER = ([\n]+).*query . Is this correct?
3) Truncate=0 should override MAX_EVENTS. Is this correct?

Any ideas?

Thanks
Lp

Re: json -> event -> Truncate=0

gkanapathy — Thu, 27 Sep 2012 17:40:13 GMT

Okay
1. I have no idea. It depends what your data looks like and and how it's formatted.
2. No. Both are in effect and if you are merging lines, then that matters.

Re: json -> event -> Truncate=0

kristian_kolb — Thu, 27 Sep 2012 17:41:15 GMT

Don't know about the possible UI limits, but depending on how you set SHOULD_LINEMERGE, you'll have different options;

These go together

SHOULD_LINEMERGE = true
MAX_EVENTS
MUST_BREAK...
BREAK_ONLY...

These also go togehter;

SHOULD_LINEMERGE = false
LINE_BREAKER
TRUNCATE

UPDATE:

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

will tell you more precisely how to set your config, but I do believe that you should have:

SHOULD_LINEMERGE=false
LINE_BREAKER = <regex>
TRUNCATE = 0

The key here is to craft the correct regex, and yours look a bit off. The regex should ideally match what comes first on the line in what you want to be a new event, e.g.

LINE_BREAKER = ([\r\n]+)\d{4}-\d\d-\d\d\s\d\d:\d\d:\d\d.\d\d\d\s\+\d{4}

would match (and create a break to a new event) for a newline followed by a timestamp formatted like; 2012-09-27 20:34:19.342 +0200

The newline character(s) would be stripped from the event, but not the (in this case) timestamp.

Hope this helps,

Kristian

Re: json -> event -> Truncate=0

lpolo — Thu, 27 Sep 2012 17:43:38 GMT

Thanks,

Let me try setting
SHOULD_LINEMERGE = false.

Re: json -> event -> Truncate=0

lpolo — Mon, 28 Sep 2020 12:31:23 GMT

it did not work with
SHOULD_LINEMERGE = false. I set it back to:

[sourcetype_name]
LINE_BREAKER = ([\n]+).*query
SHOULD_LINEMERGE = true
TRUNCATE = 0

Re: json -> event -> Truncate=0

RicoSuave — Thu, 27 Sep 2012 18:24:12 GMT

So have you adjusted this parameter for the eventsviewer module? Have you tried a CLI search and checked for events greater than 500 lines?

Re: json -> event -> Truncate=0

kristian_kolb — Thu, 27 Sep 2012 18:36:18 GMT

see update above /k

Re: json -> event -> Truncate=0

lpolo — Thu, 27 Sep 2012 19:13:20 GMT

I think that you are right. This regex looks that is sufficient:

LINE_BREAKER = ([n]+).*query
to detect the first line of the event:

2012-09-27 19:03:24,705

but seems that is not. Tomorrow morning I will try with this regex:

LINE_BREAKER = ([n]+)\d{4}-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d\s<query\sid=

then, set SHOULD_LINEMERGE to false.

Thanks Kristian

Re: json -> event -> Truncate=0

lpolo — Fri, 28 Sep 2012 12:01:41 GMT

Thanks Kristian,

your recommendation worked.