Multi Line Events "truncate"

[vitki]

Hi

I am testing the log length with sending about two pages of data only 1 character. Lets say "b" so the data will look like "bbbbbbbbbb" for 3 pages long.

The data with tcpdump to the indexer looks fine, but on the indexer the event gets split into 257 line events. I have change the etc/system/local/ props.conf and limits.conf, inserted MAX_EVENT = 1000, truncate = 100000 but it has no effect.

any ideas?

Here is the props.conf

[default]
CHARSET = UTF-8
LINE_BREAKER_LOOKBEHIND = 1000
TRUNCATE = 100000
DATETIME_CONFIG = /etc/datetime.xml
ANNOTATE_PUNCT = True
HEADER_MODE =
MAX_DAYS_HENCE=2
MAX_DAYS_AGO=2000
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = false
MAX_EVENTS = 5000
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
TRANSFORMS =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
LEARN_SOURCETYPE = true
maxDist = 100

[log4j]
pulldown_type = true
MAX_EVENTS = 7000
MAX_TIMESTAMP_LOOKAHEAD=25
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true

[vitki]

My Solution is the following, there might be some bad suggestions here but it worked for me.

mv limits.conf to $SPLUNK_HOME/etc/system/local
backup and remove following conf files in
$SPLUNK_HOME/etc/system/default

outputs.conf  pdf_server.conf  prefs.conf  props.conf
viewstates.conf

only the following conf files in $SPLUNK_HOME/etc/system/local
remove any other.

inputs.conf props.conf README server.conf tenants.conf

props.conf must look like this (clear file then copy info in)

[default]
CHARSET = UTF-8
LINE_BREAKER_LOOKBEHIND = 1000
TRUNCATE = 100000
DATETIME_CONFIG = /etc/datetime.xml
ANNOTATE_PUNCT = True
HEADER_MODE =
MAX_DAYS_HENCE=2
MAX_DAYS_AGO=2000
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = true
MAX_EVENTS = 5000
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
TRANSFORMS =
SEGMENTATION          = indexing
SEGMENTATION-all      = full
SEGMENTATION-inner    = inner
SEGMENTATION-outer    = outer
SEGMENTATION-raw      = none
SEGMENTATION-standard = standard
LEARN_SOURCETYPE      = false
maxDist = 100

[log4j]
#pulldown_type = true
MAX_EVENTS = 7000
#MAX_TIMESTAMP_LOOKAHEAD=25
SHOULD_LINEMERGE = true
TRUNCATE = 1000000

restart splunk

Like i said there might be some bad advice in the answer. I am still trying to find out what i did wrong in the first place... but this is working for me for now. thanks kristian.kolb

[vitki]

It was me It was me It was me.... my problem was that my access permissions on the local directory where my *.conf files was NOT accessable to the splunk user... so it could not read the files...

Just goes to show ... Don't assume - Make sure !!!

[vitki]

Yip, i am trying to figure out what went wrong, if i do not delete the files in the default directory i am back to where i started.

I have now setup a new instillation and i am testing it with that now. I agree 100% the default did needs to stay as is. will post my new answer.

[kristian_kolb]

Yes, unfortunately there are some aspects of your solution that are less than perfect.

Never add/remove/edit files in a 'default' directory. That's what the 'local' directories are there for. Any setting in a .conf file in a 'local' directory will override the corresponding 'default'.

Your solution will perhaps work for now, but you will run into trouble when you upgrade to the next version (all default .conf files will be restored) or whenever Splunk needs to know a setting - and it's just not there anymore.

Proceed with caution,

Kristian

[vitki]

Hay i just saw that all the mulit events is 1024 characters long.... and the show source data is fine with no multi events.

[vitki]

Mmmm i have made the change as requested and BOOM!!! nothing 😞

I am still getting the same result = munti events... if i send the data via netcat to the splunkforwarder i get about 12 events with only limited bogus data in the events on the indexer. If i send it directly to the indexer i get the same results. if i netcat it to the forwarder with no dates i get the 381 events again.

[kristian_kolb]

I saw that as well. If that was from $SPLUNK_HOME/etc/system/local/props.conf, then that should be ok.

However, I advise you to;
a) create better sample events starting with at timestamp
2012-09-12 13:14:15 bbbbbbbbbbbbb * 1 million if you like.

b) use SHOULD_LINEMERGE = false if your events are single-line. This also means that you do not set BREAK_ONLY... and MAX_EVENTS stuff.

c) set TRUNCATE to a sufficiently high value to accomodate your looooong events.

[your_sourcetype]
SHOULD_LINEMERGE = false
TRUNCATE = 1000000

[vitki]

MAX_EVENTS = 7000 in my conf file am i missing another place where this must go? well i have inserted a date at the start of my data and the end of it. now i get a 381 event and then 227 then a 226 event and last the date in a event.

[kristian_kolb]

so you're probably hitting the MAX_EVENTS limit. I believe that it's set at 256 by default.

I think that one of the problems is that you're sending bogus data, i.e. it does not have any timestamps. Generally, Splunk will not have trouble dealing with large events.

[vitki]

I get more than 1 event with each 257 lines till the last event with maybe 129 lines or so.

[kristian_kolb]

well done. I was just about to edit your "answer" to make it part of the original question.

Also, do you get one (or more) events, each 257 lines long, or did you get 257 one-line events?

[kristian_kolb]

Is that 257 one-line events, or several 257-line events?

are there any line-breaks in the source data?

What are your line-breaking rules (SHOULD_LINEMERGE, LINE_BREAKER, MUST_BREAK..., BREAK_ONLY.. etc)?