Getting Data In

Multi Line Events "truncate"

vitki
Explorer

Hi

I am testing the log length with sending about two pages of data only 1 character. Lets say "b" so the data will look like "bbbbbbbbbb" for 3 pages long.

The data with tcpdump to the indexer looks fine, but on the indexer the event gets split into 257 line events. I have change the etc/system/local/ props.conf and limits.conf, inserted MAX_EVENT = 1000, truncate = 100000 but it has no effect.

any ideas?

Here is the props.conf

[default]
CHARSET = UTF-8
LINE_BREAKER_LOOKBEHIND = 1000
TRUNCATE = 100000
DATETIME_CONFIG = /etc/datetime.xml
ANNOTATE_PUNCT = True
HEADER_MODE =
MAX_DAYS_HENCE=2
MAX_DAYS_AGO=2000
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = false
MAX_EVENTS = 5000
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
TRANSFORMS =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
LEARN_SOURCETYPE = true
maxDist = 100

[log4j]
pulldown_type = true
MAX_EVENTS = 7000
MAX_TIMESTAMP_LOOKAHEAD=25
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true

0 Karma

vitki
Explorer

My Solution is the following, there might be some bad suggestions here but it worked for me.

mv limits.conf to $SPLUNK_HOME/etc/system/local
backup and remove following conf files in
$SPLUNK_HOME/etc/system/default

outputs.conf  pdf_server.conf  prefs.conf  props.conf
viewstates.conf

only the following conf files in $SPLUNK_HOME/etc/system/local
remove any other.

inputs.conf props.conf README server.conf tenants.conf

props.conf must look like this (clear file then copy info in)

[default]
CHARSET = UTF-8
LINE_BREAKER_LOOKBEHIND = 1000
TRUNCATE = 100000
DATETIME_CONFIG = /etc/datetime.xml
ANNOTATE_PUNCT = True
HEADER_MODE =
MAX_DAYS_HENCE=2
MAX_DAYS_AGO=2000
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = true
MAX_EVENTS = 5000
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
TRANSFORMS =
SEGMENTATION          = indexing
SEGMENTATION-all      = full
SEGMENTATION-inner    = inner
SEGMENTATION-outer    = outer
SEGMENTATION-raw      = none
SEGMENTATION-standard = standard
LEARN_SOURCETYPE      = false
maxDist = 100

[log4j]
#pulldown_type = true
MAX_EVENTS = 7000
#MAX_TIMESTAMP_LOOKAHEAD=25
SHOULD_LINEMERGE = true
TRUNCATE = 1000000

restart splunk

Like i said there might be some bad advice in the answer. I am still trying to find out what i did wrong in the first place... but this is working for me for now. thanks kristian.kolb

0 Karma

vitki
Explorer

It was me It was me It was me.... my problem was that my access permissions on the local directory where my *.conf files was NOT accessable to the splunk user... so it could not read the files...

Just goes to show ... Don't assume - Make sure !!!

0 Karma

vitki
Explorer

Yip, i am trying to figure out what went wrong, if i do not delete the files in the default directory i am back to where i started.

I have now setup a new instillation and i am testing it with that now. I agree 100% the default did needs to stay as is. will post my new answer.

0 Karma

kristian_kolb
Ultra Champion

Yes, unfortunately there are some aspects of your solution that are less than perfect.

Never add/remove/edit files in a 'default' directory. That's what the 'local' directories are there for. Any setting in a .conf file in a 'local' directory will override the corresponding 'default'.

Your solution will perhaps work for now, but you will run into trouble when you upgrade to the next version (all default .conf files will be restored) or whenever Splunk needs to know a setting - and it's just not there anymore.

Proceed with caution,

Kristian

0 Karma

vitki
Explorer

Hay i just saw that all the mulit events is 1024 characters long.... and the show source data is fine with no multi events.

0 Karma

vitki
Explorer

Mmmm i have made the change as requested and BOOM!!! nothing 😞

I am still getting the same result = munti events... if i send the data via netcat to the splunkforwarder i get about 12 events with only limited bogus data in the events on the indexer. If i send it directly to the indexer i get the same results. if i netcat it to the forwarder with no dates i get the 381 events again.

0 Karma

kristian_kolb
Ultra Champion

I saw that as well. If that was from $SPLUNK_HOME/etc/system/local/props.conf, then that should be ok.

However, I advise you to;
a) create better sample events starting with at timestamp
2012-09-12 13:14:15 bbbbbbbbbbbbb * 1 million if you like.

b) use SHOULD_LINEMERGE = false if your events are single-line. This also means that you do not set BREAK_ONLY... and MAX_EVENTS stuff.

c) set TRUNCATE to a sufficiently high value to accomodate your looooong events.

[your_sourcetype]
SHOULD_LINEMERGE = false
TRUNCATE = 1000000

vitki
Explorer

MAX_EVENTS = 7000 in my conf file am i missing another place where this must go? well i have inserted a date at the start of my data and the end of it. now i get a 381 event and then 227 then a 226 event and last the date in a event.

0 Karma

kristian_kolb
Ultra Champion

so you're probably hitting the MAX_EVENTS limit. I believe that it's set at 256 by default.

I think that one of the problems is that you're sending bogus data, i.e. it does not have any timestamps. Generally, Splunk will not have trouble dealing with large events.

0 Karma

vitki
Explorer

I get more than 1 event with each 257 lines till the last event with maybe 129 lines or so.

0 Karma

kristian_kolb
Ultra Champion

well done. I was just about to edit your "answer" to make it part of the original question.

Also, do you get one (or more) events, each 257 lines long, or did you get 257 one-line events?

0 Karma

kristian_kolb
Ultra Champion

Is that 257 one-line events, or several 257-line events?

are there any line-breaks in the source data?

What are your line-breaking rules (SHOULD_LINEMERGE, LINE_BREAKER, MUST_BREAK..., BREAK_ONLY.. etc)?

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...