topic Re: stop splunk from splitting a file. in Getting Data In

stop splunk from splitting a file.

jstillwell — Sat, 04 Sep 2010 16:06:01 GMT

I'm trying to read some config files into splunk, ala change management. I'm not using fschange, I'm using a tcp monitor, and pushing them into splunk from different hosts.

I use a ***SPLUNK*** header to set source, sourcetype, host properly. As that info isn't in the files anywhere. Those values get picked up fine.

Some of the files are left as one huge event, as they should be. Other files are getting split by line, which is what I don't want. How can I tell splunk I don't want a sourcetype split up at all. One tcp connection = one event.

I've tried various combinations of things.

[database_config]
LINE_BREAKER = disable_linebreaking_with_impossible_regex
BREAK_ONLY_BEFORE = disable_linebreaking_with_impossible_regex
LEARN_MODEL = false
SHOULD_LINEMERGE = false

addition info.

I tried rsyncing the files over and using fschange (for the fullEvent parameter), but that was worse, I got the same line breaking results, but couldn't override the sourcetype using conf files. And was left with 20 random ones splunk invented, for the same file type. I tried a large volume of different configurations on multiple different OSes.

btool reports nothing is overriding my sourcetype definition.

I don't trust what splunk test sourcetype says. When I give it a file it ends up creating stuff in the 'learned' app, which doesn't happen when I import via tcp.

splunk is doing splitting fine for the logs I push into it.

Re: stop splunk from splitting a file.

gkanapathy — Sat, 04 Sep 2010 23:18:00 GMT

You need to increase/set TRUNCATE and MAX_EVENTS to be large enough to fit your entire file.

Also, a regex that will be guaranteed to never match is (?!) that is open-paren, question-mark, exclamation-mark, close-paren. An equivalent is (*FAIL)

Re: stop splunk from splitting a file.

jstillwell — Sun, 05 Sep 2010 03:17:38 GMT

This doesn't work. I suspect splunk is ignoring my sourcetype stanza in props.conf. I don't know how to verify this.

Re: stop splunk from splitting a file.

gkanapathy — Sun, 05 Sep 2010 07:20:33 GMT

Can you please clarify, are you using Splunk forwarders to monitor files, or are you sending data in to a plain TCP input port? Or really, what do you mean by "using a tcp monitor"?

Re: stop splunk from splitting a file.

gkanapathy — Sun, 05 Sep 2010 07:20:51 GMT

And if using forwarders, are these light or heavy forwarders?

Re: stop splunk from splitting a file.

jstillwell — Sun, 05 Sep 2010 13:39:01 GMT

plain TCP input port. no forwarders anywhere. I just have a [tcp://:port] stanza in inputs.conf, and homebrew scripts that connect and send the data into that port.