Getting Data In

stop splunk from splitting a file.

jstillwell
Explorer

I'm trying to read some config files into splunk, ala change management. I'm not using fschange, I'm using a tcp monitor, and pushing them into splunk from different hosts.

I use a ***SPLUNK*** header to set source, sourcetype, host properly. As that info isn't in the files anywhere. Those values get picked up fine.

Some of the files are left as one huge event, as they should be. Other files are getting split by line, which is what I don't want. How can I tell splunk I don't want a sourcetype split up at all. One tcp connection = one event.

I've tried various combinations of things.

[database_config]
LINE_BREAKER = disable_linebreaking_with_impossible_regex
BREAK_ONLY_BEFORE = disable_linebreaking_with_impossible_regex
LEARN_MODEL = false
SHOULD_LINEMERGE = false

addition info.

I tried rsyncing the files over and using fschange (for the fullEvent parameter), but that was worse, I got the same line breaking results, but couldn't override the sourcetype using conf files. And was left with 20 random ones splunk invented, for the same file type. I tried a large volume of different configurations on multiple different OSes.

btool reports nothing is overriding my sourcetype definition.

I don't trust what splunk test sourcetype says. When I give it a file it ends up creating stuff in the 'learned' app, which doesn't happen when I import via tcp.

splunk is doing splitting fine for the logs I push into it.

Tags (3)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

You need to increase/set TRUNCATE and MAX_EVENTS to be large enough to fit your entire file.

Also, a regex that will be guaranteed to never match is (?!) that is open-paren, question-mark, exclamation-mark, close-paren. An equivalent is (*FAIL)

jstillwell
Explorer

plain TCP input port. no forwarders anywhere. I just have a [tcp://:port] stanza in inputs.conf, and homebrew scripts that connect and send the data into that port.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

And if using forwarders, are these light or heavy forwarders?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Can you please clarify, are you using Splunk forwarders to monitor files, or are you sending data in to a plain TCP input port? Or really, what do you mean by "using a tcp monitor"?

0 Karma

jstillwell
Explorer

This doesn't work. I suspect splunk is ignoring my sourcetype stanza in props.conf. I don't know how to verify this.

0 Karma
Get Updates on the Splunk Community!

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...

Cloud Platform & Enterprise: Classic Dashboard Export Feature Deprecation

As of Splunk Cloud Platform 9.3.2408 and Splunk Enterprise 9.4, classic dashboard export features are now ...