Getting Data In

stop splunk from splitting a file.

jstillwell
Explorer

I'm trying to read some config files into splunk, ala change management. I'm not using fschange, I'm using a tcp monitor, and pushing them into splunk from different hosts.

I use a ***SPLUNK*** header to set source, sourcetype, host properly. As that info isn't in the files anywhere. Those values get picked up fine.

Some of the files are left as one huge event, as they should be. Other files are getting split by line, which is what I don't want. How can I tell splunk I don't want a sourcetype split up at all. One tcp connection = one event.

I've tried various combinations of things.

[database_config]
LINE_BREAKER = disable_linebreaking_with_impossible_regex
BREAK_ONLY_BEFORE = disable_linebreaking_with_impossible_regex
LEARN_MODEL = false
SHOULD_LINEMERGE = false

addition info.

I tried rsyncing the files over and using fschange (for the fullEvent parameter), but that was worse, I got the same line breaking results, but couldn't override the sourcetype using conf files. And was left with 20 random ones splunk invented, for the same file type. I tried a large volume of different configurations on multiple different OSes.

btool reports nothing is overriding my sourcetype definition.

I don't trust what splunk test sourcetype says. When I give it a file it ends up creating stuff in the 'learned' app, which doesn't happen when I import via tcp.

splunk is doing splitting fine for the logs I push into it.

Tags (3)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

You need to increase/set TRUNCATE and MAX_EVENTS to be large enough to fit your entire file.

Also, a regex that will be guaranteed to never match is (?!) that is open-paren, question-mark, exclamation-mark, close-paren. An equivalent is (*FAIL)

jstillwell
Explorer

plain TCP input port. no forwarders anywhere. I just have a [tcp://:port] stanza in inputs.conf, and homebrew scripts that connect and send the data into that port.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

And if using forwarders, are these light or heavy forwarders?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Can you please clarify, are you using Splunk forwarders to monitor files, or are you sending data in to a plain TCP input port? Or really, what do you mean by "using a tcp monitor"?

0 Karma

jstillwell
Explorer

This doesn't work. I suspect splunk is ignoring my sourcetype stanza in props.conf. I don't know how to verify this.

0 Karma
Get Updates on the Splunk Community!

Splunk MCP & Agentic AI: Machine Data Without Limits

  Discover how the Splunk Model Context Protocol (MCP) Server can revolutionize the way your organization ...

Finding Based Detections General Availability

Overview  We’ve come a long way, folks, but here in Enterprise Security 8.4 I’m happy to announce Finding ...

Get Your Hands Dirty (and Your Shoes Comfy): The Splunk Experience

Hands-On Learning and Technical Seminars  Sometimes, you just need to see the code. For those looking for a ...