topic Re: Entire file contents as a single event in Getting Data In

Entire file contents as a single event

keiche — Wed, 11 Aug 2010 03:04:00 GMT

I would like to know how to setup Splunk to monitor a local input directory, BUT the new files which are added (which contain multiple lines) are ingested by Splunk and only create 1 new event per file (containing all of the file's contents). I do have the ability to manipulate the file data to add line-breaks if that is the solution.

Re: Entire file contents as a single event

gkanapathy — Wed, 11 Aug 2010 04:20:39 GMT

Just set the LINE_BREAKER for the sourcetype to something that will never match, such as (?!). You will also probably also need to increase MAX_EVENTS (default is only 500 lines, there isn't a hard limit I know of) and TRUNCATE to something larger than the biggest file size (or I think 0 is unlimited).

Re: Entire file contents as a single event

lguinn2 — Wed, 11 Aug 2010 05:24:14 GMT

Do you want Splunk to create one event per file, or do you want it to create one event per line?

Re: Entire file contents as a single event

ftk — Mon, 28 Sep 2020 09:10:34 GMT

I use a regular monitor stanza combined with a custom sourcetype to index full files of interest.

I use the following monitor to index changes to my splunk configs for example (inputs.conf):

[monitor://C:\Program Files\Splunk\etc\...\*.conf]
followTail = False
sourcetype = splunk_config
index = my_custom_index
disabled = false

and define the splunk_config sourcetype in props.conf as such:

[splunk_config]
BREAK_ONLY_BEFORE=goblygook
MAX_EVENTS=200000
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime
pulldown_type = true
LEARN_MODEL = false

this combination will index all files under splunk\etc ending in .conf. The BREAK_ONLY_BEFORE=gooblybook basically tells splunk not to break the event (in this case the conf file) until it encounters "gooblygook" which shouldn't be in any of your files.

Re: Entire file contents as a single event

gcoles — Tue, 21 Feb 2012 23:13:13 GMT

As a note to anyone else who may be using this page as a reference, I had been using the LINE_BREAKER directive to do this (as outlined by gkanapathy), but this stopped working when we upgraded our indexers to 4.3. I had to change our props.conf entries for these kinds of inputs to use the method shown by ftk. I verified that the first method fails whether using lightweight or heavy forwarders, as long as the machine that is processing the props.conf for the sourcetype is 4.3.

Re: Entire file contents as a single event

amfranz — Wed, 04 Apr 2012 17:34:13 GMT

In regards to gcoles findings about the first approach not working with Splunk 4.3:

LINE_BREAKER = (?!)

This approach still works in Splunk 4.3 with a minor modification. The expression needs to be surrounded by an additional pair of parantheses:

LINE_BREAKER = ((?!))

I think this is because Splunk 4.3 requires the regular expression to have at least one capture expression, and earlier Splunk versions did not enforce this. The "(?!)" is merely a lookahead expression, the additional pair of parentheses does add a capture expression.

Re: Entire file contents as a single event

lguinn2 — Thu, 30 Jan 2014 16:56:37 GMT

Update: Check out this answer to the same question Each File as One Single Splunk Event

[mysinglefilesourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((*FAIL))
TRUNCATE = 99999999

I think this is newer information

Re: Entire file contents as a single event

Yashar_Shah — Wed, 05 May 2021 13:32:27 GMT

It doesn't enough if you forward data to the indexer. It is just useful for local file monitoring.

Re: Entire file contents as a single event

adobrzeniecki — Mon, 17 May 2021 17:06:38 GMT

Is this still good in 2021?