topic Re: Each File as One Single Splunk Event in Splunk Search

Each File as One Single Splunk Event

jefferson_santa — Sun, 13 Oct 2013 17:50:43 GMT

Hi everyone,

I need solve a issue as simple as that: my system generate many files and each file is a isolated event.

Each file has many lines (more than 700 lines) but to my business each file is just one single event.

How configure Splunk to treat each file as a single event?

Thanks,

Jefferson Santana

Re: Each File as One Single Splunk Event

gkanapathy — Sun, 13 Oct 2013 19:22:28 GMT

The easiest and most efficient way is to set a single sourcetype for your file, and define the rules for this sourcetype:

[mysinglefilesourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((*FAIL))
TRUNCATE = 99999999

This disables line-merging, which sounds wrong, but in fact, you don't want or need it since you won't be breaking the file into separate lines in the first place. The specified LINE_BREAKER is a special PCRE regex that will never break on any line the file, guaranteed. The TRUNCATE setting is there to make sure the entire file is counted as the event, because the default max size is only 10000 characters. You should set it above the expected maximum size of your file. It's not recommend to set it to 0 (no limit) because something could go wrong, or you might drop in some file that shouldn't be there.

Re: Each File as One Single Splunk Event

dwaddle — Sun, 13 Oct 2013 19:27:43 GMT

One way is to set up a dummy/impossible LINE_BREAKER.

In inputs.conf

[monitor:///path/to/files]
sourcetype=my_system

In props.conf (on indexer if using universal forwarder):

[my_system]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]*)-=-=-=-=ThIs-iS-An-ImPoSsiBle-StRiNg=-=-=-=-

If these files change, you may want to also set the CHECK_METHOD on the forwarder itself.
In props.conf (on the forwarder):

[source::/path/to/files/...]
CHECK_METHOD=entire_md5

Re: Each File as One Single Splunk Event

gkanapathy — Sun, 13 Oct 2013 19:30:59 GMT

The LINE_BREAKER you want is actually either ((?!)) or ((*FAIL)), both of which are guaranteed to fail regardless of the content of your input file.

Re: Each File as One Single Splunk Event

ckurtz — Tue, 11 Mar 2014 23:37:21 GMT

To be clear, the above stanza is in props.conf on the indexer. The inputs.conf on the forwarder would be a normal monitor stanza, such as dwaddle suggests below.

Re: Each File as One Single Splunk Event

ss026381 — Mon, 06 Mar 2017 20:32:49 GMT

I am using splunk plugin in Jenkins. Where would I make change so that Splunk consider Jenkins log file as one event? I do not have access to .conf files.

If I have to change in .conf file, I may ask admin to make this change but I don't know what change I have to make. Help is appreciated. ,I am using splunk plugin in Jenkins to send Jenkins logs to the Splunk. I want Splunk to treat one log file as a single event. Where would I use ((?!)) or ((*FAIL)) to achieve this? Do I have to make changes to prop.conf and input.conf? What if I do not have access to those files on Splunk server?

Re: Each File as One Single Splunk Event

aaraneta_splunk — Mon, 06 Mar 2017 20:40:53 GMT

@ss026381 - This question you left a comment on is quite old and may not generate much activity. I would recommend asking a new question. Thank you!

Re: Each File as One Single Splunk Event

DalJeanis — Mon, 06 Mar 2017 20:50:19 GMT

Any way to just move that ss026381 comment to a new question? It seems pretty complete, but it's seemingly unrelated to where ss026381 posted it.

Re: Each File as One Single Splunk Event

ss026381 — Mon, 06 Mar 2017 20:56:00 GMT

Yea I created new question. Thanks guys

Re: Each File as One Single Splunk Event

splk97 — Wed, 20 Mar 2019 12:30:51 GMT

@gkanapathy : Thanks for sharing this. Really useful. I am facing a similar issue to ingest all line in a file as single. But the config works for me only in stand-alone environment. And not when deployed on Heavy forwarder.

Is that because logs are coming partially parsed ( and event-segmented by UF)?