Archive

Each File as One Single Splunk Event

Hi everyone,

I need solve a issue as simple as that: my system generate many files and each file is a isolated event.

Each file has many lines (more than 700 lines) but to my business each file is just one single event.

How configure Splunk to treat each file as a single event?

Thanks,

Jefferson Santana

Tags (2)

Communicator

I am using splunk plugin in Jenkins. Where would I make change so that Splunk consider Jenkins log file as one event? I do not have access to .conf files.

If I have to change in .conf file, I may ask admin to make this change but I don't know what change I have to make. Help is appreciated. ,I am using splunk plugin in Jenkins to send Jenkins logs to the Splunk. I want Splunk to treat one log file as a single event. Where would I use ((?!)) or ((*FAIL)) to achieve this? Do I have to make changes to prop.conf and input.conf? What if I do not have access to those files on Splunk server?

0 Karma

Splunk Employee
Splunk Employee

@ss026381 - This question you left a comment on is quite old and may not generate much activity. I would recommend asking a new question. Thank you!

0 Karma

SplunkTrust
SplunkTrust

Any way to just move that ss026381 comment to a new question? It seems pretty complete, but it's seemingly unrelated to where ss026381 posted it.

0 Karma

Communicator

Yea I created new question. Thanks guys

0 Karma

SplunkTrust
SplunkTrust

One way is to set up a dummy/impossible LINE_BREAKER.

In inputs.conf

[monitor:///path/to/files]
sourcetype=my_system

In props.conf (on indexer if using universal forwarder):

[my_system]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]*)-=-=-=-=ThIs-iS-An-ImPoSsiBle-StRiNg=-=-=-=-

If these files change, you may want to also set the CHECK_METHOD on the forwarder itself.
In props.conf (on the forwarder):

[source::/path/to/files/...]
CHECK_METHOD=entire_md5

Splunk Employee
Splunk Employee

The LINE_BREAKER you want is actually either ((?!)) or ((*FAIL)), both of which are guaranteed to fail regardless of the content of your input file.

Splunk Employee
Splunk Employee

The easiest and most efficient way is to set a single sourcetype for your file, and define the rules for this sourcetype:

[mysinglefilesourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((*FAIL))
TRUNCATE = 99999999

This disables line-merging, which sounds wrong, but in fact, you don't want or need it since you won't be breaking the file into separate lines in the first place. The specified LINE_BREAKER is a special PCRE regex that will never break on any line the file, guaranteed. The TRUNCATE setting is there to make sure the entire file is counted as the event, because the default max size is only 10000 characters. You should set it above the expected maximum size of your file. It's not recommend to set it to 0 (no limit) because something could go wrong, or you might drop in some file that shouldn't be there.

Path Finder

To be clear, the above stanza is in props.conf on the indexer. The inputs.conf on the forwarder would be a normal monitor stanza, such as dwaddle suggests below.

0 Karma

Engager

@gkanapathy : Thanks for sharing this. Really useful. I am facing a similar issue to ingest all line in a file as single. But the config works for me only in stand-alone environment. And not when deployed on Heavy forwarder.

Is that because logs are coming partially parsed ( and event-segmented by UF)?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!