Splunk Search

Each File as One Single Splunk Event

jefferson_santa
Engager

Hi everyone,

I need solve a issue as simple as that: my system generate many files and each file is a isolated event.

Each file has many lines (more than 700 lines) but to my business each file is just one single event.

How configure Splunk to treat each file as a single event?

Thanks,

Jefferson Santana

Tags (2)

ss026381
Communicator

I am using splunk plugin in Jenkins. Where would I make change so that Splunk consider Jenkins log file as one event? I do not have access to .conf files.

If I have to change in .conf file, I may ask admin to make this change but I don't know what change I have to make. Help is appreciated. ,I am using splunk plugin in Jenkins to send Jenkins logs to the Splunk. I want Splunk to treat one log file as a single event. Where would I use ((?!)) or ((*FAIL)) to achieve this? Do I have to make changes to prop.conf and input.conf? What if I do not have access to those files on Splunk server?

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

@ss026381 - This question you left a comment on is quite old and may not generate much activity. I would recommend asking a new question. Thank you!

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Any way to just move that ss026381 comment to a new question? It seems pretty complete, but it's seemingly unrelated to where ss026381 posted it.

0 Karma

ss026381
Communicator

Yea I created new question. Thanks guys

0 Karma

dwaddle
SplunkTrust
SplunkTrust

One way is to set up a dummy/impossible LINE_BREAKER.

In inputs.conf

[monitor:///path/to/files]
sourcetype=my_system

In props.conf (on indexer if using universal forwarder):

[my_system]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]*)-=-=-=-=ThIs-iS-An-ImPoSsiBle-StRiNg=-=-=-=-

If these files change, you may want to also set the CHECK_METHOD on the forwarder itself.
In props.conf (on the forwarder):

[source::/path/to/files/...]
CHECK_METHOD=entire_md5

gkanapathy
Splunk Employee
Splunk Employee

The LINE_BREAKER you want is actually either ((?!)) or ((*FAIL)), both of which are guaranteed to fail regardless of the content of your input file.

gkanapathy
Splunk Employee
Splunk Employee

The easiest and most efficient way is to set a single sourcetype for your file, and define the rules for this sourcetype:

[mysinglefilesourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((*FAIL))
TRUNCATE = 99999999

This disables line-merging, which sounds wrong, but in fact, you don't want or need it since you won't be breaking the file into separate lines in the first place. The specified LINE_BREAKER is a special PCRE regex that will never break on any line the file, guaranteed. The TRUNCATE setting is there to make sure the entire file is counted as the event, because the default max size is only 10000 characters. You should set it above the expected maximum size of your file. It's not recommend to set it to 0 (no limit) because something could go wrong, or you might drop in some file that shouldn't be there.

ckurtz
Path Finder

To be clear, the above stanza is in props.conf on the indexer. The inputs.conf on the forwarder would be a normal monitor stanza, such as dwaddle suggests below.

0 Karma

splk97
Engager

@gkanapathy : Thanks for sharing this. Really useful. I am facing a similar issue to ingest all line in a file as single. But the config works for me only in stand-alone environment. And not when deployed on Heavy forwarder.

Is that because logs are coming partially parsed ( and event-segmented by UF)?

0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...