Getting Data In

Inputs.conf Monitoring 1 file 1 event

jeffflynn
Explorer

I am monitoring the logs for an application that spits out 3 xml files per day that I want indexed to 1 event per entire file. Everything is setup and working properly when the files are initially indexed because the files are already complete.

The problem is when the application is actually building the file.
1. The application is creates the file
2. Splunk indexes what is created
3. the application adds to the file
4. splunk indexes the addition to another event
5. application adds more to the file
6. splunk indexes changes to another event.
7. so on and so forth.

Since Monitor doesn't use Interval to index new files every however many seconds or via cron job I kinda at a loss. I have tried setting the time_before_close=120 but that still indexes the file at creation and then will index the file modifications of the next 2 min in another event. I am wanting 1 event per file. This isn't a high demand application so indexing the files once a day would be acceptable.

Here is my inputs and props which are pretty simple.

[monitor://whatever]
disabled=false
index=data
whitelist=.xml
blacklist=garbage
sourcetype=info
time_before_close=120

[info]
break_only_before=GOBBLEDEEGOOP
max_events=200000
time_prefix=start-time

Tags (2)
0 Karma
1 Solution

jeffflynn
Explorer

I think my best chance at getting what I need is to run a script every morning that will copy the new files to a subfolder. Then index the files from the subfolder using a batch command instead of monitor.

View solution in original post

0 Karma

jeffflynn
Explorer

I think my best chance at getting what I need is to run a script every morning that will copy the new files to a subfolder. Then index the files from the subfolder using a batch command instead of monitor.

0 Karma

Runals
Motivator

Have you looked into the batch command as opposed to monitor?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You may write a small script to read the file and setup a scripted data input to read script's output.
http://docs.splunk.com/Documentation/Splunk/6.1.1/Data/Setupcustominputs

jeffflynn
Explorer

yea. that just lazy typing on my part. i have everything in caps in my props file

0 Karma

Ayn
Legend

First of all, props.conf settings are case sensitive so "break_only_before" needs to be BREAK_ONLY_BEFORE, and so on.

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...