Splunk Dev

How to make Splunk read the file when only date stamp of the file changes.

lakromani
Builder

I have a system where I use SSH to pull out status data from a remote system
This is then stored to a file that Splunk i set to monitor.
My problem is that the files is read from the system every 5 minutes, but Splunk only shows indexed data when file content is changed.
I would like Splunk to show all the content every time the file changes date (5 min cron job), even if nothing has changed within the file.
Is this possible?

Example first run:
red=1
yellow=2

time stamp of file 09:05
Splunk now show two events.

Second run:
red=1
yellow=2

time stamp of file 09:10
Splunk now shows no events.
I need to show both every 5 min, even if they do not change.

Third run:
red=1
yellow=3

time stamp of file 09:15
Splunk now shows all event again, since content of file has change.

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi lakromani,
you don't need to write a file and then read it, you could use a scripted input (see https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ScriptedInputsIntro ).
In other words, you should create a script with the command you have to run and then schedule and run it in from Splunk.
In this way script output will be indexed with the indexing time and you have your result to search.
To schedule and run a script you have to put your script in $SPLUNK_HOME/bin or $SPLUNK_HOME/etc/apps/your_app/bin folder and then follow the web gui procedure [Settings -- Data Inputs -- Scripts -- New].

Bye.
Giuseppe

View solution in original post

0 Karma

inventsekar
SplunkTrust
SplunkTrust

the file is read by splunk and getting indexed.
but Splunk only shows indexed data when file content is changed.
I would like Splunk to show all the content every time the file changes date (data?!?!?!) , even if nothing has changed within the file.

little bit confusing. is this is the real issue ?

when you search, splunk shows only the recent changed data, not whole data.
when you search, splunk should show the whole content of the file, even if there was no recent updates

what query you are using

0 Karma

lakromani
Builder

It should state "date" in the title, so:
I would like Splunk to show all the content every time the file changes date stamp.

See updated post.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi lakromani,
you don't need to write a file and then read it, you could use a scripted input (see https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ScriptedInputsIntro ).
In other words, you should create a script with the command you have to run and then schedule and run it in from Splunk.
In this way script output will be indexed with the indexing time and you have your result to search.
To schedule and run a script you have to put your script in $SPLUNK_HOME/bin or $SPLUNK_HOME/etc/apps/your_app/bin folder and then follow the web gui procedure [Settings -- Data Inputs -- Scripts -- New].

Bye.
Giuseppe

0 Karma

lakromani
Builder

Would this then give me all the different status for all the event in the file, or only the event that do change?

See updated post.

0 Karma

gcusello
SplunkTrust
SplunkTrust

try to use scripted input, it's the solution.
Bye.
Giuseppe

0 Karma

lakromani
Builder

Can confirm its working.
Learning some every day, thanks.

0 Karma

lakromani
Builder

Will try, thanks.

0 Karma

gcusello
SplunkTrust
SplunkTrust

In search you can show all the indexed data or filter them as you like.
The problem is to take logs only when changed or always.
Using your solution, you index only changes, using scripted inputs, you index script output at every run.
Based on the solution you choose you have to build you search.
What is your need: an alert when there's a change? or to show always situation?
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...