Getting Data In

Is it possible to set up a schedule to index all files in a directory?

Ari_McEwing
New Member

Hello Splunk Community,

I am having difficulty monitoring a local directory on my machine. The files are not getting updated. Instead, new files are being added to the directory. Is it possible set up a schedule that indexes all files in a directory? All files are .csv's and are formatted the same way (I.e. same columns in each file). I'd like for Splunk to "monitor" the directory for new files and immediately index them as a certain type under a certain host. Can this be done?

Thanks

0 Karma

somesoni2
Revered Legend

This is how I did similar thing

1) Created a scripted input which will run at a cron schedule, which will copy all the files from that folder to a new directory (accessible to splunk).
2) Created a 'batch' type monitoring input to monitor all the files in the new directory and delete it after that.

[batch://Path/to/new/directory]
...other settings...
crcSalt = <SOURCE>
move_policy =  sinkhole
0 Karma

somesoni2
Revered Legend

This will cause a duplicate copy of all files be available in the servers, so ensure you've ample space in the filesystem you'll be copying the files to.

0 Karma

Ari_McEwing
New Member

What do you mean by "a new directory (accessible to Splunk)"? Is this in reference to a source where Splunk is monitoring a local directory? My main issue is that files added to a Directory being monitored are not being indexed and are not updating with my dashboards.

0 Karma
Get Updates on the Splunk Community!

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

🔐 Trust at Every Hop: How mTLS in Splunk Enterprise 10.0 Makes Security Simpler

From Idea to Implementation: Why Splunk Built mTLS into Splunk Enterprise 10.0  mTLS wasn’t just a checkbox ...