Getting Data In

re-read a directory

dinisco
Explorer

Having some trouble with a directory monitor:

[monitor:///usr/local/ecc_to_splunk/pickup/*.disk.*]

This monitor loaded the data, but I deleted it (for unrelated reasons) and am having trouble getting splunk to read it again. After deleting the data in splunk using |delete, I removed the files from the dir, disabled then enabled the monitor, then put the same files back.

It seems as though splunk isn't reading the files because it already has once. Is there a way to override this and force splunk to read them? Thanks.

Tags (1)

joshrabinowitz
Path Finder

So I had the same problem, I had | delete-ed a bunch of data, but then wanted to re-add to splunk.

When using ./splunk add oneshot all the data was added back to splunk BUT the timestamp for ALL the data was from when it was re-added, not the original modtime of the file (input is a directory with 2000+ log files). is there a way to have it re-index using the timestamp of the files?

joshrabinowitz
Path Finder

well I'm dumb, and should read things first, like putting new data into a test index to make sure it looks ok and test props.conf etc. i guess i can just make a new index and splunk should index with correct timestamps

gkanapathy
Splunk Employee
Splunk Employee

You can force Splunk to forget all file history that it has read by cleaning out the fishbucket directory (while Splunk is down) on the machine where it was read from. This probably isn't what you want. You can also have Splunk re-index a specific file using:

./splunk add oneshot /usr/local/ecc_to_splunk/pickup/file1.disk.ext

You can't wildcard this, you have to run this for each specific file name, though you could of course script that in the shell.

gkanapathy
Splunk Employee
Splunk Employee

you can add a -sourcetype mysourcetype flag to the commmand line above.

dinisco
Explorer

Thanks this is exactly what I was looking for. Is there a way to set the sourcetype? I have one specifically defined for these files in tranforms.conf.

netwrkr
Communicator

If I understand you correctly, Splunk has previously indexed the data. Even if you delete the source file(s) and then later on re-add them, I do not think Splunk will re-index them as they already existing within Splunk.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...