topic Re: re-read a directory in Getting Data In

re-read a directory

dinisco — Tue, 15 Mar 2011 13:14:23 GMT

Having some trouble with a directory monitor:

[monitor:///usr/local/ecc_to_splunk/pickup/*.disk.*]

This monitor loaded the data, but I deleted it (for unrelated reasons) and am having trouble getting splunk to read it again. After deleting the data in splunk using |delete, I removed the files from the dir, disabled then enabled the monitor, then put the same files back.

It seems as though splunk isn't reading the files because it already has once. Is there a way to override this and force splunk to read them? Thanks.

Re: re-read a directory

netwrkr — Tue, 15 Mar 2011 20:29:41 GMT

If I understand you correctly, Splunk has previously indexed the data. Even if you delete the source file(s) and then later on re-add them, I do not think Splunk will re-index them as they already existing within Splunk.

Re: re-read a directory

gkanapathy — Tue, 15 Mar 2011 21:20:00 GMT

You can force Splunk to forget all file history that it has read by cleaning out the fishbucket directory (while Splunk is down) on the machine where it was read from. This probably isn't what you want. You can also have Splunk re-index a specific file using:

./splunk add oneshot /usr/local/ecc_to_splunk/pickup/file1.disk.ext

You can't wildcard this, you have to run this for each specific file name, though you could of course script that in the shell.

Re: re-read a directory

dinisco — Tue, 15 Mar 2011 23:41:43 GMT

Thanks this is exactly what I was looking for. Is there a way to set the sourcetype? I have one specifically defined for these files in tranforms.conf.

Re: re-read a directory

gkanapathy — Wed, 16 Mar 2011 01:28:47 GMT

you can add a -sourcetype mysourcetype flag to the commmand line above.

Re: re-read a directory

joshrabinowitz — Wed, 15 Jun 2011 12:41:03 GMT

So I had the same problem, I had | delete-ed a bunch of data, but then wanted to re-add to splunk.

When using ./splunk add oneshot all the data was added back to splunk BUT the timestamp for ALL the data was from when it was re-added, not the original modtime of the file (input is a directory with 2000+ log files). is there a way to have it re-index using the timestamp of the files?

Re: re-read a directory

joshrabinowitz — Wed, 15 Jun 2011 12:57:35 GMT

well I'm dumb, and should read things first, like putting new data into a test index to make sure it looks ok and test props.conf etc. i guess i can just make a new index and splunk should index with correct timestamps