Developing for Splunk Enterprise

What is the best method to reindex the file after deleting the data?

Takajian
Builder

I sometimes found that splunk indexed the data not as I expected. For example, the data was garbled or line breaks had problems. I could index new file properly after I changed the setting in props.conf to the appropriate one. Then, I would like to delete the indexed data with old setting, and reindex the file with new setting.

I could not remove all index, so I cannot use the "clean" command in the command line. So, I think I need to use "delete" command in splunkweb. But the "delete" command does not affect metadata.
So, I cannot index the file again. I think because the fishbucket still recognizes that the file was indexed and splunk should not reindex the file again.

Could anybody give me advice for this?

Tags (1)

ChrisG
Splunk Employee
Splunk Employee

You can also try the btprobe command, which is described in the Troubleshooting Manual.

David
Splunk Employee
Splunk Employee

I usually opt for one-shotting the input back into the database after issuing a delete. See here for more: http://splunk-base.splunk.com/answers/919/what-is-the-best-way-to-load-archived-logs

If you're in the scenario I recently was, of needing to bring in a very large number of individual files, you can also use xargs

find /path/to/YourLogDirectory/ | xargs -n 1 -I xxx /opt/splunk/bin/splunk add oneshot xxx -index main -sourcetype YourSourceType -auth admin:changeme

neiljpeterson
Communicator

In powershell I was able to use the following to oneshot a directory

forfiles /p D:\tutorialdata /s /c "cmd /c if @isdir==FALSE D:\Splunk\bin\splunk.exe add oneshot @PATH"

/p = path -- /s = search sub-directories -- /c = command -- if the result is not a directory path add it. -- @PATH returns the file name with the path prepended.

0 Karma

crt89
Communicator

I cant seem to make this work? Do I have to make a script here or I should input this command in cmd

0 Karma

slierninja
Communicator

In a windows environment, the following approach should also work....forfiles /p C:\pathToFiles /c "splunk add oneshot @file -index customIndex -host=customHost -sourcetype=customType -auth:admin:pass" leveraging the forfiles DOS command (similar to find/xargs in *nix environments).

0 Karma

johndunlea
Explorer

You could just rename the file something else, and tell Splunk to monitor that file.

Then once Splunk reads in the contents of the "new" file (it will be a different source - but you could force this to be changed at index time), you can rename the file back to its original name, and Splunk will continue monitoring it as it would usually (provided that the inputs for that original file are still present).

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!