I sometimes found that splunk indexed the data not as I expected. For example, the data was garbled or line breaks had problems. I could index new file properly after I changed the setting in props.conf to the appropriate one. Then, I would like to delete the indexed data with old setting, and reindex the file with new setting.
I could not remove all index, so I cannot use the "clean" command in the command line. So, I think I need to use "delete" command in splunkweb. But the "delete" command does not affect metadata.
So, I cannot index the file again. I think because the fishbucket still recognizes that the file was indexed and splunk should not reindex the file again.
Could anybody give me advice for this?
You could just rename the file something else, and tell Splunk to monitor that file.
Then once Splunk reads in the contents of the "new" file (it will be a different source - but you could force this to be changed at index time), you can rename the file back to its original name, and Splunk will continue monitoring it as it would usually (provided that the inputs for that original file are still present).
I usually opt for one-shotting the input back into the database after issuing a delete. See here for more: http://splunk-base.splunk.com/answers/919/what-is-the-best-way-to-load-archived-logs
If you're in the scenario I recently was, of needing to bring in a very large number of individual files, you can also use xargs
find /path/to/YourLogDirectory/ | xargs -n 1 -I xxx /opt/splunk/bin/splunk add oneshot xxx -index main -sourcetype YourSourceType -auth admin:changeme
In a windows environment, the following approach should also work....
forfiles /p C:\pathToFiles /c "splunk add oneshot @file -index customIndex -host=customHost -sourcetype=customType -auth:admin:pass" leveraging the
forfiles DOS command (similar to find/xargs in *nix environments).
In powershell I was able to use the following to oneshot a directory
forfiles /p D:\tutorialdata /s /c "cmd /c if @isdir==FALSE D:\Splunk\bin\splunk.exe add oneshot @PATH"
/p = path -- /s = search sub-directories -- /c = command -- if the result is not a directory path add it. -- @PATH returns the file name with the path prepended.