i did not configure the indexer server properly initially hence the log is indexed locally. After i configured the indexer server correctly, the forwarder did not forward the data over to indexer and was also not index locally. What happen and how to resolve this?
@SplunkCSIT, I agree - I feel this was not that time though. I'm more than happy to help you, but that's very hard without getting details on your case. Splunk is a complex product and depending on how exactly you've configured it it can behave very differently, so it's crucial that you provide us with details.
I don't know what your problem is or why things are happening, but here is something you must know about re-indexing data. Splunk keeps track of what was indexed and therefor does not re-index the same data again .... unless you do some work.
The radical method is to clean the fishbucket index. That will remove the memory of every files, be warned it will re-index all data.
splunk clean eventdata -index _fishbucket
You could selectively forgot a single file from the fish bucket like this:
splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file $FILE --reset
You could manually re-index each file with the
splunk add oneshot "/path/to/my/file.log" -sourcetype mysourcetype
You could modify the first line of the files to re-index, by default Splunk checks the first 256 chars of a file to differentiate them. If you had a simple comment on the first line it wil reindex it
You could change the crcSalt, create a new input for a new folder, add all the correct sourcetypes, etc... add the option
crcSalt=<SOURCE> then move or copy the files to be re-index to the folder.
hope this helps ...