The permissions were incorrect for files being monitored. The files appeared to be indexed but they are not in Splunk. I went in and altered the file to trick the CRC Check Sum thinking it would trick the system into re-indexing the items. I get the following messages when I saved the revised files (real-time) in the _internal index:
Will begin reading at offset=0 for file=
And
group=per_source_thruput, series="/opt/splunk/*.txt", kbps=###, eps=###, kb=###, ev=###, avg_age=###, max_age=###
I am not seeing denied/failed messages. The information is still not indexing.
Please let me know if you have any suggestions.
Thanks, Jenn
The files were not able to reload because there was a TIME_FORMAT error associated. I was able to get the formatting corrected and reload the files. The error message was over a week old, but it was the root cause. I have included the link to the other question associated in case others encounter something similar.
Thanks again for everyone's assistance.
The files were not able to reload because there was a TIME_FORMAT error associated. I was able to get the formatting corrected and reload the files. The error message was over a week old, but it was the root cause. I have included the link to the other question associated in case others encounter something similar.
Thanks again for everyone's assistance.
What is your data input configuration (inputs.conf ) from the forwarder? When updating the content to trick the CRC, what portion of the file you updated, from start of file or end of file?
I altered the start of the file. I added a space initially, but it didn't work. It would not let me save it stating there were no changes. I then added | (pipes). The file updated, but did not index.
Hi Jenn,
I think this Splunk Answer may be what you're after:
http://answers.splunk.com/answers/72562/how-to-reindex-data-from-a-forwarder.html
Thanks for the reference. i was reading about the one shot and fish bucket clean up. I will see if this will work. Thanks!
Quickest way is to delete a specific file from the fishbucket (state monitoring.)
./splunk cmd btprobe -d /path/to/splunk/var/lib/splunk/fishbucket/splunk_private_db --file /full/path/to/filename.txt --reset
That will reset Splunk's internal monitor for files, and force it to reread the specific file. If you have only a hand full of files, this works easily. If you're dealing with thousands of files, then you'd want to script this as wildcards do not work.
Here is a good answers article on various methods : http://answers.splunk.com/answers/72562/how-to-reindex-data-from-a-forwarder.html
If I understand correctly, this will clean out the file so then I can re-index it. It won't impact other files since I am specifying which to look for. I will try this.
I tried the string, but I am getting a file path not recognized. I am in the server, drilled down to the splunk_private_db and then added the string. The first path was the splunk_private_bd and the second was the path to the file. I was able to tab and have it pull the information (auto-fill function in Unix). Which I would expect that if I tab and it auto-fills that the path exists. I am guessing it is a user error and I played around based on documentation, but I am not catching it.
In the server:
logged in as Splunk.
navigate to the /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/ folder
add the following details:
./splunk cmd btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/ --file /opt/splunk/F1/F2/text_file.TXT --reset
Error message:
-bash: ./splunk: No such file or directory
I am going to search on error messages related to the command and see if i can find anything. Please keep me posted if you see anything I missed.
Thanks, Jenn
If your using bash on a *nix based system make sure your in the $SPLUNK_HOME/bin directory before you run that command. Or add /opt/splunk/bin/splunk cmd [etc] to your command.