Hello,
I have an inputs.conf monitor
stanza configured to ingest syslog data that has been written to a file with rsyslog
.
However, I misconfigured the monitor
stanza to send data to a non-existant index. It appears that this data has not been ingested into any other index at this time (there was no indexes.conf lastChanceIndex
set at that time).
I now would like to reingest all data non-destructively to the proper index.
How can I do configure a monitor
stanza to reingest previously ingested data? Do I set the initCrcLength
to 0
?
Thanks,
Matt
What i would do for this is identify which files you want to ingest, ensure that those files are not in the index (to avoid duplicates) and then oneshot the files rather than messing with initCrcLength or adding a crc salt to the input.
Create a list of the files that you are missing including the path
Then ...
Option one:
Feed the file list into oneshot, ensuring that you use correct INDEXNAME and SOURCETYPENAME that your input is pointing to. (the correct index...not the one that doesn't exist 😉 )
#/bin/bash
while read missingfile; do
echo "$SPLUNK_HOME/bin/splunk add oneshot '$missingfile' -index INDEXNAME -sourcetype SOURCETYPENAME"
done < ./LIST_OF_FILENAMES.txt
Option two:
Selectively reset the location on the files you are wanting to (re)ingest from thefishbucket
Taking the same list of files that you want to reinput.. run something like:
#/bin/bash
$SPLUNK_HOME/bin/splunk stop
while read missingfile; do
echo "$SPLUNK_HOME/bin/splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file '$missingfile' --reset
done < ./LIST_OF_FILENAMES.txt
$SPLUNK_HOME/bin/splunk start
Splunk UF agent to 'reprocess' a monitor file. Noticed that there are events in the SplunkUF monitored file are indexed. Wondering if
./splunk cmd btprobe -d /opt/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/messages --reset
would do an incremental reindex of the missed events of the file?
You could reset the fishbucket
but IMHO, the simplest way BY FAR is to create a short script to iterate over the missing files using the ${SPLUNK_HOME}/bin/splunk add oneshot
command and take over total control of what gets resent.