Knowledge Management

How can I tell if a file has been recorded correctly by the tailing processor ?

mzorzi
Splunk Employee
Splunk Employee

I have an Universal Forwarder reading files from a directory, but the Indexer has not received them yet.

How can I tell if the files have been really read ?

How can I be sure that Spunk has uniquely identified my files?

Tags (1)
1 Solution

mzorzi
Splunk Employee
Splunk Employee

A very good tool to track the activity by the tailing processor is the command btprobe which is shipped with your Splunk instance.

Btprobe is a tool to query the fishbucket and to understand if a file has been read by Splunk or not, for instance this helps on cases where an UF is reading files in a directory but the indexer has not received the data yet.

1) Btprobe can calculate the crc from the file, that's it, it takes the first 256 bytes and it hashes them.

For Example, let's calculate the CRC of the file /Applications/splunk/var/log/splunk/metrics.log MacbookPro1:bin splunker$ /Applications/splunk/bin/splunk cmd btprobe --compute-crc /Applications/splunk/var/log/splunk/metrics.log Result is: crc=0x28fd140888afea4d decimal=2953538957510437453 2) Btprobe can also query the fishbucket and return other informations, like how far has been reading on the file ( sptr ) and seekpointer (scrc) Example: MacbookPro1:bin splunker$ /Applications/splunk/bin/splunk cmd btprobe -d /Applications/splunk/var/lib/splunk/fishbucket/splunk_private_db/ -k 0x28fd140888afea4d Result is: key=0x28fd140888afea4d scrc=0xeb608f10ccbd8974 sptr=4142935 fcrc=0x4318978e98eb1eb8 flen=0 mdtm=1352194213 wrtm=1352194213 3) Another important feature of btprobe is that it can return all the entries in the fishbucket: Example: MacbookPro1:bin splunker$ /Applications/splunk/bin/splunk cmd btprobe -k ALL -d /Applications/splunk/var/lib/splunk/fishbucket/splunk_private_db/ dumping entire tree to stdout BTree: order=255 Node @24 Node @8200 key=0x18f1972348eccd sptr=279 key=0x5ba274e60fe2b0 sptr=1117 key=0x9c26b90f063e72 sptr=529 key=0xcdf0a44c5536b8 sptr=577 ……… ……… ………

View solution in original post

mzorzi
Splunk Employee
Splunk Employee

A very good tool to track the activity by the tailing processor is the command btprobe which is shipped with your Splunk instance.

Btprobe is a tool to query the fishbucket and to understand if a file has been read by Splunk or not, for instance this helps on cases where an UF is reading files in a directory but the indexer has not received the data yet.

1) Btprobe can calculate the crc from the file, that's it, it takes the first 256 bytes and it hashes them.

For Example, let's calculate the CRC of the file /Applications/splunk/var/log/splunk/metrics.log MacbookPro1:bin splunker$ /Applications/splunk/bin/splunk cmd btprobe --compute-crc /Applications/splunk/var/log/splunk/metrics.log Result is: crc=0x28fd140888afea4d decimal=2953538957510437453 2) Btprobe can also query the fishbucket and return other informations, like how far has been reading on the file ( sptr ) and seekpointer (scrc) Example: MacbookPro1:bin splunker$ /Applications/splunk/bin/splunk cmd btprobe -d /Applications/splunk/var/lib/splunk/fishbucket/splunk_private_db/ -k 0x28fd140888afea4d Result is: key=0x28fd140888afea4d scrc=0xeb608f10ccbd8974 sptr=4142935 fcrc=0x4318978e98eb1eb8 flen=0 mdtm=1352194213 wrtm=1352194213 3) Another important feature of btprobe is that it can return all the entries in the fishbucket: Example: MacbookPro1:bin splunker$ /Applications/splunk/bin/splunk cmd btprobe -k ALL -d /Applications/splunk/var/lib/splunk/fishbucket/splunk_private_db/ dumping entire tree to stdout BTree: order=255 Node @24 Node @8200 key=0x18f1972348eccd sptr=279 key=0x5ba274e60fe2b0 sptr=1117 key=0x9c26b90f063e72 sptr=529 key=0xcdf0a44c5536b8 sptr=577 ……… ……… ………

View solution in original post

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!