Getting Data In

Why is the batch input not indexing certain files?

grantsmiley
Path Finder

I have created a sub folder on a windows splunk indexer in which each night a sub directory named for today's date with several files gets copied from a source for indexing & subsequent deleting. There is no whitelist/blacklist applied, everything in this folder goes into the same index, same source type, etc. Simple batch input. My issue is certain files are not being indexed and deleted. They tend to be the bigger ones. Is it possible the file isn't done writing when splunk tried to index it and it gives up or something? I can't fathom why some files work and some don't. The data does not get indexed for the files that are not deleted.

Tags (4)
0 Karma
1 Solution

grantsmiley
Path Finder

It was a crcSalt issue, resolved. Important safety tip, Batch input, even though destructive, still recognizes crcSalt of files that aren't there.

View solution in original post

0 Karma

grantsmiley
Path Finder

It was a crcSalt issue, resolved. Important safety tip, Batch input, even though destructive, still recognizes crcSalt of files that aren't there.

View solution in original post

0 Karma

gjanders
SplunkTrust
SplunkTrust

Please accept your own answer so this is marked as resolved.

Thanks!

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/
0 Karma

robertl_kinney
Explorer

I'd appreciate an ELI5 regarding the statement in the accepted answer: "batch input, even though destructive, still recognizes crcSalt of files that aren't there."

I've inherited a similar situation where batch input won't delete certain files. I suspected it has something to do with the initCrcLength parameter, but I'm having a hard time grasping how that setting works with batch.

[batch:///opt/sbox/data/Proxies/]
move_policy = sinkhole
disabled = false
index = iron_proxy
sourcetype = cisco:wsa:squid
host_regex = \/opt\/sbox\/data\/Proxies\/(\w+)
initCrcLength = 1048576
0 Karma

gjanders
SplunkTrust
SplunkTrust

The crc check works the same way in batch files as in normal files (as far as I know).
Therefore the initCrcLength would be the number of bytes to read from the file to run the checksum on.

You could also use the props.conf to change your checksum method (CHECK_METHOD) or add a crcSalt if the file names are unique.

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/
0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!