Getting Data In

Splunk as CMDB - initCrcLength set to max creates duplicates

f1dot4
Explorer

Hi,
i want to use splunk as GUI for a CMDB. I know, that not the default use case, but splunk exists already and i like the possibilities for visualization.

I'm indexing textfiles with meta-data of hosts as content, the filename contains timestamp & hostname (this is already working). The point is, that those text-files are created every day, and most of the time, there is no change between the days (no software/hw change, etc) - so there is no need to index them again.

As written in the docs, splunk looks for the first 256 bytes (initCrcLength) to check if the file is already indexed to handle logrotation. Since my case is not a normal logfile, the important change in my files can occur also at the end of the textfile. To ensure that i won't miss a change at the end of the file, i increased initCrcLength to it's maximum of 1048576 (Bytes).
My files are smaller than 1048576 Bytes, so from my point of view, splunk should not index files with the same content (checked from beginning to the end of the file).

My Problem with this configuration is, that exactly this is happening (duplicates are going to be indexed), my inputs.conf:

[monitor://D:\CMDB\lokal\test\*.txt]
host_regex = test\\(.*?)_
initCrcLength = 1048576
disabled = false
index = cmdb
sourcetype = cmdb

Any ideas?
BR, Lukas

jplumsdaine22
Influencer

You might check if they're the same. I think something similar to this python to this is happening under the hood:

import glob, os, hashlib
os.chdir("D:/CMDB/lokal/test/")    

for file in glob.glob("*.txt"):
 currentfile = open(file, 'rb')
 hash = hashlib.md5()
 hash.update(currentfile.read(1048576))
 print currentfile, hash.hexdigest()
 currentfile.close()
0 Karma

dwaddle
SplunkTrust
SplunkTrust

Let's try this differently. Leave initCrcLength alone and set in props.conf:

[source::D:\CMDB\lokal\test\*.txt]
CHECK_METHOD=entire_md5

jplumsdaine22
Influencer

Looks good to me - although I was unclear whether Splunk considers the hostname when comparing crcs

0 Karma

f1dot4
Explorer

Hi, in theory this sounds good. i removed the initCrcLength param from inputs.conf and added CHECK_METHOD = entire_md5 to props.conf. WIth btool, i checked that this config is active.

When i copy the txt file, the md5sum is equal, but it is going to be indexed again.

props.conf:

[source::D:\CMDB\lokal\test\*.txt]
CHECK_METHOD = entire_md5

inputs.conf

 [monitor://D:\CMDB\lokal\test\*.txt]
 host_regex = test\\(.*?)_
 disabled = false
 index = cmdb
 sourcetype = cmdb

md5 check:

md5sum "AUD-S-K001-01__10.txt"
768b4568fa45d8d6771d5ca8160dc483 *AUD-S-K001-01__10.txt

md5sum "AUD-S-K001-01__11.txt"
768b4568fa45d8d6771d5ca8160dc483 *AUD-S-K001-01__11.txt
0 Karma

jplumsdaine22
Influencer

What error are you getting that shows the files are being reindexed?

If you search for index_internal you should see why the reindexing is occuring.

0 Karma

f1dot4
Explorer

There is no error with reindexing - but when i have 2 files with different filenames and exact the same content, they're both indexed. Since most of the files are identical - the index is filling up with lots of duplicate entries - this is what i'm trying to avoid with the initCrcLength setting.

0 Karma

jplumsdaine22
Influencer

if you run

$ head -c   1048576 <filename>.txt | md5sum 

against all those files, do you get the same hash?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...