topic Re: Cant add my complete list of sources. in Splunk Search

Cant add my complete list of sources.

clintla — Tue, 21 Jun 2011 22:17:56 GMT

I've got 2 folders of config data- both have 21 files.

Splunk is only adding 17 from one folder & 9 from the other.
All these files contain very similar data & all the filename
formats are identical

I've tried things like deleting all files, running
splunk clean all -f
then creating all files new.
Nogo.

Tried instead of adding the directory but just adding full path
to the files I noticed it was missing but it says that they are
already added.

Tried making a new directory of all the missing files & adding
this new directory for splunk to consume- nogo. Still just sees
26 sources.

If I do a search on the source that is not listed- finds no hits
& these files are not searchable in anyway through splunk.

Looking for ways to troubleshoot this problem.

Tried copying one of the directories to another server running
another trial version (4.2.1) & it only saw the same 9 files.

Checked the rights/permissions of the files, checked data.. all
the same. Not sure why Splunk has a problem with these data files.

Re: Cant add my complete list of sources.

mw — Tue, 21 Jun 2011 23:14:35 GMT

Find the relevant stanza in your inputs.conf and add:

crcSalt=<SOURCE>

That's a literal "".

From the inputs.conf doc:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty.

Re: Cant add my complete list of sources.

clintla — Tue, 21 Jun 2011 23:37:11 GMT

Didnt work, when I check
Manager » Data inputs » Files & directories

it seems like it sees the files. (per below- sees 23 files- not sure why it sees 2 more
- Maybe system files in there?)

C:\getdisks Regular Expresion diskinfo default 23 system Enabled | Disable Clone | Delete

C:\getrgs Regular Expresion rginfo default 23 system Enabled | Disable Clone | Delete

Re: Cant add my complete list of sources.

mw — Tue, 21 Jun 2011 23:44:19 GMT

So you edited inputs.conf and restarted?

Re: Cant add my complete list of sources.

clintla — Tue, 21 Jun 2011 23:47:20 GMT

I tried stopping/starting splunk service. Nogo
Then tried the splunk clean all -f process. Nogo
Then just restarted the server- nogo.

Re: Cant add my complete list of sources.

mw — Mon, 28 Sep 2020 09:41:43 GMT

try searching the _internal index for any mention of one of the files that wasn't indexed: index=_internal myfilename

Re: Cant add my complete list of sources.

clintla — Thu, 23 Jun 2011 17:08:05 GMT

So what does this mean? I can search this & it finds the file. Still though it shows 26 sourcetypes. Its a trial license- this there are limiting usage restraints?

Re: Cant add my complete list of sources.

jbsplunk — Thu, 23 Jun 2011 18:47:37 GMT

What does it show you about the file if you look at the rest endpoint from the command line? From $SPLUNK_HOME/bin you can run 'splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus'. It should tell you what the status of the file is, if Splunk read it, what the size was when it was read, and to what percentage splunk read the file.

Re: Cant add my complete list of sources.

jbsplunk — Thu, 23 Jun 2011 19:04:43 GMT

Yes, it is, but your syntax is incorrect. It looks like you've called 'Filestatus', which is nonexistent, and not 'FileStatus'.

Re: Cant add my complete list of sources.

clintla — Thu, 23 Jun 2011 22:10:43 GMT

Yea.. Its being ignored but Splunk refuses to index these files. I've put in the crcSalt command to all the missing files, stop/restart splunk service, restart server- still will not index these files. Still get that same error after all this:
ignored file (crc conflict, needs crcSalt)/s:key

Re: Cant add my complete list of sources.

jbsplunk — Thu, 23 Jun 2011 22:15:51 GMT

What is the exact syntax of the crcSalt setting you used in your inputs.conf?

Re: Cant add my complete list of sources.

clintla — Thu, 23 Jun 2011 22:29:44 GMT

[monitor://C:\getdisks]
crcSalt=<>
crcSalt=<>
crcSalt=<>
crcSalt=<>
crcSalt=<>
disabled = false
followTail = 0
sourcetype = diskinfo
host_regex = :\getdisks\(.*)-DISK.txt$
\cr
what is CRCsalt? what in the output is different?
all these scripts run the same every time- yet w/
new files/folder/splunk installs- still has the
issues with the same files.

Re: Cant add my complete list of sources.

jbsplunk — Thu, 23 Jun 2011 23:04:10 GMT

You should be doing this, verbatim, in your input:

crcSalt=

What you have there isn't going to work if that you've got in your inputs.

Re: Cant add my complete list of sources.

clintla — Thu, 23 Jun 2011 23:08:34 GMT

Yea.. tried that several times too.
as well as crcSalt=<> and crcSalt=<> nogo

Re: Cant add my complete list of sources.

mw — Thu, 23 Jun 2011 23:29:23 GMT

I'm not sure you're understanding. You want to cut and paste this: crcSalt=<SOURCE>

We're not telling you to replace <SOURCE> with the "source" of the data. Literally put that string in there. It will work if you do this correctly.