Splunk Search

Cant add my complete list of sources.

clintla
Contributor

I've got 2 folders of config data- both have 21 files.

Splunk is only adding 17 from one folder & 9 from the other.
All these files contain very similar data & all the filename
formats are identical

I've tried things like deleting all files, running
splunk clean all -f
then creating all files new.
Nogo.

Tried instead of adding the directory but just adding full path
to the files I noticed it was missing but it says that they are
already added.

Tried making a new directory of all the missing files & adding
this new directory for splunk to consume- nogo. Still just sees
26 sources.

If I do a search on the source that is not listed- finds no hits
& these files are not searchable in anyway through splunk.

Looking for ways to troubleshoot this problem.

Tried copying one of the directories to another server running
another trial version (4.2.1) & it only saw the same 9 files.

Checked the rights/permissions of the files, checked data.. all
the same. Not sure why Splunk has a problem with these data files.

Tags (2)
1 Solution

jbsplunk
Splunk Employee
Splunk Employee

What does it show you about the file if you look at the rest endpoint from the command line? From $SPLUNK_HOME/bin you can run 'splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus'. It should tell you what the status of the file is, if Splunk read it, what the size was when it was read, and to what percentage splunk read the file.

View solution in original post

jbsplunk
Splunk Employee
Splunk Employee

What does it show you about the file if you look at the rest endpoint from the command line? From $SPLUNK_HOME/bin you can run 'splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus'. It should tell you what the status of the file is, if Splunk read it, what the size was when it was read, and to what percentage splunk read the file.

mw
Splunk Employee
Splunk Employee

I'm not sure you're understanding. You want to cut and paste this: crcSalt=<SOURCE>

We're not telling you to replace <SOURCE> with the "source" of the data. Literally put that string in there. It will work if you do this correctly.

0 Karma

clintla
Contributor

Yea.. tried that several times too.
as well as crcSalt=<> and crcSalt=<> nogo

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

You should be doing this, verbatim, in your input:

crcSalt=

What you have there isn't going to work if that you've got in your inputs.

clintla
Contributor

[monitor://C:\getdisks]
crcSalt=<>
crcSalt=<>
crcSalt=<>
crcSalt=<>
crcSalt=<>
disabled = false
followTail = 0
sourcetype = diskinfo
host_regex = :\getdisks\(.*)-DISK.txt$
\cr
what is CRCsalt? what in the output is different?
all these scripts run the same every time- yet w/
new files/folder/splunk installs- still has the
issues with the same files.

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

What is the exact syntax of the crcSalt setting you used in your inputs.conf?

clintla
Contributor

Yea.. Its being ignored but Splunk refuses to index these files. I've put in the crcSalt command to all the missing files, stop/restart splunk service, restart server- still will not index these files. Still get that same error after all this:
ignored file (crc conflict, needs crcSalt)/s:key

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

Yes, it is, but your syntax is incorrect. It looks like you've called 'Filestatus', which is nonexistent, and not 'FileStatus'.

clintla
Contributor

Didnt work, when I check
Manager » Data inputs » Files & directories

it seems like it sees the files. (per below- sees 23 files- not sure why it sees 2 more
- Maybe system files in there?)

C:\getdisks Regular Expresion diskinfo default 23 system Enabled | Disable Clone | Delete

C:\getrgs Regular Expresion rginfo default 23 system Enabled | Disable Clone | Delete

0 Karma

clintla
Contributor

So what does this mean? I can search this & it finds the file. Still though it shows 26 sourcetypes. Its a trial license- this there are limiting usage restraints?

0 Karma

mw
Splunk Employee
Splunk Employee

try searching the _internal index for any mention of one of the files that wasn't indexed: index=_internal myfilename

0 Karma

clintla
Contributor

I tried stopping/starting splunk service. Nogo
Then tried the splunk clean all -f process. Nogo
Then just restarted the server- nogo.

0 Karma

mw
Splunk Employee
Splunk Employee

So you edited inputs.conf and restarted?

0 Karma

mw
Splunk Employee
Splunk Employee

Find the relevant stanza in your inputs.conf and add:

crcSalt=<SOURCE>

That's a literal "".

From the inputs.conf doc:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...