Different xml files have not been indexed: directory monitoring.

[spisiakmi]

Hi. I created directory monitor in Splunk. This monitor monitors a directory, which is situated on the SplunkServer. In the directory there are 232 xml files. I connected this directory with the index and sourcetype in Splunk. There have been indexed only 137 xml files. I found out, that the missing files in Splunk have the same date-time with another indexed file on the minute level. The time differs only in seconds. Can you help me, what is wrong?

[spisiakmi · Accepted solution]

I edited inputs.conf and added crcSalt = SOURCE to the monitor. It helped.

[oscar84x]

What indicates to you that it's related to the time?
Could it be a similarity in the data they contain? Perhaps they have a long, similar header and initCrcLength is not looking in long enough?

[spisiakmi]

Hi oscar84x, it was a mistake, that I though, it is time related. Each file has different timestamp of course. In 1 minute interval there are differencies on the sec level. The first line of each xml file is the same. The second line has some differencies and the rest of the content is total different.

[pgerke_cc]

Is the content of your xml files very different from each other or is ist closely related to each other?

[spisiakmi]

@pgerke_cc:
Here is an example: If I compare 2 different xml logFiles. The first has been indexed, the second neither. The first xml file has timestamp ‎9. ‎Juli ‎2019, ‏‎08:53:17 and the second one has 9. ‎Juli ‎2019, ‏‎08:53:32.
Only the first row is always the same:
The second row is very similar: the time differs, and some atributes.
Unfortunately I cannot share the content of the xml files.

[spisiakmi]

I wanted to paste the first line, but no special chars are here in the forum allowed.

[pgerke_cc]

Ok no problem, but then is probably not the issue I was thinkg of. Can you see a pattern of the indexed vs the unindexed xmls? Like are the unindexed files espacially longer or do they have maye a corrupt syntax so that the source type is not matching?

[spisiakmi]

I tried to index the missing file separately with the same sourcetype in tmp index. It worked. I tried to change the content of the missing file in the monitored directory. The file became new timestamp and despite it, it has not been indexed.