Hi. I created directory monitor in Splunk. This monitor monitors a directory, which is situated on the SplunkServer. In the directory there are 232 xml files. I connected this directory with the index and sourcetype in Splunk. There have been indexed only 137 xml files. I found out, that the missing files in Splunk have the same date-time with another indexed file on the minute level. The time differs only in seconds. Can you help me, what is wrong?
What indicates to you that it's related to the time?
Could it be a similarity in the data they contain? Perhaps they have a long, similar header and initCrcLength is not looking in long enough?
Hi oscar84x, it was a mistake, that I though, it is time related. Each file has different timestamp of course. In 1 minute interval there are differencies on the sec level. The first line of each xml file is the same. The second line has some differencies and the rest of the content is total different.
Here is an example: If I compare 2 different xml logFiles. The first has been indexed, the second neither. The first xml file has timestamp 9. Juli 2019, 08:53:17 and the second one has 9. Juli 2019, 08:53:32.
Only the first row is always the same:
The second row is very similar: the time differs, and some atributes.
Unfortunately I cannot share the content of the xml files.
Ok no problem, but then is probably not the issue I was thinkg of. Can you see a pattern of the indexed vs the unindexed xmls? Like are the unindexed files espacially longer or do they have maye a corrupt syntax so that the source type is not matching?
I tried to index the missing file separately with the same sourcetype in tmp index. It worked. I tried to change the content of the missing file in the monitored directory. The file became new timestamp and despite it, it has not been indexed.