Getting Data In

Different xml files have not been indexed: directory monitoring.

spisiakmi
Communicator

Hi. I created directory monitor in Splunk. This monitor monitors a directory, which is situated on the SplunkServer. In the directory there are 232 xml files. I connected this directory with the index and sourcetype in Splunk. There have been indexed only 137 xml files. I found out, that the missing files in Splunk have the same date-time with another indexed file on the minute level. The time differs only in seconds. Can you help me, what is wrong?

0 Karma
1 Solution

spisiakmi
Communicator

I edited inputs.conf and added crcSalt = SOURCE to the monitor. It helped.

View solution in original post

0 Karma

spisiakmi
Communicator

I edited inputs.conf and added crcSalt = SOURCE to the monitor. It helped.

View solution in original post

0 Karma

oscar84x
Contributor

What indicates to you that it's related to the time?
Could it be a similarity in the data they contain? Perhaps they have a long, similar header and initCrcLength is not looking in long enough?

0 Karma

spisiakmi
Communicator

Hi oscar84x, it was a mistake, that I though, it is time related. Each file has different timestamp of course. In 1 minute interval there are differencies on the sec level. The first line of each xml file is the same. The second line has some differencies and the rest of the content is total different.

0 Karma

pgerke_cc
Path Finder

Is the content of your xml files very different from each other or is ist closely related to each other?

0 Karma

spisiakmi
Communicator

@pgerke_cc:
Here is an example: If I compare 2 different xml logFiles. The first has been indexed, the second neither. The first xml file has timestamp ‎9. ‎Juli ‎2019, ‏‎08:53:17 and the second one has 9. ‎Juli ‎2019, ‏‎08:53:32.
Only the first row is always the same:
The second row is very similar: the time differs, and some atributes.
Unfortunately I cannot share the content of the xml files.

0 Karma

spisiakmi
Communicator

I wanted to paste the first line, but no special chars are here in the forum allowed.

0 Karma

pgerke_cc
Path Finder

Ok no problem, but then is probably not the issue I was thinkg of. Can you see a pattern of the indexed vs the unindexed xmls? Like are the unindexed files espacially longer or do they have maye a corrupt syntax so that the source type is not matching?

0 Karma

spisiakmi
Communicator

I tried to index the missing file separately with the same sourcetype in tmp index. It worked. I tried to change the content of the missing file in the monitored directory. The file became new timestamp and despite it, it has not been indexed.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!