Getting Data In

Different xml files have not been indexed: directory monitoring.

spisiakmi
Communicator

Hi. I created directory monitor in Splunk. This monitor monitors a directory, which is situated on the SplunkServer. In the directory there are 232 xml files. I connected this directory with the index and sourcetype in Splunk. There have been indexed only 137 xml files. I found out, that the missing files in Splunk have the same date-time with another indexed file on the minute level. The time differs only in seconds. Can you help me, what is wrong?

0 Karma
1 Solution

spisiakmi
Communicator

I edited inputs.conf and added crcSalt = SOURCE to the monitor. It helped.

View solution in original post

spisiakmi
Communicator

I edited inputs.conf and added crcSalt = SOURCE to the monitor. It helped.

oscar84x
Contributor

What indicates to you that it's related to the time?
Could it be a similarity in the data they contain? Perhaps they have a long, similar header and initCrcLength is not looking in long enough?

0 Karma

spisiakmi
Communicator

Hi oscar84x, it was a mistake, that I though, it is time related. Each file has different timestamp of course. In 1 minute interval there are differencies on the sec level. The first line of each xml file is the same. The second line has some differencies and the rest of the content is total different.

0 Karma

pgerke_cc
Explorer

Is the content of your xml files very different from each other or is ist closely related to each other?

0 Karma

spisiakmi
Communicator

@pgerke_cc:
Here is an example: If I compare 2 different xml logFiles. The first has been indexed, the second neither. The first xml file has timestamp ‎9. ‎Juli ‎2019, ‏‎08:53:17 and the second one has 9. ‎Juli ‎2019, ‏‎08:53:32.
Only the first row is always the same:
The second row is very similar: the time differs, and some atributes.
Unfortunately I cannot share the content of the xml files.

0 Karma

spisiakmi
Communicator

I wanted to paste the first line, but no special chars are here in the forum allowed.

0 Karma

pgerke_cc
Explorer

Ok no problem, but then is probably not the issue I was thinkg of. Can you see a pattern of the indexed vs the unindexed xmls? Like are the unindexed files espacially longer or do they have maye a corrupt syntax so that the source type is not matching?

0 Karma

spisiakmi
Communicator

I tried to index the missing file separately with the same sourcetype in tmp index. It worked. I tried to change the content of the missing file in the monitored directory. The file became new timestamp and despite it, it has not been indexed.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...