Getting Data In

Different xml files have not been indexed: directory monitoring.

spisiakmi
Communicator

Hi. I created directory monitor in Splunk. This monitor monitors a directory, which is situated on the SplunkServer. In the directory there are 232 xml files. I connected this directory with the index and sourcetype in Splunk. There have been indexed only 137 xml files. I found out, that the missing files in Splunk have the same date-time with another indexed file on the minute level. The time differs only in seconds. Can you help me, what is wrong?

0 Karma
1 Solution

spisiakmi
Communicator

I edited inputs.conf and added crcSalt = SOURCE to the monitor. It helped.

View solution in original post

spisiakmi
Communicator

I edited inputs.conf and added crcSalt = SOURCE to the monitor. It helped.

oscar84x
Contributor

What indicates to you that it's related to the time?
Could it be a similarity in the data they contain? Perhaps they have a long, similar header and initCrcLength is not looking in long enough?

0 Karma

spisiakmi
Communicator

Hi oscar84x, it was a mistake, that I though, it is time related. Each file has different timestamp of course. In 1 minute interval there are differencies on the sec level. The first line of each xml file is the same. The second line has some differencies and the rest of the content is total different.

0 Karma

pgerke_cc
Explorer

Is the content of your xml files very different from each other or is ist closely related to each other?

0 Karma

spisiakmi
Communicator

@pgerke_cc:
Here is an example: If I compare 2 different xml logFiles. The first has been indexed, the second neither. The first xml file has timestamp ‎9. ‎Juli ‎2019, ‏‎08:53:17 and the second one has 9. ‎Juli ‎2019, ‏‎08:53:32.
Only the first row is always the same:
The second row is very similar: the time differs, and some atributes.
Unfortunately I cannot share the content of the xml files.

0 Karma

spisiakmi
Communicator

I wanted to paste the first line, but no special chars are here in the forum allowed.

0 Karma

pgerke_cc
Explorer

Ok no problem, but then is probably not the issue I was thinkg of. Can you see a pattern of the indexed vs the unindexed xmls? Like are the unindexed files espacially longer or do they have maye a corrupt syntax so that the source type is not matching?

0 Karma

spisiakmi
Communicator

I tried to index the missing file separately with the same sourcetype in tmp index. It worked. I tried to change the content of the missing file in the monitored directory. The file became new timestamp and despite it, it has not been indexed.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...