Why does my Splunk is not indexing all the files that pass thru inputs.conf?

[danielgp89]

Hello Everyone!

I created an inputs.conf for index different files, but after a few files indexed it stop indexing new incoming files. For the record the files are historical.

What could be going wrong?

This is my config:

[monitor:/data01/reports-splunk]
disabled = 0
index = racf
sourcetype = Zsecure

[woodcock]

It probably is not stopping; it is probably getting bogged down because you have too many files co-resident with the files being monitored. If there are hundreds of files, things get slow, if there are thousands of files, forwarding breaks down completely. It has nothing to do with whether the files are being monitored or not; the mere presence of the files creates trouble. Perform proper OS-level housekeeping to move the old files someplace else (or delete them).

[danielgp89]

Hello @thesplunkmonkey sorry to answer you until now!

Let me answer your questions:

1. If that's a typo in the actual inputs.conf file, that could be an issue.

A.- hehe yes I typed wrong

2. Did you also either restart the UF or ensure that the inputs configs are reloaded?

A.- Yes, After a Changed the inputs I restarted the splunk

3. Are you dropping the files on a server running a UF, on an HF, or on an indexer directly?

A.- I'm using a Splunk Enterprise with the 60 days trial. Search Head and Indexer in the same intallation

4.- The first 256 bytes of the file are used by splunk to determine of the file it's reading is a match to a file it's already indexed. I know you say that the data is different in the reports, but do they perhaps have the same header on the report that is long enough that the first 256 bytes of the file are all the same, and therefore splunk is ignoring it because it already matches what it's got for a file in it's fishbucket?

A. Yes they have the same header well in this case the same structure, but it suppose that splunk can index anything and everything every time? How can I change this 256 bytes?

5.- Have you checked your date/time values on the reports to ensure that they are correct, the timezone is correct, etc? If that's wrong, you could be seeing your data indexed either in the past or the future, and outside of your expected search date/time range which may make it appear to be missing.

A.- Yes, the file that I upload I check the date that is suppose to be, but if not, in the Timeline I specify all time to check if there is the data in another date, but no! It doesn't appear any data.

The thing That I'm doing to solve this problem is to uninstall and install again the splunk, but imagine how can I fix this issue if this goes to production and the data is not indexing in of all the different sources.

I attach you some screenshots of the reports that I'm uploading to the inputs.conf

As I told you, we use different paths, because the data that each txt file has is different and we need to differentiate them with the source, even when the structure of the data is the same:

Here are the screnshots of the txt files with the path of each one

[default] host = MX-AIPHS-01

[monitor:///data01/reportes-splunk/racf-commands]
disabled = 0 index = racf sourcetype =
Zsecure2

[monitor:///data01/reportes-splunk/racf-resource]
disabled = 0 index = racf sourcetype =
Zsecure2

I didn't upload more sample because the post didn't let me, but all the txt files have the same structure.

[adonio]

not enough information in your question @danielgp89
start here https://docs.splunk.com/Documentation/Splunk/7.3.1/Troubleshooting/Cantfinddata

[thesplunkmonkey]

It's hard to say given the limited information.

But you seem to be lacking a couple if /.

Monitor stanza should be monitor:// followed by the monitored path /data01/reports-splunk.
Try:

[monitor:///data01/reports-splunk]

If that's not it, please provide some more info about the files being monitored.

[danielgp89]

Hello @thesplunkmonkey

Many thanks for your answer and for your time.

I tried what you said, but unfortunately it didn't work.

We are downloading manually some reports from a mainframe tool and we are uploading each report to different paths. We do this, because the files are too big, and we can't upload them manually in the splunk web interface, that's when we created the inputs.conf.

[default] host = MX-AIPHS-01

[monitor:///data01/reportes-splunk/racf-commands]
disabled = 0 index = racf sourcetype =
Zsecure2

[monitor:///data01/reportes-splunk/resource-access]
disabled = 0 index = racf sourcetype =
Zsecure2

[monitor:///data01/reportes-splunk/viol-warn]
disabled = 0 index = racf sourcetype =
Zsecure2

[monitor:///data01/reportes-splunk/racf-dataset]
disabled = 0 index = racf sourcetype =
Zsecure2

[monitor:///data01/reportes-splunk/racf-resource]
disabled = 0 index = racf sourcetype =
Zsecure2

[monitor:///data01/reportes-splunk/racf-grupos]
disabled = 0 index = racf sourcetype =
Zsecure2

[monitor:///data01/reportes-splunk/report-bau]
disabled = 0 index = bau sourcetype =
cics_vn

For example when we upload two reports in two different paths (///data01/reportes-splunk/racf-commands or ///data01/reportes-splunk/racf-grupos) Splunk can read them and index them, but after we try with the other restant paths splunk doesn't read the new reports.

Each report has different data, but they share the same structure. We separate in different paths because we want that the source has the meaning of the report data.

[thesplunkmonkey]

It may just be a typo on your comment above, but i see the path is different (reportes-splunk vs resports-splunk) from the original post. If that's a typo in the actual inputs.conf file, that could be an issue. For now, i'll assume it's just a typo on here and move on. 🙂

This may be a silly question, but after you pushed out the update to inputs.conf with the new paths, did you also either restart the UF or ensure that the inputs configs are reloaded?

Are you dropping the files on a server running a UF, on an HF, or on an indexer directly?

The first 256 bytes of the file are used by splunk to determine of the file it's reading is a match to a file it's already indexed. I know you say that the data is different in the reports, but do they perhaps have the same header on the report that is long enough that the first 256 bytes of the file are all the same, and therefore splunk is ignoring it because it already matches what it's got for a file in it's fishbucket?

Have you checked your date/time values on the reports to ensure that they are correct, the timezone is correct, etc? If that's wrong, you could be seeing your data indexed either in the past or the future, and outside of your expected search date/time range which may make it appear to be missing.

Looking forward to your response.

[danielgp89]

Hello @thesplunkmonkey

Time since I post this question!

Just for everyone that maybe could have the same issue.

The solution was in the 256 bytes that you mention, thats why the reports where not indexing to splunk.

I applied in the inputs.conf in all stanzas the next variable:

initCrcLength = 1048576

With this variable the problem was fix

Thanks a lot for your help