When I search splunk, I'm missing all data from about 100 of the files that list monitor shows that I'm watching?

[cmwhitmanjr]

I'm having an issue with one of my monitored paths.  Here's the monitor stanza, the blacklist line should only blacklist one file in a directory of about 420 log files:

[monitor:///logs/reg*/last/...]
sourcetype = xxxx:Regional
blacklist = xxxx_\d{4}-\d{2}-\d{2}\.log
index = xxxx
disabled = false
crcSalt = <SOURCE>

The output of splunk list monitor shows me all the files I expect to see based on the above stanza.  Splunkd.log shows no problems reading any of them.  My problem is that when I search splunk, I'm missing all data from roughly 100 of the files, files that list monitor shows that I'm watching.   I recently added the crcSalt=<SOURCE> line thinking that would help, it has not.  Am I missing something obvious?

[PickleRick]

crcSalt=<source> makes sense very very rarely. Usually, if necessary, it is enough to increase crcInitLength.

About your case - do "splunk list inputstatus" and see which files are monitored.

And do

| tstats min(_time) max(_time) count where index=xxxx sourcetype=xxxx:Regional by source

[cmwhitmanjr]

I suppose I should have mentioned that prior to adding crcSalt=<source> I was using crcInitLength=1024 and we had the same issue.  I had hoped that by changing the value, it would resolve the issue, it has not.  

splunk list inputstatus shows the files as monitored.  For example:

/logs/reg1/last/1_CABLE_REEL_CABLE_REEL_2023-01-02.log
file position = 710603
file size = 710603
parent = /logs/reg*/last/...
percent = 100.00
type = open file

The tstats search does not return the above source.  

[PickleRick]

Ok, if btool shows that effective input stanza looks like that (the settings aren't by any chance getting overwritten by something else), it should work. Are you sure other sources from the same forwarder are getting ingested properly?

Also, make sure the events aren't redirected to another index or recast to another source/sourcetype. But for that you have to analyze your config on the receiving end.

[cmwhitmanjr]

Yes, I'm sure.  There are somewhere between 400-450 logs in that directory.  We have just over 300 being indexed properly.  They all ingest into the same index / sourcetype.  

I've searched all indexes, I don't see this data popping up anywhere else.  It's just not there at all.  

[PickleRick]

OK. So you can check whether your data is getting received by the downstream component by searching

index=_internal group=per_source_thruput series="your_source_file_path"

You should see the events firstly from your UF, and then from your HF/indexer.

One of the reasons why the events might not be indexed is when the timestamp parsed from the event is too far back into the past (unlikely) or too far ahead into the future (more common) that splunk considers that an error and doesn't process the event. But that should generate a warning in the splunkd.log.

See the MAX_DAYS_HENCE, MAX_DAYS_AGO, MAX_DIFF_SECS_AGO, MAX_DIFF_SECS_HENCE settings.

[gcusello]

Hi @cmwhitmanjr,

crSalt is useful when you have to index files with different names but the same content in the first 256 chars.

If you haven't this condition you can remove it.

One question: why do you have three dots at the end of the monitor header?

Three dots is used when you can have different folder structure levels, if you have always the same structure, you don't need it and you can use *.

[monitor:///logs/reg*/last/*]

Ciao.

Giuseppe

[cmwhitmanjr]

@gcusello Makes sense, as I mentioned in my reply to @PickleRick that line was added trying to solve a problem, but it made no difference in the files ingested anyway.  

One question: why do you have three dots at the end of the monitor header?

     I didn't set it up, I inherited administration after the fact.  I'm not sure why this choice was made over using the asterisk.  There aren't multiple levels of folders inside this directory, so it really doesn't make sense to use dots.  You think changing it will help the problem?

[gcusello]

Hi @cmwhitmanjr,

maybe it could solve: try it.

Ciao,

Giuseppe

[cmwhitmanjr]

I tried this, no change.