Why doesn't My heavy forwarder pull a certain log file in?

[phamxuantung]

Hello,

I have a heavy forwarder install on a server to monitor a certain log file. We used to read that log just fine, but after some bug fixed about log generation(on server side) and that server restart, I can't read that log file at all. 

Our inputs.conf was

[monitor:///data/ESB/ACH/LOG/]
disabled = 0
sourcetype = napas.itso.app.achlog	
index = napas.ach.app.log

And it still can read all the logs file in there, but can't read the one that we need. I have restarted the agent, restart the server and restart splunkd connection but it still can't read the one that we need.

We can read

/data/ESB/ACH/LOG/iib_log_summary_2022-07-12.log

but can't read

/data/ESB/ACH/LOG/iib_log_detail_2022-07-12.log

We check the read permission on both file but they're the same.

How can I troubleshoot it?

[gcusello]

Hi @phamxuantung,

Splunk doesn't read a file twice even if the file has the same name..

So maybe the content of the unread file was already read.

You can check if the content of the file was already indexed searching for the content of the file.

You can force Splunk to index the file adding a dedicated stanza with the crcSal option:

[monitor:///data/ESB/ACH/LOG/iib_log_detail_2022-07-12.log]
disabled = 0
sourcetype = napas.itso.app.achlog	
index = napas.ach.app.log
crcSalt = <SOURCE>

 Ciao.

Giuseppe

[phamxuantung]

As you could see, the file that I want to read will have a different name each day, with default name will be iib_log_detail<day>.log (today log is "iib_log_detail_2022-07-12.log" and tomorrow will be "iib_log_detail_2022-07-13.log").

I added your stanza as well as one another as show

After restart Splunkd, search that index and count by source, it still wouldn't show up

[gcusello]

Hi @phamxuantung,

the different name is relevant only if you use the crcSalt option otherwise it isn't relevant: if the file content was already indexed it isn't indexed twice, even if the filename is different!

If you continue to not find the file also after adding the new stanza, please, try to search the file in all time, maybe there's a timestamp parsing error, in other words, search in all time

| metasearch index=napas.ach.app.log source="*iib_log_detail_*.log"

Ciao.

Giuseppe

[phamxuantung]

I run the query you provide on All time and don't see the file, so it's not in the index.

I check the log file itself, the latest one I received was iib_log_detail_2022-07-04.log (8 days ago), the file structure is the same with iib_log_detail_2022-07-12.log so I don't think there's a timestamp parsing error.

[gcusello]

Hi @phamxuantung,

Sorry, I haven't any other idea to identify your issue!

Last try: rename the unread file and modify the stanza in inputs.conf pointing to the new file.

Ciao.

Giuseppe