Getting Data In

Why doesn't My heavy forwarder pull a certain log file in?

phamxuantung
Communicator

Hello,

I have a heavy forwarder install on a server to monitor a certain log file. We used to read that log just fine, but after some bug fixed about log generation(on server side) and that server restart, I can't read that log file at all. 

Our inputs.conf was

 

 

 

 

 

[monitor:///data/ESB/ACH/LOG/]
disabled = 0
sourcetype = napas.itso.app.achlog	
index = napas.ach.app.log

 

 

 

 

 

And it still can read all the logs file in there, but can't read the one that we need. I have restarted the agent, restart the server and restart splunkd connection but it still can't read the one that we need.

We can read

/data/ESB/ACH/LOG/iib_log_summary_2022-07-12.log

but can't read

/data/ESB/ACH/LOG/iib_log_detail_2022-07-12.log

 

We check the read permission on both file but they're the same.

Capture.PNG

How can I troubleshoot it?

Labels (4)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phamxuantung,

Splunk doesn't read a file twice even if the file has the same name..

So maybe the content of the unread file was already read.

You can check if the content of the file was already indexed searching for the content of the file.

You can force Splunk to index the file adding a dedicated stanza with the crcSal option:

[monitor:///data/ESB/ACH/LOG/iib_log_detail_2022-07-12.log]
disabled = 0
sourcetype = napas.itso.app.achlog	
index = napas.ach.app.log
crcSalt = <SOURCE>

 Ciao.

Giuseppe

0 Karma

phamxuantung
Communicator

As you could see, the file that I want to read will have a different name each day, with default name will be iib_log_detail<day>.log (today log is "iib_log_detail_2022-07-12.log" and tomorrow will be "iib_log_detail_2022-07-13.log").

I added your stanza as well as one another as show

stanza.PNG

After restart Splunkd, search that index and count by source, it still wouldn't show up

source.PNG

 

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phamxuantung,

the different name is relevant only if you use the crcSalt option otherwise it isn't relevant: if the file content was already indexed it isn't indexed twice, even if the filename is different!

If you continue to not find the file also after adding the new stanza, please, try to search the file in all time, maybe there's a timestamp parsing error, in other words, search in all time

| metasearch index=napas.ach.app.log source="*iib_log_detail_*.log"

Ciao.

Giuseppe

0 Karma

phamxuantung
Communicator

I run the query you provide on All time and don't see the file, so it's not in the index.

I check the log file itself, the latest one I received was iib_log_detail_2022-07-04.log (8 days ago), the file structure is the same with iib_log_detail_2022-07-12.log so I don't think there's a timestamp parsing error.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phamxuantung,

Sorry, I haven't any other idea to identify your issue!

Last try: rename the unread file and modify the stanza in inputs.conf pointing to the new file.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...