Getting Data In

Why doesn't My heavy forwarder pull a certain log file in?

phamxuantung
Communicator

Hello,

I have a heavy forwarder install on a server to monitor a certain log file. We used to read that log just fine, but after some bug fixed about log generation(on server side) and that server restart, I can't read that log file at all. 

Our inputs.conf was

 

 

 

 

 

[monitor:///data/ESB/ACH/LOG/]
disabled = 0
sourcetype = napas.itso.app.achlog	
index = napas.ach.app.log

 

 

 

 

 

And it still can read all the logs file in there, but can't read the one that we need. I have restarted the agent, restart the server and restart splunkd connection but it still can't read the one that we need.

We can read

/data/ESB/ACH/LOG/iib_log_summary_2022-07-12.log

but can't read

/data/ESB/ACH/LOG/iib_log_detail_2022-07-12.log

 

We check the read permission on both file but they're the same.

Capture.PNG

How can I troubleshoot it?

Labels (4)
0 Karma

gcusello
Esteemed Legend

Hi @phamxuantung,

Splunk doesn't read a file twice even if the file has the same name..

So maybe the content of the unread file was already read.

You can check if the content of the file was already indexed searching for the content of the file.

You can force Splunk to index the file adding a dedicated stanza with the crcSal option:

[monitor:///data/ESB/ACH/LOG/iib_log_detail_2022-07-12.log]
disabled = 0
sourcetype = napas.itso.app.achlog	
index = napas.ach.app.log
crcSalt = <SOURCE>

 Ciao.

Giuseppe

0 Karma

phamxuantung
Communicator

As you could see, the file that I want to read will have a different name each day, with default name will be iib_log_detail<day>.log (today log is "iib_log_detail_2022-07-12.log" and tomorrow will be "iib_log_detail_2022-07-13.log").

I added your stanza as well as one another as show

stanza.PNG

After restart Splunkd, search that index and count by source, it still wouldn't show up

source.PNG

 

Tags (1)
0 Karma

gcusello
Esteemed Legend

Hi @phamxuantung,

the different name is relevant only if you use the crcSalt option otherwise it isn't relevant: if the file content was already indexed it isn't indexed twice, even if the filename is different!

If you continue to not find the file also after adding the new stanza, please, try to search the file in all time, maybe there's a timestamp parsing error, in other words, search in all time

| metasearch index=napas.ach.app.log source="*iib_log_detail_*.log"

Ciao.

Giuseppe

0 Karma

phamxuantung
Communicator

I run the query you provide on All time and don't see the file, so it's not in the index.

I check the log file itself, the latest one I received was iib_log_detail_2022-07-04.log (8 days ago), the file structure is the same with iib_log_detail_2022-07-12.log so I don't think there's a timestamp parsing error.

0 Karma

gcusello
Esteemed Legend

Hi @phamxuantung,

Sorry, I haven't any other idea to identify your issue!

Last try: rename the unread file and modify the stanza in inputs.conf pointing to the new file.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...