Dashboards & Visualizations

heavy forwarder not reading logs in syslog from 26 nov 2020

KumaKa
Engager

Hi please anyone help me to sort this issue.

i can see logs getting populated in the syslog.but its not getting ingested into splunk since 26th November 2020 all of a sudden.What may have happened for such log drop.I have been looking around too many forums.But still not able to rectify issue.No configuration changes have been made in the HF too.

please help me to sort it out.many thanks

Labels (1)
0 Karma

gcusello
Esteemed Legend

Hi @KumaKa,

something must be happened otherwise you have the syslogs!

at first, check if you're receiving logs from the HF (Splunk internal logs) and check if you're receiving other syslogs from that HF the the missed ones.

If you're not receiving any log check the connection between HF and indexer

If you're receiving Splunk Internal logs but not other syslogs, check the input and the the ports on the server (telnet on the used port (514?).

if you're receiving other syslogs, check the routes between the sources and the HF (always using telnet).

Ciao.

Giuseppe

0 Karma

KumaKa
Engager

thanks for your reply.

i am getting internal logs as well as few other syslog in SH.But couple of the logs are not ingested into splunk for last few days.

0 Karma

gcusello
Esteemed Legend

Hi @KumaKa,

as I said, if you're receiving internal logs and other syslogs from that HF, this means that the connection between HF and Indexer is OK anche that the ports in HF are open.

So the problem is surely in the connection between appliance and HF.

If your appliance permits a telnet use it for testing connection, otherwise analize network traffic.

Ciao.

Giuseppe

0 Karma

KumaKa
Engager

@gcusello Thanks for your reply.

In our case the syslog server as well as the HF is the same machine.what i can find in syslog is upto date logs.But HF is not reading the logs from the syslog.

I have checked the inputs.conf file.which is all perfect.i am not able to find any issues.

0 Karma

gcusello
Esteemed Legend

Hi @KumaKa,

let me understand: you have an ng-syslog server on your HF and you read the files from the ng-syslog server, is it correct?

Are you sure that syslogs are received and written in a file?

if yes check the input and the reading permissions.

Ciao.

Giuseppe

0 Karma

KumaKa
Engager

yes you are right.i can see logs getting generated every hour in cd /syslog.but not getting those events in the search head.I have checked the inputs.conf file.not able to find any issue

 

below is the inputs.conf 

 

[monitor:///syslog/websense_dlp/shared/.../*.log]
disabled = false
sourcetype = websense:dlp:system:cef
host_segment = 4
index = websense-dlp_sec
crcSalt = <SOURCE>

0 Karma

gcusello
Esteemed Legend

Hi @KumaKa,

what's the name of the file containing syslogs? it's always new or it's always the same?

the option crcSalt = <SOURCE> reads only files with new names.

Ciao.

Giuseppe

0 Karma

KumaKa
Engager

Hi every hour new file will be created with new name

-rw-------. 1 splunk splunk   16062 Nov 30 21:57 2020-11-30-21.log
-rw-------. 1 splunk splunk   11547 Nov 30 22:53 2020-11-30-22.log
-rw-------. 1 splunk splunk    8192 Nov 30 23:55 2020-11-30-23.log.

Mean while I got a message from splunk which states that HF disk space is getting full.

please see the attached photo of the messageUntitled.jpg

 

 

 

0 Karma
Get Updates on the Splunk Community!

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...