Monitoring Splunk

logs missing in splunk

kharade0009
Loves-to-Learn Lots

Hello,

I have configure splunk forwarder to send logs to splunk on 6 servers.logs are psuhing to the splunk for sometimes.but for it gets stop for some hours and again it gets restarted after some hours.

 

can someone help to gets the exact issue here?

input.conf

[monitor:///var/log/application/*.log]
sourcetype = app-us-west
index = us_west

disabled = false
recursive = true

output.conf

indexAndForward]
index = false

[tcpout]
defaultGroup = default
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:default]
autoLB = true
autoLBFrequency = 30
forceTimebasedAutoLB = true
server = splunk-fwd-:9997
useACK = true

limits.conf

maxKBps = 0

Labels (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @kharade0009,

by default, Splunk UF continously forwards logs, so if you have a pause in forwarding, with the only exception of scripted inputs, this could be caused by external problems as network congestion.

In addition, you should check the upgrade frequency of your data source: maybe they are't continously  upgraded.

Ciao.

Giuseppe

kharade0009
Loves-to-Learn Lots

Hello @gcusello 

Data being updated for other sourcetypes.We are using three source types with single indexer.

Data is not getting updated only for one sourcetype.Also for this sourcetype application logs which we are forwarding are zipping because of log rotation. I can see mismatch at the time of log rotation of the log file.

but issue is intermittent. anything else could be the issue other than network?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @kharade0009,

if you're continously receiving data of other sourcetypes from the same machine, the problem isn't a network congestion.

So analyze your data source to understand the reason of the pauses.

Only one question: you said that the logs are zipped, but could you take the logs before zipping instead of zipped logs?

If you configured your input to take only zipped logs, probably this is the problem.

Ciao.

Giuseppe

0 Karma

kharade0009
Loves-to-Learn Lots

Hello @gcusello 

in my inputs.conf file I have mentioned as monitor:///var/log/application/*.log.

My log are creating as app-stats.2022-01-04-101.log.gz zip file.

It seems it is taking my .log file only.not sure if any thing wrong with indexer or AutoLB.

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @kharade0009,

maybe the zip file isn't continously updated but it's created and updated in one shot, so your UF read the zip file content in one shot and you index logs in one shot and not in continous way.

See if you can continously write logs in the file system instead in one shot in the zip file, otherwise I think that you can do few.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...