topic Re: logs missing in splunk in Monitoring Splunk

logs missing in splunk

kharade0009 — Tue, 04 Jan 2022 05:07:33 GMT

Hello,

I have configure splunk forwarder to send logs to splunk on 6 servers.logs are psuhing to the splunk for sometimes.but for it gets stop for some hours and again it gets restarted after some hours.

can someone help to gets the exact issue here?

input.conf

[monitor:///var/log/application/*.log]
sourcetype = app-us-west
index = us_west

disabled = false
recursive = true

output.conf

indexAndForward]
index = false

[tcpout]
defaultGroup = default
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:default]
autoLB = true
autoLBFrequency = 30
forceTimebasedAutoLB = true
server = splunk-fwd-:9997
useACK = true

limits.conf

maxKBps = 0

Re: logs missing in splunk

gcusello — Tue, 04 Jan 2022 08:02:12 GMT

Hi @kharade0009,

by default, Splunk UF continously forwards logs, so if you have a pause in forwarding, with the only exception of scripted inputs, this could be caused by external problems as network congestion.

In addition, you should check the upgrade frequency of your data source: maybe they are't continously upgraded.

Ciao.

Giuseppe

Re: logs missing in splunk

kharade0009 — Tue, 04 Jan 2022 11:32:12 GMT

Hello @gcusello

Data being updated for other sourcetypes.We are using three source types with single indexer.

Data is not getting updated only for one sourcetype.Also for this sourcetype application logs which we are forwarding are zipping because of log rotation. I can see mismatch at the time of log rotation of the log file.

but issue is intermittent. anything else could be the issue other than network?

Re: logs missing in splunk

gcusello — Tue, 04 Jan 2022 14:41:25 GMT

Hi @kharade0009,

if you're continously receiving data of other sourcetypes from the same machine, the problem isn't a network congestion.

So analyze your data source to understand the reason of the pauses.

Only one question: you said that the logs are zipped, but could you take the logs before zipping instead of zipped logs?

If you configured your input to take only zipped logs, probably this is the problem.

Ciao.

Giuseppe

Re: logs missing in splunk

kharade0009 — Tue, 04 Jan 2022 19:03:01 GMT

Hello @gcusello

in my inputs.conf file I have mentioned as monitor:///var/log/application/*.log.

My log are creating as app-stats.2022-01-04-101.log.gz zip file.

It seems it is taking my .log file only.not sure if any thing wrong with indexer or AutoLB.

Re: logs missing in splunk

gcusello — Wed, 05 Jan 2022 09:28:56 GMT

Hi @kharade0009,

maybe the zip file isn't continously updated but it's created and updated in one shot, so your UF read the zip file content in one shot and you index logs in one shot and not in continous way.

See if you can continously write logs in the file system instead in one shot in the zip file, otherwise I think that you can do few.

Ciao.

Giuseppe