Showing warn message in index=_internal logs . Missiing some logs from forwarders

[arunsony]

In the internal logs showing a warn message as below for particular servers. Is below are the reasons for missing some logs. If so how to troubleshoot this ?

1. WARN TailingProcessor Insufficient Permissions to read . file /apps/logs/application.log.2017-2-2-44.log (hint:No such file or directory ).

2 . Info Thruputsprocessor the current maxkbps reached to max. Try to increase the maxkbps in limits.conf.

[woodcock]

You need to do a chmod g+r on those files and make sure that the user that is running the splunk process is a member of the group that the user is that owns the files.

[arunsony]

All the permissions are there for the file and directory. But still missing few logs .

[woodcock]

The text "Insufficient Permissions to read" is quite conclusive.

[arunsony]

Where can we the see whether the logs are missing or not in splunk ?

[woodcock]

index=* source="*application.log.2017-2-2-44.log" | stats count by source

[s2_splunk]

Maybe sharing your inputs.conf for this monitor input and the output of ls -alrt /apps/logs would be helpful in making progress towards a resolution.
The error message contains "(hint:No such file or directory )" and there has to be a reason for that.

[arunsony]

Actually on the server there are 4 logs but splunk is showing only 2 logs. where can we confirm in splunk about the missing logs ?

[woodcock]

I don't understand why you need confirmation; the log is crystal clear. Splunk can see the log (enter the directory) but it cannot read it. If it cannot read it, of course the source's data will not be in Splunk.

[arunsony]

Even after giving complete permissions also seeing the same warn message. What could be the problem ?

[woodcock]

Maybe you are looking at old errors?

[gcusello]

HI arunsony,
first message is related to grants to access files in the target server, to monitor your file, you have to give to splunk additional grants.
the second one is related to a parameter to limit the thruput of Universal Forwarders and Heavy Forwarders towards the indexer.

from https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Limitsconf

[thruput]
    maxKBps = <integer>
    * If specified and not zero, this limits the speed through the thruput processor in the ingestion pipeline to the specified rate in kilobytes per second.
    * To control the CPU load while indexing, use this to throttle the number of events this indexer processes to the rate (in KBps) you specify.
    * Note that this limit will be applied per ingestion pipeline. For more information about multiple ingestion pipelines see 
      parallelIngestionPipelines in the server.conf.spec file.
    * With N parallel ingestion pipelines the thruput limit across all of the ingestion pipelines will be N * maxKBps.
    * Default: 0 (unlimited)

Anyway, you don't lose logs for the maxKBps issue, you only receive logs slower, instead you don't ingest logs for the first problem.

Bye.
Giuseppe