topic Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders in Getting Data In

Showing warn message in index=_internal logs . Missiing some logs from forwarders

arunsony — Mon, 31 Jul 2017 13:42:55 GMT

In the internal logs showing a warn message as below for particular servers. Is below are the reasons for missing some logs. If so how to troubleshoot this ?

WARN TailingProcessor Insufficient Permissions to read . file /apps/logs/application.log.2017-2-2-44.log (hint:No such file or directory ).

2 . Info Thruputsprocessor the current maxkbps reached to max. Try to increase the maxkbps in limits.conf.

Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders

gcusello — Mon, 31 Jul 2017 14:52:30 GMT

HI arunsony,
first message is related to grants to access files in the target server, to monitor your file, you have to give to splunk additional grants.
the second one is related to a parameter to limit the thruput of Universal Forwarders and Heavy Forwarders towards the indexer.

from https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Limitsconf

[thruput]
    maxKBps = <integer>
    * If specified and not zero, this limits the speed through the thruput processor in the ingestion pipeline to the specified rate in kilobytes per second.
    * To control the CPU load while indexing, use this to throttle the number of events this indexer processes to the rate (in KBps) you specify.
    * Note that this limit will be applied per ingestion pipeline. For more information about multiple ingestion pipelines see 
      parallelIngestionPipelines in the server.conf.spec file.
    * With N parallel ingestion pipelines the thruput limit across all of the ingestion pipelines will be N * maxKBps.
    * Default: 0 (unlimited)

Anyway, you don't lose logs for the maxKBps issue, you only receive logs slower, instead you don't ingest logs for the first problem.

Bye.
Giuseppe

Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders

woodcock — Mon, 31 Jul 2017 14:52:46 GMT

You need to do a chmod g+r on those files and make sure that the user that is running the splunk process is a member of the group that the user is that owns the files.

Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders

arunsony — Mon, 31 Jul 2017 15:23:49 GMT

All the permissions are there for the file and directory. But still missing few logs .

Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders

woodcock — Mon, 31 Jul 2017 15:35:41 GMT

The text "Insufficient Permissions to read" is quite conclusive.

Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders

arunsony — Mon, 31 Jul 2017 15:43:05 GMT

Where can we the see whether the logs are missing or not in splunk ?

Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders

woodcock — Mon, 31 Jul 2017 15:58:08 GMT

index=* source="*application.log.2017-2-2-44.log" | stats count by source

Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders

arunsony — Mon, 31 Jul 2017 16:01:19 GMT

Actually on the server there are 4 logs but splunk is showing only 2 logs. where can we confirm in splunk about the missing logs ?

Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders

woodcock — Mon, 31 Jul 2017 16:14:38 GMT

I don't understand why you need confirmation; the log is crystal clear. Splunk can see the log (enter the directory) but it cannot read it. If it cannot read it, of course the source's data will not be in Splunk.

Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders

arunsony — Mon, 31 Jul 2017 16:31:42 GMT

Even after giving complete permissions also seeing the same warn message. What could be the problem ?

Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders

s2_splunk — Mon, 31 Jul 2017 17:04:15 GMT

Maybe sharing your inputs.conf for this monitor input and the output of ls -alrt /apps/logs would be helpful in making progress towards a resolution.
The error message contains "(hint:No such file or directory )" and there has to be a reason for that.

Re: Showing warn message in index=_internal logs . Missiing some logs from forwarders

woodcock — Mon, 31 Jul 2017 17:39:05 GMT

Maybe you are looking at old errors?