How can I search on _internal logs from forwarders in my environment?

[yonphang]

Hello everyone,

It seems like I couldn't find any previous answer on this from the community. I have more than 1000 forwarders installed in Windows/Unix servers. I do not have any RDP nor SSH access into those servers due to security reasons. Once in a while, forwarders do not ping back to the server, so I need to access the Splunk "logs" in the directory without having to RDP nor ssh into those servers. I was told that there's already an index which does the internal logging thing and it's doing so in all forwarder agents.

So I was trying to run an index=_internal search in the deployment server, but it returned me with just a hostname, and the hostname was the hostname of the deployment server. Am I doing it wrong? I was trying to check the _internal logs from the forwarders.

Can anyone shed me some light on this?

Thank you

[MuS · Accepted solution]

Hi yonphang,

in the docs about forwarding search head internal logs http://docs.splunk.com/Documentation/Splunk/6.3.0/DistSearch/Forwardsearchheaddata you can find the instruction to do so.

The important part is to add forwardedindex.filter.disable = true in outputs.conf of each of the forwarders and restart them. After that you will be able to search them in index=_internal

Hope this helps ...

cheers, MuS

[yonphang]

does this means the forwarder agent will forward it's splunk log to the master server?
I don't know if the local log gets forwarded automatically?
can you provide me more detail on this?

Thanks a lot

[MuS]

Yes, by setting this option the internal logs of the forwarder will be forwarded to the server set in the outputs.conf of the forwarder.
More details? Hmm, the basics are the same if you forward search heads internal logs or forwarder internal logs - so I don't know if there is more detailed information available....sorry