Deployment Architecture

Is it possible to see _audit and _internal events from all search heads on each of them?

m_efremov
Explorer

I have a Splunk 6.2 search head cluster integrated with indexer cluster. If I search in index=_audit, I see _audit events from all members of indexer cluster, but only from one member of search head cluster (from search head on which this query is performed). Same situation with _internal index (but in _internal I also see events from all my forwarders). I want to see _audit and _internal data from all my splunk search heads (not only from all indexers and forwarders). Is it possible? How to configure?

0 Karma
1 Solution

Steve_G_
Splunk Employee
Splunk Employee

For instructions on how to forward internal data from the search heads to the indexers, see http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

View solution in original post

dveuve_splunk
Splunk Employee
Splunk Employee

Once you have the right config in place to get the data in, also think about how to make sure that you are able to make sense of that data. I have an app designed to analyze _audit data and provide analytics for user behavior and adoption, in case that is helpful. https://splunkbase.splunk.com/app/2632/

0 Karma

m_efremov
Explorer

Interestig, but there is the notification "Do not install on a Search Head Cluster" at app's documentation: https://splunkbase.splunk.com/app/2632/#/documentation . I have a search head cluster and can't test this app on it.

0 Karma

dveuve_splunk
Splunk Employee
Splunk Employee

You should generally have at least one non-clustered search head in your environment to allow for management applications such as the DMC and this app. This can live on administrative functions such as the deployer. You may want to consider setting one up in the future.

0 Karma

Steve_G_
Splunk Employee
Splunk Employee

For instructions on how to forward internal data from the search heads to the indexers, see http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

View solution in original post

m_efremov
Explorer

I have been tested it and it works. Solutions from previous comments are also correct, main idea is to point search heads to indexers in outputs.conf

0 Karma

jimodonald
Contributor

forward those events to your indexers just like you'd configure any forwarder.

m_efremov
Explorer

Thank you for answer, it may be a good solution, but is there any native method? I have no forwarders on indexer cluster members, but _internal and _audit log from all of them accessible on all other indexers and from search head cluster members (that integrated with the indexer cluster).

0 Karma

dveuve_splunk
Splunk Employee
Splunk Employee

M_efremov, just configure an outputs.conf on your search heads to point to your indexers. There is no requirement for an external forwarder, and this is the native, best practice approach for Splunk. This will ensure that any logs, and also any summary indexed data, will be available everywhere you need it.

0 Karma

yannK
Splunk Employee
Splunk Employee

You can use the same outputs.conf you deployed on forwarders, or use the sh UI (receiving and forwarding)

Remember that you need to make sure that the destination indexes exists on the indexers (by example if you had custom summary indexes on the SH)

0 Karma

m_efremov
Explorer

I will test this solution and report here about results, thanks again

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!