Long post, newish to splunk, search strings are still a foreign language to me.
So I am tasked with incorporating azure gov into splunk. Splunk support recommended to use a particular app for micro...
See more...
Long post, newish to splunk, search strings are still a foreign language to me.
So I am tasked with incorporating azure gov into splunk. Splunk support recommended to use a particular app for microsoft cloud services. The app is easy enough to configure and whatnot. But having issue with creating an index for the app and ingesting into splunk.
We have the master node/deployment server, 8 indexers, 5 search heads, 2 heavy forwarders.
How do i create an index in a index cluster?
I ask because the directions seem easy enough, however there are some hiccups. When I look at our indexes listed in splunk web, it does not match what is shown in the indexes.conf files. Which is in itself an issue.
These are the locations that I have found indexes.conf
$SPLUNK_HOME/var/lib/splunk
lists all my indexes and their dat files
$SPLUNK_HOME/etc/system/default/
the default files
$SPLUNK_HOME/etc/system/local/
has a listing of almost 80 indexes, but not all that are in the web portal search head, missing some of the sensitive indexes with naming conventions for systems like our txs and usr like txs_systemlog, usr-firewall, etc.
I went to our master node and the location $SPLUNK_HOME/etc/master-apps/_cluster/local/ to look at what the indexes.conf file says there...but its not present. Yet we obviously have indexes across our cluster.
So here are the issues:
1 - This prevents me from creating the needed index "usr-azure" as I do not where to put it.
2 - why are some indexes, like the sensitive ones, not listed in the conf files but are listed in the /var/lib/splunk/ ?
3 - Why is my master node web showing 48 indexes
yet my indexers separately show 99 indexes?
Additionally, another issue. I know we need to use CLI and edit the indexes.conf file for a indexer cluster, but I tried to do it via the web on indexer1, Settings > Indexes (under Data), and I can click the New Index button. All is good, but when I get to the the App selection, it only lists all the apps.
Whereas all the indexes listed show TWC_all_indexes
Q4 - how do i get that for this app setting "TWC_all_indexes" for new index I am creating? I assume it has something to do with the index clustering and a setting on the master node. But I don't even see that option in the indexes.conf file.