Deployment Architecture

Currently audit_db, access_summarydb are consuming space in PCI Search Head. How do I find the indexes.conf file and its configurations?

Motivator

Hi All, Currently we have noticed under this path /opt/splunk/var/lib/splunk/ audit_db & access_summarydb has occupied more space in PCI search head. I am trying to find out what is the retention period configured in the indexes.conf but I am unable to find the indexes.conf file in the PCI search head.

How to find the exact configuration using the Splunk cmd btool command? Though I had tried below command it listed out entire list in that instance .

 ./splunk cmd btool indexes list --debug 

Distributed Environment and we are using Splunk version 6.2.1 with 4 search head, 5 indexer, 1 search job and license /deployment manager.

Kindly guide me in fixing this issue.

thanks in advance.

0 Karma

Legend

There should not be any indexes on the search head. If there are indexes on the search head, you should be forwarding them to the indexer tier. You can see how to do this by reading Best practice: Forward search head data to the indexing layer

If there are no indexes on the search head, then you need to go to the indexers and look for indexes.conf there.
If you set the size of the index to a smaller size, Splunk will immediately begin to reduce the size of the index by removing the oldest data. Remember that you need to restart each indexer after editing indexes.conf. [NB - If you are using an indexer cluster, do the right thing via the cluster master!]

Finally, be sure to make indexes.conf the same on every indexer...

0 Karma

Motivator

Hi lguinn, can you provide us steps on how to reduce the size of the index.

thanks in advance.

0 Karma

Motivator

Hi lguinn, thanks for your response but I think these are the splunk internal index data that are present in the splunk search head under this folder /opt/splunk/var/lib/splunk/ audit_db , not sure why it is configured like this. when I tried to execute this command find -name indexes.conf | xargs grep access_summarydb in search head instance, I have got this output.

./SA-AccessProtection/default.old.20140313-215106/indexes.conf:homePath = $SPLUNK_DB/access_summarydb/db
./SA-AccessProtection/default.old.20140313-215106/indexes.conf:coldPath = $SPLUNK_DB/access_summarydb/colddb
./SA-AccessProtection/default.old.20140313-215106/indexes.conf:thawedPath = $SPLUNK_DB/access_summarydb/thaweddb
./SA-AccessProtection/default.old.20150226-133140/indexes.conf:homePath = $SPLUNK_DB/access_summarydb/db
./SA-AccessProtection/default.old.20150226-133140/indexes.conf:coldPath = $SPLUNK_DB/access_summarydb/colddb
./SA-AccessProtection/default.old.20150226-133140/indexes.conf:thawedPath = $SPLUNK_DB/access_summarydb/thaweddb
./SA-AccessProtection/default/indexes.conf:homePath = $SPLUNK_DB/access_summarydb/db
./SA-AccessProtection/default/indexes.conf:coldPath = $SPLUNK_DB/access_summarydb/colddb
./SA-AccessProtection/default/indexes.conf:thawedPath = $SPLUNK_DB/access_summarydb/thaweddb
./SplunkPCIComplianceSuiteInstaller/default.old.20150226-132847/src/etc/apps/SA-AccessProtection/default/indexes.conf:homePath = $SPLUNK_DB/access_summarydb/db

Not sure what I need to do exactly to reduce the size, I am confused now and worried how to fix this now.

I thought of changing the space in Splunk portal -->setting-->indexes -->audit & access_summarydb both Max Size for entire index = 500000 MB to 300000 MB.

So kindly let provide us some steps to fix this issue that will be really help me in resolving this issues.

thanks in advance.

0 Karma

Splunk Employee
Splunk Employee

To find the index config for the audit index, try this:
on that specific search head:
cd $SPLUNK_HOME
find . -name indexes.conf | xargs grep audit
That should show you a list of files that contain a definition for the _audit index. You can do thee same for any other.
I would suggest forwarding data from the search head to the indexers though https://docs.splunk.com/Documentation/Splunk/6.5.1/DistSearch/Forwardsearchheaddata

This will alleviate the disk issue on your search heads and also allow you to search internal indexes centrally.

0 Karma

Motivator

Hi All, Can any one guide us in getting this issue fixed as we are running short of space in prod environment.

Can we change the Max Size for entire index from 500000 to 300000 MB will this reduce the size of the index ?
What will be the impact if we reduce the index size from 500000 to 300000 MB.

thanks in advance.

0 Karma

Motivator

Hi All, Can any one guide us in getting this issue fixed as we are finding space crunch to manage it.
Below are the settings configured :
From Splunk portal -->setting-->indexes -->audit & access_summarydb both has below setting configured.
Max Size for entire index = 500000 MB
Max Size (MB) /hot/warm/cold bucket are set to auto
Frozen archive path is not set

Currently the size of the audit_db indexes is 30GB & access_summarydb is 51 GB, so kindly let me know how to reduce the size as I am finding very difficult to manage the space in prod environment.

thanks in advance.

0 Karma