How can one have visibility through searches of Data Integrity checks operations in Splunk?

[skender27]

Hi,

Me and my collegue are thinking of a report (then placed in a dashboard) about having visibility on data integrity check Splunk performs.
Once activated the stanza in indexes.conf and executed the Splunk comand: splunk check-integrity –index <index_name>, how can we in search time have some visualization about such activity?

Thanks,
Skender

[gcusello · Accepted solution]

You have to do the following steps:
- create a script with your command,
- put it into your app's bin directory,
- create an input stanza in your inputs.conf

for example:
script file check_integrity.sh
splunk check-integrity –index index_to_check

inputs.conf
[script://$SPLUNK_HOME/etc/apps/yourapp/bin/check_integrity.sh]
disabled = 0
index = your_index
interval = 60.0
sourcetype = your_sourcetype
source = your_source

index_to_check is the index to check integrity
your_index is the index you create where you store the script results

Bye.
Giuseppe

[gurlest]

We are trying to do something similar with a loop to catch all of our indexes:

#!bin/bash
for path in /opt/data/*/*; do splunk check-integrity -index ${path##*/} -verbose; done

However, I have noticed that the results of the command are only output to splunkd.log. I can find the output of the commands in the "_internal" index, but not in the index specified in inputs.conf

Do you have any guidance on how to get the results ingested into an index other than _internal?

[potnuru]

Hi @gurlest
Did you find the solution of the above query?
I am looking for the solution for the same query.

I need to capture the output of check-integrity command (which runs through script ) to specified index in Splunk.

[gurlest]

Yes, we did. We have been able to get it working like a champ. However, I would be a poor Splunk Admin if I didn't say this, as well: THIS IS HUGELY RESOURCE INTENSIVE. Ok. Now that that's out of the way... 🙂

Inputs > data_integrity_checks/local/inputs.conf

[script://./bin/integrity_checks.sh]
disabled = 0
interval = 3 10 * * 6
index = system_events
sourcetype = audit:integrity

Script > data_integrity_checks/bin/integrity_checks.sh

#!/bin/bash
IDX='/opt/splunk/etc/slave-apps/data_integrity_checks/local/dataintegrity_indexes.txt'
for index in $(<$IDX); do $SPLUNK_HOME/bin/splunk check-integrity -index $index -verbose 2>&1; done

The script runs based on the list of indexes on the dataintegrity_indexes.txt file, or you could just pass a list of index names.

[potnuru]

@gurlest Thank you for the response, I will try the above given solution.

[skender27]

Hi Giuseppe,

I tried, but I get only indexed the Splunk comands and no output ("Total buckets checked... etc etc").
Since I am using Windows OS there should be smth to modify with the .bat file to get the standard output...

Is it possible to get the indexed events in the _audit index (and not a custom one)?

Thanks,
Skender

[gcusello]

you have to give the execution rights to the script: if you use operative commands like "query session" the script is correctly executed and output indexed in Splunk.
everyway, you could schedule om Windows the execution of the script as an administrator writing results in a txt file that you can take with Splunk.
Bye.
Giuseppe

[skender27]

I understand.
Thanks again!

Skender