Splunk FIM documentation

[verizonrap2017]

Has anyone worked with ./splunk check-integrity and if yes do you know how to interpret the results? This link does not provide information on how to interpret the results - https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/Dataintegritycontrol

I was provided cursory information but it still does not tell me enough to know when a compromise may have occurred and where.

Example

[tscroggins]

Hi @verizonrap2017,

The command output should match the information you were provided and be otherwise self-explanatory.

A warm bucket with data integrity enabled should have the following files in rawdata:

journal.zst (if zstd compression is used)
l1Hashes_0_<instance_guid>.dat
l2Hash_0_<instance_guid>.dat
slicemin.dat
slicesv2.dat

Calling check-integrity against an unmodified zstd rawdata journal:

$ /opt/splunk/bin/splunk check-integrity -bucketPath /opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0
...
Operating on: idx= bucket='/opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0'
Integrity check succeeded on bucket with path=/opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0
Total buckets checked=1, succeeded=1, failed=0
...

Calling check-integrity against a recompressed zstd rawdata journal:

$ cp journal.zst journal.zst.backup
$ zstd -d journal.zst
$ zstd journal
$ /opt/splunk/bin/splunk check-integrity -bucketPath /opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0
...
Operating on: idx= bucket='/opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0'
Error reading compressed journal /opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0/rawdata/journal.zst while streaming: single-segment zstd compressed block in frame was 352493 bytes long (max should be 131072)
Error parsing rawdata inside bucket path="/opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0": msg="Error reading compressed journal /opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0/rawdata/journal.zst while streaming: single-segment zstd compressed block in frame was 352493 bytes long (max should be 131072)"
Integrity check error for bucket with path=/opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0, Reason=Journal has no hashes.
Total buckets checked=1, succeeded=0, failed=1
...

Calling check-integrity against a recompressed zstd streamed rawdata journal:

$ cp journal.zst journal.zst.backup
$ zstd -d journal.zst
$ cat journal | zstd --no-check - -o journal.zst
$ /opt/splunk/bin/splunk check-integrity -bucketPath /opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0
...
Operating on: idx= bucket='/opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0'
Integrity check failed for bucket with path=/opt/splunk/var/lib/splunk/checkme/db/db_1715457510_1715457464_0, Reason=Hash of journal slice# 1 did not match the expected value in l1Hashes_0_<instance_guid>.dat
Total buckets checked=1, succeeded=0, failed=1
...

Irrespective of how the rawdata journal or hashes are modified, if the calculated hashes do match the saved hashes, the integrity check fails.

If your rawdata journal and hashes are stored together, I wouldn't trust them for evidence of compromise. While a failed integrity check does indicate a problem with either the rawdata journal or hashes, a successful integrity check only confirms that the current rawdata journal and hashes are in agreement. If both were compromised, you would have no way of knowing using only the integrity check.