Splunk 7.2 searches do not work after install.

[nick405060]

I can't run a search on either the Splunk 7.2 indexer or search head that I just installed. I get the error "Could not create search." I have no idea how to proceed and there is zero real documentation about this extremely fundamental error (I tried messing with limits.conf).

I haven't done much configuration-wise so basically Splunk 7.2 gives this error - at least on a clean Ubuntu 16.04 VM - right out of the box. Fun times.

[sandeeprachuri · Accepted solution]

@nick405060 added as a answer

I can see that minimum disk space is fallen below 5000MB. This will stop searches. As a workaround, change that to 1000 or 500MB and give a restart. I used to do this every time. Once our /OPT folder increased to 60GB from 6GB, I changed back the setting to 5000MB.

[nick405060]

Resized my partitions and that fixed the problem.

[sandeeprachuri]

I can see that minimum disk space is fallen below 5000MB. This will stop searches. As a workaround, change that to 1000 or 500MB and give a restart. I used to do this every time. Once our /OPT folder increased to 60GB from 6GB, I changed back the setting to 5000MB.

[nick405060]

Can a moderator move this comment to be an answer? Thanks!

[sandeeprachuri]

@nick405060 , Do you see any error messages related to dispatcher?

I used to see same error when all of our installation space occupied by a large search. As soon as we stop that search, we get back our space and then searches will run normally.

Try to use default values in your limits.conf and give a restart.

Thanks,
Sandeep

[nick405060]

I do have dispatch errors, however I haven't ran a large search yet (that I know of) and have rebooted Splunk a bunch of times. Dispatch folder is completely empty.

[sandeeprachuri]

@nick405060 , It's strange really. Can you post those errors?

Also, make sure there are no special characters inserted in .conf files. Check recently changed .conf files. I usually press "CTRL + C/V/S" while doing changes in VI editor.

I had this issue sometime back, After the restart Splunk web was unavailable. It took sometime for me to figure out the error.

[nick405060]

Thanks a ton for helping me out. Replaced all conf files. Here are the current 7 messages Splunk gives me:

Dispatch Command: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch.
10/31/2018, 3:19:00 PM
Audit event generator: Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly.
10/31/2018, 3:15:18 PM
Failed to start KV Store process. See mongod.log and splunkd.log for details.
10/31/2018, 12:23:33 PM
Splunk has found 34 orphaned searches owned by 1 unique disabled users.Click to view the orphaned scheduled searches. Reassign them to a valid user to re-enable or alternatively disable the searches.
10/31/2018, 12:23:33 PM
Disk Monitor: The index processor has paused data flow. Current free disk space on partition '/' has fallen to 4297MB, below the minimum of 5000MB. Data writes to index path '/opt/splunk/var/lib/splunk/audit/db'cannot safely proceed. Increase free disk space on partition '/' by removing or relocating data. Learn more.
10/31/2018, 12:23:32 PM
KV Store changed status to failed. KVStore process terminated.
10/31/2018, 12:23:32 PM
KV Store process terminated abnormally (exit code 1, status exited with code 1). See mongod.log and splunkd.log for details.
10/31/2018, 12:23:32 PM

[nick405060]

(And I know it looks like there's a disk space problem, but I provisioned 70GB on disk 1 and 500GB on disk 2 of my VM, and there's nothing else on the server besides the clean Splunk instance I just installed, so I'm not sure how that is contributing)

[nick405060]

fdisk -l:

Disk /dev/sda: 70 GiB, 75161927680 bytes, 146800640 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc1cc14d8

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 1499135 1497088 731M 83 Linux
/dev/sda2 1501182 146798591 145297410 69.3G 5 Extended
/dev/sda5 1501184 146798591 145297408 69.3G 8e Linux LVM

Disk /dev/sdb: 500 GiB, 536870912000 bytes, 1048576000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/1ABC--ABC01--AB1--vg-root: 8.4 GiB, 8975810560 bytes, 17530880 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/1SPL--INF01--DC1--vg-swap_1: 976 MiB, 1023410176 bytes, 1998848 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes