Splunk Search

Splunk 7.2 searches do not work after install.

nick405060
Motivator

I can't run a search on either the Splunk 7.2 indexer or search head that I just installed. I get the error "Could not create search." I have no idea how to proceed and there is zero real documentation about this extremely fundamental error (I tried messing with limits.conf).

I haven't done much configuration-wise so basically Splunk 7.2 gives this error - at least on a clean Ubuntu 16.04 VM - right out of the box. Fun times.

Tags (1)
0 Karma
1 Solution

sandeeprachuri
Path Finder

@nick405060 added as a answer

I can see that minimum disk space is fallen below 5000MB. This will stop searches. As a workaround, change that to 1000 or 500MB and give a restart. I used to do this every time. Once our /OPT folder increased to 60GB from 6GB, I changed back the setting to 5000MB.

View solution in original post

sandeeprachuri
Path Finder

@nick405060 added as a answer

I can see that minimum disk space is fallen below 5000MB. This will stop searches. As a workaround, change that to 1000 or 500MB and give a restart. I used to do this every time. Once our /OPT folder increased to 60GB from 6GB, I changed back the setting to 5000MB.

nick405060
Motivator

Resized my partitions and that fixed the problem.

sandeeprachuri
Path Finder

I can see that minimum disk space is fallen below 5000MB. This will stop searches. As a workaround, change that to 1000 or 500MB and give a restart. I used to do this every time. Once our /OPT folder increased to 60GB from 6GB, I changed back the setting to 5000MB.

nick405060
Motivator

Can a moderator move this comment to be an answer? Thanks!

0 Karma

sandeeprachuri
Path Finder

@nick405060 , Do you see any error messages related to dispatcher?

I used to see same error when all of our installation space occupied by a large search. As soon as we stop that search, we get back our space and then searches will run normally.

Try to use default values in your limits.conf and give a restart.

Thanks,
Sandeep

0 Karma

nick405060
Motivator

I do have dispatch errors, however I haven't ran a large search yet (that I know of) and have rebooted Splunk a bunch of times. Dispatch folder is completely empty.

0 Karma

sandeeprachuri
Path Finder

@nick405060 , It's strange really. Can you post those errors?

Also, make sure there are no special characters inserted in .conf files. Check recently changed .conf files. I usually press "CTRL + C/V/S" while doing changes in VI editor.

I had this issue sometime back, After the restart Splunk web was unavailable. It took sometime for me to figure out the error.

0 Karma

nick405060
Motivator

Thanks a ton for helping me out. Replaced all conf files. Here are the current 7 messages Splunk gives me:

Dispatch Command: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch.
10/31/2018, 3:19:00 PM
Audit event generator: Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly.
10/31/2018, 3:15:18 PM
Failed to start KV Store process. See mongod.log and splunkd.log for details.
10/31/2018, 12:23:33 PM
Splunk has found 34 orphaned searches owned by 1 unique disabled users.Click to view the orphaned scheduled searches. Reassign them to a valid user to re-enable or alternatively disable the searches.
10/31/2018, 12:23:33 PM
Disk Monitor: The index processor has paused data flow. Current free disk space on partition '/' has fallen to 4297MB, below the minimum of 5000MB. Data writes to index path '/opt/splunk/var/lib/splunk/audit/db'cannot safely proceed. Increase free disk space on partition '/' by removing or relocating data. Learn more.
10/31/2018, 12:23:32 PM
KV Store changed status to failed. KVStore process terminated.
10/31/2018, 12:23:32 PM
KV Store process terminated abnormally (exit code 1, status exited with code 1). See mongod.log and splunkd.log for details.
10/31/2018, 12:23:32 PM

0 Karma

nick405060
Motivator

(And I know it looks like there's a disk space problem, but I provisioned 70GB on disk 1 and 500GB on disk 2 of my VM, and there's nothing else on the server besides the clean Splunk instance I just installed, so I'm not sure how that is contributing)

0 Karma

nick405060
Motivator

fdisk -l:

Disk /dev/sda: 70 GiB, 75161927680 bytes, 146800640 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc1cc14d8

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 1499135 1497088 731M 83 Linux
/dev/sda2 1501182 146798591 145297410 69.3G 5 Extended
/dev/sda5 1501184 146798591 145297408 69.3G 8e Linux LVM

Disk /dev/sdb: 500 GiB, 536870912000 bytes, 1048576000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/1ABC--ABC01--AB1--vg-root: 8.4 GiB, 8975810560 bytes, 17530880 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/1SPL--INF01--DC1--vg-swap_1: 976 MiB, 1023410176 bytes, 1998848 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...