You need to increase the timeout settings on your F5. I'm not sure where these are configured on an F5, but in haproxy I have the following values configured: timeout connect 5000 timeout client 50000 timeout server 50000 ... View more

I would suggest testing the parameters I supplied. I think the most important change in your config is the max_searches_perc, which you previously had/have set to 200%. Try the suggested settings then evaluate how your system performs. ... View more

Are you certain that the data range of the query you are using as a test falls within the date/time range of your acceleration? If you query outside of that range then your query will be as slow as if it was not accelerated. ... View more

You can perform an eval on one of the fields being returned, or add stats, timechart, etc. Any of those should make it qualify. ... View more

You won't be able to move the root filesystem to sdb, but you can extend your root VG using sdb. Though, I'm a little confused that you have three partitions on a newly presented disk. I'm guessing sdb is already being used, and you are not yet seeing the new disk. To know for certain I would need to see output from the following commands: pvs vgs lvs ... View more

Block size is defined at the OS level when you format a disk partition or logical volume. The options available are specific to a filesystem type, so not all settings will be available for all formats. You can set the block size at that point, but not after. Or at least I would not recommend trying to do so. Below is the syntax for setting block size to 4K on a logical partition in Linux using XFS. Example: mkfs.xfs -b 4096 /dev/vg00-someLV00 ... View more

Those parameters are defined in indexes.conf. If you have not set them explicitly, then the default values are assumed. You can see the state of all buckets in your index, as well as other useful information with "dbinspect". Run this from the search bar: | dbinspect index=your_index_name_here ... View more

Any changes to CPU or memory allocation on VMWare VM's require that the VM be powered off first. You can add more virtual disks, NIC's, etc. in uptime, but not memory or CPU. ... View more

Your query is not using any transforming/streaming commands and therefore does not qualify. You need to use stats, timechart, etc. This section has more detailed information: https://docs.splunk.com/Documentation/Splunk/7.2.5/Knowledge/Aboutsummaryindexing ... View more

Unfortuantely, the only way to expand the centos-root VG in this case would be to have a new disk presented to the system. Then you can add that disk to the VG with vgextend, then increase the root LV with lvextend. Unless, by chance, you have unused disks already presented to the system. You can check this by running: fdisk -l |grep sd Then look for any disks that do not have a partition created. Or you can run pvs to find any disks with a physical volume created that are not part of any VG. Those could also be used. Additionally, have you checked the size of the log files that Splunk is generating on your machine? Depending on how busy it is, and how long it's been running, those can add up quickly. Especially if you don't have logrotate configured. Try checking those with these commands (assuming /opt/splunk is your $SPLUNK_HOME): du -sh /opt/splunk/var/log/splunk/ du -sh /opt/splunk/var/log/introspection ll -h /opt/splunk/var/log/splunk/ ll -h /opt/splunk/var/log/introspection ... View more

If you deploy indexes.conf via the master at /opt/splunk/etc/master-apps/_cluster/local/indexes.conf then your indexes.conf on the peer nodes (indexers) will reside at /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf, not at /etc/system/local. In a clustered environment, I'm not a fan of deploying indexes.conf any other way, though it can be done (e.g. in a custom app). If that app gets deleted, or you inadvertently deploy with the app directory missing, or wrong permissions, etc. then you're going to have problems. The /opt/splunk/etc/master-apps/_cluster/local/ is a special directory intended just for this purpose, and is documented as such: https://docs.splunk.com/Documentation/Splunk/7.0.4/Indexer/Updatepeerconfigurations ... View more

If you deploy indexes.conf via the master at /opt/splunk/etc/master-apps/_cluster/local/indexes.conf then your indexes.conf on the peer nodes (indexers) will reside at /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf. In a clustered environment, I'm not a fan of deploying indexes.conf any other way, though it can be done (e.g. in a custom app). If that app gets deleted, or you inadvertently deploy with the app directory missing, or wrong permissions, etc. then you're going to have problems. The /opt/splunk/etc/master-apps/_cluster/local/ is a special directory intended just for this purpose, and is documented as such: https://docs.splunk.com/Documentation/Splunk/7.0.4/Indexer/Updatepeerconfigurations ... View more

Glad to help! ... View more

The deployer is for pushing applications to your search head cluster. The indexer master is for pushing indexes.conf to your indexer cluster (as well as other jobs it performs). ... View more

If you have unallocated space available on the root VG then you can extend the root LV and underlying filesystem. You can check the VG allocation with the vgs command as such: 16:37:55 # vgs VG #PV #LV #SN Attr VSize VFree vg00 2 6 0 wz--n- 119.00g 90.00g In the above example I have 90G free on vg00 (my root VG). The current size of the root LV is 6GB: 16:40:16 # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg00-rootlv00 6.0G 2.7G 3.4G 45% / This means I can extend the size of my root LV by any increment up to around ~90GB (size of unallocated space in vg00). If I wanted to extend it by 10GB for example, I would use the following commands: lvextend -L+10G /dev/mapper/vg00-rootlv00 xfs_growfs /dev/mapper/vg00-rootlv00 ... View more

There are many solutions for implementing container orchestration with Docker. It just depends on what you want to accomplish and what you feel comfortable with. The most popular options would be Kuberenetes (my favorite), DC/OS, and Docker Swarm. ... View more

Always happy to help, but seems odd that you accepted your own answer after all that. ... View more

The documentation is correct that you need 3 nodes for a search head cluster. That is the only way to avoid split-brain situations. However, clustering your search heads is not required. There is no reason you can't have one to many individual search heads attached to your indexers. Though they may be doing duplicate work as your search artifacts will not be replicated. Based on the description of your requirements I would suggest that your environment is a prime candidate for running Splunk in containers. That topology will require less hardware while allowing you to scale, minimize resources, and also realize high availability. ... View more

Oh, so you were able to get them uploaded back to the deployer, I think I misread. If you SCP'd them up as root you'll need to change ownerships back to splunk on those directories on the deployer. chown -RP splunk:splunk /opt/splunk/etc/shcluster/apps chown -RP splunk:splunk /opt/splunk/etc/shcluster/users Splunk won't recognize any files or directories not owned by the splunk user. Once you make the changes it should allow you to deploy again. Don't forget to examine/remove any local files that you might have transferred up to the deployer. ... View more

Glad to help and good to hear you got it working! ... View more

Try this: cp -RP $SPLUNK_HOME/etc/apps/* deployer_name:/$SPLUNK_HOME/etc/shcluster/apps/ ... View more

Just recreate the directories: mkdir -p /opt/splunk/etc/shcluster/apps mkdir -p /opt/splunk/etc/shcluster/users Then SCP your app/users from the search head to the deployer: scp -RP $SPLUNK_HOME/etc/apps/app_name deployer_name:/$SPLUNK_HOME/etc/shcluster/apps/ scp -RP $SPLUNK_HOME/etc/apps/users deployer_name:/$SPLUNK_HOME/etc/shcluster/users/ ... View more

Assuming you have not pushed out a new version since deleting the directory on the deployer, just copy the contents from the app directory back to the deployer. From the search head: scp -RP $SPLUNK_HOME/etc/apps/ deployer_name:/$SPLUNK_HOME/etc/shcluster/apps/ After copying over, you probably want to check the contents of the local directory. It can/will contain local changes made by users on the search heads. If you deploy again with that directory intact, it will merge those into the default directory of you app. ... View more