I had one of my indexers go down a couple weeks back. Since then each of my forwarders is trying to send events to that indexer but failing with errors like WARN TcpOutputFd - Connect to 10.1.4.183:9998 failed. Connection refused So I modified my outputs.conf to remove that target indexer and restarted the forwarder (heavy). However, I'm still seeing that error. Also I'm seeing queueing errors on the forwarder: INFO TailingProcessor - Could not send data to output queue (parsingQueue), retrying... I'm thinking that the queue has retained the old indexer and is continuing to attempt event delivery. As I noted, cycling splunkd on the forwarder doesn't make a difference. I also think this is causing delays in sending events to my other indexers (5-15 minutes will go by before any events show up). How can I alleviate this problem (aside from standing up an indexer on the failed IP noted above)? ... View more

I'm trying to script something out to create an event type and then set the permissions on it. I've got the creation down just fine: curl -s -k -H "Authorization: Splunk <authstring>" https://splunksearch:8089/servicesNS/nobody/search/saved/eventtypes -d name=SA-1234 -d search='"host=web01*"' -d tags=alert-shop However, I'm unable to set the permissions using the above URI. Scouring through the documentation it seems I need to slap the acl of the object via: curl -s -k -H "Authorization: Splunk <authstring>" https://splunksearch:8089/servicesNS/nobody/search/saved/eventtypes/SA-1234/acl -d perms.read=* -d perms.write=admin,power -d sharing=app" However, the API returns the following : <msg type="ERROR">In handler &apos;eventtypes&apos;: Argument &quot;perms.read&quot; is not supported by this handler.</msg> If I remove perms.read it will just complain about another (e.g. sharing=app). How do I properly set the permissions via the API in Splunk 4.3.2? ... View more

When running these two commands on a host I get different values. What (if any) is the difference between the two? # /opt/splunk/bin/splunk show default-hostname Default hostname for data inputs: sapi09. # /opt/splunk/bin/splunk show servername Server name: sapi09.mydomain.com-$USER ... View more

I'd like to grant my Power users access to change eventtypes, savedsearches, etc. from private to app-specific/global. It seems that that is only granted to admins? ... View more

I'm running some tests with searchhead pooling and scheduled searches. I'm running a saved search every minute and the results are being sent from one search head only. They're only coming from the second search head when the first search head is down (restarted, etc.). My question is: how does Splunk determine which searchhead to run a scheduled search from? Is this configurable (round-robin, least connections, pegged to specific host, etc.)? ... View more

Is it possible to set a default permission for searches/eventtypes/tags/field extractions/etc.? When we're creating these various objects we usually immediately change the permissions from private to application (search). Is there a way this can be done by default or automatically? ... View more

I have a two search heads - but they perform different tasks. One head is for running scheduled searches and the other is for interactive searches. I'd like to utilize search head pooling - but I don't want to share any of the savedsearches.conf files. Is this possible? ... View more

Currently I have two search heads in a pooled configuration. However, I'm seeing an error where a particular user is unable to successfully log in completely. Looking at the audit logs the person can log in - but is unable to view anything: 03-30-2011 10:21:49.468 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:21:49.468, user=myuser, action=login attempt, info=succeeded][n/a] 03-30-2011 10:21:49.686 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:21:49.686, user=myuser, action=rest_properties_get, info=denied REST: /properties/app][n/a] 03-30-2011 10:21:53.828 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:21:53.828, user=myuser, action=search, info=denied REST: /search/timeparser/tz][n/a] 03-30-2011 10:21:59.430 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:21:59.430, user=myuser, action=login attempt, info=succeeded][n/a] 03-30-2011 10:21:59.624 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:21:59.624, user=myuser, action=rest_properties_get, info=denied REST: /properties/app][n/a] 03-30-2011 10:22:12.366 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:22:12.366, user=myuser, action=rest_properties_get, info=denied REST: /properties/app][n/a] 03-30-2011 10:23:09.804 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:23:09.803, user=myuser, action=rest_properties_get, info=denied REST: /properties/app][n/a] 03-30-2011 10:30:55.989 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:30:55.989, user=myuser, action=login attempt, info=succeeded][n/a] 03-30-2011 10:30:56.230 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:30:56.230, user=myuser, action=rest_properties_get, info=denied REST: /properties/app][n/a] 03-30-2011 10:33:27.240 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:33:27.240, user=myuser, action=rest_properties_get, info=denied REST: /properties/app][n/a] 03-30-2011 10:33:35.092 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:33:35.092, user=myuser, action=rest_properties_get, info=denied REST: /properties/app][n/a] 03-30-2011 10:34:03.224 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:34:03.224, user=myuser, action=rest_properties_get, info=denied REST: /properties/app][n/a] 03-30-2011 10:34:46.088 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:34:46.088, user=myuser, action=login attempt, info=succeeded][n/a] 03-30-2011 10:34:46.364 -0600 INFO AuditLogger - Audit:[timestamp=03-30-2011 10:34:46.364, user=myuser, action=rest_properties_get, info=denied REST: /properties/app][n/a] Login/access works fine on the other pooled search head. The dates are in sync and this instance runs as root (so no permission issues). ... View more