About nocostk

nocostk · ‎11-16-2012

That looks like it. 13,000 in there today, and it is pretty close to what is in the banner now. Looks like we have a few intense searches going on.

nocostk · ‎11-15-2012

I have read : http://splunk-base.splunk.com/answers/29551/too-many-search-jobs-found-in-the-dispatch-directory Yet the problem will not go away. Our dispatch directory is at : [(prod) root@splunksearch01.prod.ostk.com ~]# ls -l /opt/splunk/var/run/splunk/dispatch total 0 [(prod) root@splunksearch01.prod.ostk.com ~]# We still have cleared out the files from both dispatch and dispatchtmp, but still continue to get the message. Running the clean-dispatch gives : [(prod) root@splunksearch01.prod.ostk.com ~]# /opt/splunk/bin/splunk cmd splunkd clean-dispatch /opt/splunk/var/run/splunk/dispatch ...... (LOTS of these) ..... Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/scheduler_nobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1345788000_8c88119f3789ab7b to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1345788000_8c88119f3789ab7b. Invalid cross-device link Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1346392800_b2605fff19a01988 to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1346392800_b2605fff19a01988. Invalid cross-device link Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6 to /opt/splunk/var/run/splunk/dispatch/schedulernobody_search_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6. Invalid cross-device link total: 2199, moved: 0, failed: 200, remaining: 2199 job directories from /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch to /opt/splunk/var/run/splunk/dispatch [(prod) root@splunksearch01.prod.ostk.com ~]# It looks like it tells me we have 2199 jobs, but I can't find them anywhere. Ideas?

nocostk · ‎07-20-2012

I had one of my indexers go down a couple weeks back. Since then each of my forwarders is trying to send events to that indexer but failing with errors like WARN TcpOutputFd - Connect to 10.1.4.183:9998 failed. Connection refused So I modified my outputs.conf to remove that target indexer and restarted the forwarder (heavy). However, I'm still seeing that error. Also I'm seeing queueing errors on the forwarder: INFO TailingProcessor - Could not send data to output queue (parsingQueue), retrying... I'm thinking that the queue has retained the old indexer and is continuing to attempt event delivery. As I noted, cycling splunkd on the forwarder doesn't make a difference. I also think this is causing delays in sending events to my other indexers (5-15 minutes will go by before any events show up). How can I alleviate this problem (aside from standing up an indexer on the failed IP noted above)?

nocostk · ‎05-30-2012

Looking at alert_actions.conf you could use $search$ to reference the search used. Also, further down the page it references some other variables you may be able to use.

nocostk · ‎05-22-2012

Never mind... I'm a moron. You do slap the /acl handler. I had a problem with one of my variable.

nocostk · ‎05-22-2012

I'm trying to script something out to create an event type and then set the permissions on it. I've got the creation down just fine: curl -s -k -H "Authorization: Splunk <authstring>" https://splunksearch:8089/servicesNS/nobody/search/saved/eventtypes -d name=SA-1234 -d search='"host=web01*"' -d tags=alert-shop However, I'm unable to set the permissions using the above URI. Scouring through the documentation it seems I need to slap the acl of the object via: curl -s -k -H "Authorization: Splunk <authstring>" https://splunksearch:8089/servicesNS/nobody/search/saved/eventtypes/SA-1234/acl -d perms.read=* -d perms.write=admin,power -d sharing=app" However, the API returns the following : <msg type="ERROR">In handler 'eventtypes': Argument "perms.read" is not supported by this handler.</msg> If I remove perms.read it will just complain about another (e.g. sharing=app). How do I properly set the permissions via the API in Splunk 4.3.2?

nocostk · ‎03-01-2012

. . . thanks for making me look like an idiot! 😉 I looked last night - but maybe I was too tired, promise. Thanks, ChrisG.

nocostk · ‎03-01-2012

I'm using Splunk 4.2.3. Right now I have about 250 eventtypes I need to delete. I really don't want to do it via the webUI - is it possible through the REST api?

nocostk · ‎12-15-2011

Can anyone explain why the following two searches produce different results? It seems to me that the output should be the same - but maybe the _internal lists kb usage a little different? source="/opt/ShoppingSite/work/logs/ShoppingSite.log"|eval record_length=len(_raw)|stats sum(record_length) as record_length | eval record_length=record_length/1024 158002.983398 index="_internal" source="*metrics.log" per_source_thruput series="/opt/shoppingsite/work/logs/shoppingsite.log" | stats sum(kb) 64048.911130

nocostk · ‎04-29-2011

When running these two commands on a host I get different values. What (if any) is the difference between the two? # /opt/splunk/bin/splunk show default-hostname Default hostname for data inputs: sapi09. # /opt/splunk/bin/splunk show servername Server name: sapi09.mydomain.com-$USER

nocostk · ‎04-28-2011

Hmm, the only thing I see in there that may address this is: capability::admin_all_objects - but apparently that's like giving root access?

nocostk · ‎04-28-2011

I'd like to grant my Power users access to change eventtypes, savedsearches, etc. from private to app-specific/global. It seems that that is only granted to admins?

nocostk · ‎04-28-2011

Alright - that's what I was afraid of. Thanks for your help, lephino.

nocostk · ‎04-27-2011

I'm running some tests with searchhead pooling and scheduled searches. I'm running a saved search every minute and the results are being sent from one search head only. They're only coming from the second search head when the first search head is down (restarted, etc.). My question is: how does Splunk determine which searchhead to run a scheduled search from? Is this configurable (round-robin, least connections, pegged to specific host, etc.)?

nocostk · ‎04-25-2011

Currently I have two separate search heads. I'm trying to consolidate my configuration files so I can make use of searchhead pooling new in v. 4.2. What would be the easiest way to do this? I could copy/paste but it looks a little daunting. I also need to synchronize the files in SPLUNK_HOME/etc/users. Does anyone have any suggestions?

nocostk · ‎04-20-2011

Looks like some of the eventtypes (and tags) were disabled. I think they were before the 4.2 upgrade but 4.1x didn't really complain? I enabled them and things are working now.

nocostk · ‎04-20-2011

Is it possible to set a default permission for searches/eventtypes/tags/field extractions/etc.? When we're creating these various objects we usually immediately change the permissions from private to application (search). Is there a way this can be done by default or automatically?

nocostk · ‎04-20-2011

I'd like to use SHP as a means of keeping my eventtypes.conf and tags.conf in sync. Sometimes it's a bit tiresome to continually ask developers to create their tags/eventtypes in both locations. If someone has any other ideas I'm all ears.

nocostk · ‎04-19-2011

I have a two search heads - but they perform different tasks. One head is for running scheduled searches and the other is for interactive searches. I'd like to utilize search head pooling - but I don't want to share any of the savedsearches.conf files. Is this possible?

nocostk · ‎03-30-2011

There are both. I checked the tar'd bundle and in apps/search/local/{tags.conf,eventtypes.conf} there is reference to ShoppingSite_Errors. So they do exist on the indexer - but I'm still not clear why I'm getting the error that it is unable to find it.

Posts	61
Solutions	5
Karma Given	6
Karma Received	39
Member Since	‎06-14-2010

Online Status	Offline
Date Last Visited	‎06-05-2020 02:02 AM

Too many search jobs found in the dispatch directo...

Purge queue on forwarder / indexer down

Modify event types via API (curl)

Deleting eventtypes using REST api?

License usage discrepancy using two different sear...

Difference between 'show default-hostname' and 'sh...

Power user permission to change ownership of objec...

Searchhead pooling and scheduled searches.

Merging eventtypes.conf (and other config files).

Default permissions for searches/tags/eventtypes/e...

Re: Too many search jobs found in the dispatch dir...

Too many search jobs found in the dispatch directo...

Purge queue on forwarder / indexer down

Re: Alerts Subject Line

Re: Modify event types via API (curl)

Modify event types via API (curl)

Re: Deleting eventtypes using REST api?

Deleting eventtypes using REST api?

License usage discrepancy using two different sear...

Difference between 'show default-hostname' and 'sh...

Re: Power user permission to change ownership of o...

Power user permission to change ownership of objec...

Re: Merging eventtypes.conf (and other config file...

Searchhead pooling and scheduled searches.

Merging eventtypes.conf (and other config files).

Re: Unable to find an eventtype <eventtype>

Default permissions for searches/tags/eventtypes/e...

Re: Partial search head pooling

Partial search head pooling

Re: Unable to find an eventtype <eventtype>