We've recently moved our production search heads to a search head cluster, since last week (6.2.6?) I have noticed that any ad-hoc jobs (via REST API or WEB UI) are not expiring and quickly stack up. I've checked the limits.conf and savedsearches.conf, and have confirmed that the ttl's are set to 600 seconds or less. This only happens in a clustered environment. We have dev servers running the exact same searches without issue. In the job inspector info below, I can see that the job was created yesterday. It has completed successfully and has TTLs of 600 seconds, so why is it still there?? The expiration time just updates to now whenever I refresh the jobs list. Is there some config specific to SHC that sets the TTL for completed jobs? This is an example from the job inspector Search job inspector This search has completed and has returned 1,192 results by scanning 4,986 events in 8.404 seconds. The following messages were returned by the search subsystem: INFO: Your timerange was substituted based on your search string (SID: 1443331931.9_ECBEC051-E014-4F98-95CC-90307C8D43D7) search.log Execution costs Duration (seconds) Component Invocations Input count Output count 0.16 command.addinfo 158 4,808 4,808 0.02 command.eval 5 19,877 19,877 0.07 command.fields 158 4,808 4,808 0.00 command.presort 1 1,192 1,192 0.91 command.prestats 158 4,808 4,742 21.69 command.search 317 8,347 10,808 6.22 command.search.rawdata 149 - - 0.74 command.search.kv 149 - - 0.46 command.search.typer 149 4,808 4,808 0.32 command.search.filter 309 - - 0.15 command.search.calcfields 149 4,986 4,986 0.15 command.search.fieldalias 149 4,986 4,986 0.09 command.search.lookups 149 4,986 4,986 0.08 command.search.tags 149 4,808 4,808 0.05 command.search.summary 157 - - 0.00 command.search.index.usec_1_8 38,306 - - 0.00 command.search.index.usec_32768_262144 2 - - 0.00 command.search.index.usec_4096_32768 838 - - 0.00 command.search.index.usec_512_4096 68 - - 0.00 command.search.index.usec_64_512 153 - - 0.00 command.search.index.usec_8_64 1,053 - - 0.00 command.sort 1 1,192 1,192 0.12 command.stats 1 - 3,539 0.61 command.stats.execute_input 159 - - 0.15 command.stats.execute_output 1 - - 0.00 command.table 1 1,192 2,384 0.00 dispatch.check_disk_usage 1 - - 0.08 dispatch.createdSearchResultInfrastructure 1 - - 0.06 dispatch.evaluate 1 - - 0.06 dispatch.evaluate.search 2 - - 0.00 dispatch.evaluate.eval 5 - - 0.00 dispatch.evaluate.stats 2 - - 0.00 dispatch.evaluate.sort 1 - - 0.00 dispatch.evaluate.table 1 - - 7.32 dispatch.fetch 159 - - 0.00 dispatch.localSearch 1 - - 0.32 dispatch.parserThread 157 - - 0.00 dispatch.stream.local 1 - - 22.39 dispatch.stream.remote 157 - 32,716,802 0.03 dispatch.writeStatus 12 - - 0.26 startup.configuration 9 - - 3.49 startup.handoff 9 - - Search job properties bundleVersion 4206439116757466412 canSummarize 1 **createTime 2015-09-27T15:32:11.000+10:00** cursorTime 1970-01-01T10:00:00.000+10:00 defaultSaveTTL 604800 **defaultTTL 600** delegate None diskUsage 188416 **dispatchState DONE** doneProgress 1.0 dropCount 0 eai:acl { "app": "apm_snpm", "can_write": "1", "modifiable": "1", "owner": "username", "perms": { "read": [ "username" ], "write": [ "username" ] }, "sharing": "global", **"ttl": "600"** } earliestTime 2015-09-13T00:00:00.000+10:00 eventAvailableCount 0 eventCount 4808 eventFieldCount 0 eventIsStreaming True eventIsTruncated True eventSearch search (eventtype="summary_cvc_util") eventtype=summary_sanitized earliest=1442066400 latest=1443276000 CVC_ID="CVC000000123456" eventSorting none isBatchModeSearch True isDone True isFailed False isFinalized False isGoodSummarizationCandidate 1 isPaused False isPreviewEnabled False isRealTimeSearch False isRemoteTimeline False isSaved False isSavedSearch False isTimeCursored 1 isZombie False keywords cvc_id::cvc000000123456 earliest::1442066400 eventtype::summary_cvc_util eventtype::summary_sanitized latest::1443276000 tclass::4 label None latestTime 2015-09-27T00:00:00.000+10:00 modifiedTime 2015-09-28T10:10:59.478+10:00 normalizedSearch litsearch foo bar numPreviews 0 pid 19020 priority 5 reduceSearch foo bar request { "namespace": "apm_snpm", "search": "| savedsearch cvc_util_up_down_green cvcid=\"CVC000000123456\" startdate=\"1442066400\" enddate=\"1443276000\" | search tclass=4 | sort 0 date | table date, ACCESS_SEEKER_ID, CSA_ID, POI_CODE, POI_STATE, CVC_ID, tclass, bandwidth, inboundUtilizationPcnt, inboundThroughputMbps, inboundDroppedPcnt, inboundDroppedMbps, outboundUtilizationPcnt, outboundThroughputMbps, outboundDroppedPcnt, outboundDroppedMbps" } resultCount 1192 resultIsStreaming False resultPreviewCount 1192 runDuration 8.404 runtime { "auto_cancel": "0", "auto_pause": "0" } scanCount 4986 search | savedsearch cvc_util_up_down_green cvcid="CVC000000123456" startdate="1442066400" enddate="1443276000" | search tclass=4 | sort 0 date | table date, ACCESS_SEEKER_ID, CSA_ID, POI_CODE, POI_STATE, CVC_ID, tclass, bandwidth, inboundUtilizationPcnt, inboundThroughputMbps, inboundDroppedPcnt, inboundDroppedMbps, outboundUtilizationPcnt, outboundThroughputMbps, outboundDroppedPcnt, outboundDroppedMbps searchCanBeEventType 0 searchEarliestTime 1442066400.000000000 searchLatestTime 1443276000.000000000 searchProviders [ "indexer1-heavy", "indexer2-heavy", "indexer3-heavy", "indexer4-heavy", "indexer5-heavy", "indexer6-heavy", "indexer7-heavy", "indexer8-heavy", "searchead1-heavy" ] sid 1443331931.9_ECBEC051-E014-4F98-95CC-90307C8D43D7 statusBuckets 0 ttl 600 Additional info search.log Server info: Splunk 6.2.6, foo.bar.local:8000, Mon Sep 28 10:10:59 2015 User: keithmuggleton ... View more

A 3rd party developer has created a web app using the Splunk Javascript SDK to query Splunk search heads. When app connects to a search head which is part of a Search Head Cluster (SHC), it cannot retrieve a list of jobs. Therefore the app fails. I removed 1 SH from the cluster and disabled it from SHC as per the documentation. The javascript then works correctly when accessing that standalone SH via the REST API and Jacascript SDK. When pointed to the REST port of a remaining member of the SHC, the script once again fails to return a jobs list. This is the error returned when accessing the jobs list. TypeError: Cannot read property 'owner' of undefined at Object.root.namespaceFromProperties (/data/snpm/SNPM-hg/savvi/web/savvi_nodejs/node_modules/splunk-sdk/lib/utils.js:396:29) at root.Jobs.root.Collection.extend.instantiateEntity (/data/snpm/SNPM-hg/savvi/web/savvi_nodejs/node_modules/splunk-sdk/lib/service.js:3558:41) at null.instantiateEntity (/data/snpm/SNPM-hg/savvi/web/savvi_nodejs/node_modules/splunk-sdk/lib/utils.js:47:23) at root.Collection.root.Resource.extend._load (/data/snpm/SNPM-hg/savvi/web/savvi_nodejs/node_modules/splunk-sdk/lib/service.js:1257:35) at prototype.(anonymous function) (/data/snpm/SNPM-hg/savvi/web/savvi_nodejs/node_modules/splunk-sdk/lib/jquery.class.js:39:28) at null.<anonymous> (/data/snpm/SNPM-hg/savvi/web/savvi_nodejs/node_modules/splunk-sdk/lib/utils.js:47:23) at null._load (/data/snpm/SNPM-hg/savvi/web/savvi_nodejs/node_modules/splunk-sdk/lib/utils.js:47:23) at /data/snpm/SNPM-hg/savvi/web/savvi_nodejs/node_modules/splunk-sdk/lib/service.js:1342:26 at reloginIfNecessary (/data/snpm/SNPM-hg/savvi/web/savvi_nodejs/node_modules/splunk-sdk/lib/context.js:199:30) at wrappedCallback (/data/snpm/SNPM-hg/savvi/web/savvi_nodejs/node_modules/splunk-sdk/lib/http.js:176:21) However, the app is able to create search jobs which complete successfully on both standalone and clustered SHs. But on the clustered SHs it cannot tell when the job is completed. I have tried multiple variations on app owner etc etc. Basic jobs test is is below var splunkjs = require('splunk-sdk'); // Create a Service instance and log in var service = new splunkjs.Service({ username:"valid_username", password:"valid_password", scheme:"https", host:"searchhead.cluster.xyz.com", port:"8089" }); // Print installed apps to the console to verify login service.jobs().fetch(function(err, jobs) { if (err) { console.log("Error retrieving jobs: ", err); return; } console.log("Jobs:"); var jobsList = jobs.list(); for(var i = 0; i < jobsList.length; i++) { var job = jobsList[i]; console.log(" Job " + i + ": " + job.name); } }); Anyone seen this before??? ... View more

Does anyone know how to convert this savedsearch request to a REST endpoint request? |savedsearch mysearch replace_me="value" I can run a normal savedsearch "apm_cvc_qtr_bh_test" using curl in the app apm_snpm curl -k -u svc_user_bob:password https://localhost:8089/servicesNS/svc_user_bob/apm_snpm/saved/searches/apm_cvc_qtr_bh_test/dispatch -d trigger_actions=1 but i don't know how to pass the replace_me key and value Any ideas? ... View more

Is there anything I should do before using user defined eventtype in a rest api call? my username is svc_user_bob (real name changed to protect the innocent) as svc_user_bob I created the following eventtype Name: sam_service.CombinedQueueGroupEgressLogRecord Description: ds=service.CombinedQueueGroupEgressLogRecord All well and good, I can return events of that eventtype. When I do a rest call via curl curl -k -u svc_user_bob:myFakePassword https://localhost:8089/services/search/jobs -d search="search eventtype=sam_service.CombinedQueueGroupEgressLogRecord" I get a SID which in job inspector shows as failed. On looking in the search log I get the following error 12-01-2014 18:00:08.615 WARN StringSearchExpander - sid:1417417038.14 Eventtype 'sam_service.CombinedQueueGroupEgressLogRecord' does not exist or is disabled. Why? Other queries run via rest work OK but not this eventtype. I've deleted and created in several times just to make sure it's not a hanging config issue from some old eventtype with the same name. ... View more

Is there anything I should do before using user defined eventtype in a rest api call? my username is svc_user_bob (real name changed to protect the innocent) as svc_user_bob I created the following eventtype Name: sam_service.CombinedQueueGroupEgressLogRecord Description: eventtype=sam_data ds=service.CombinedQueueGroupEgressLogRecord All well and good, I can return events of that eventtype. When I do a rest call via curl curl -k -u svc_user_bob:myFakePassword https://localhost:8089/services/search/jobs -d search="search eventtype=sam_service.CombinedQueueGroupEgressLogRecord" I get a SID which in job inspector shows as failed. On looking in the search log I get the following error 12-01-2014 18:00:08.615 WARN StringSearchExpander - sid:1417417038.14 Eventtype 'sam_service.CombinedQueueGroupEgressLogRecord' does not exist or is disabled. Why? Other queries run via rest work OK but not this eventtype. I've deleted and created in several times just to make sure it's not a hanging config issue from some old eventtype with the same name. ... View more