Why are ad-hoc jobs not expiring in a Splunk 6.2.6 search head cluster?

[kmugglet]

We've recently moved our production search heads to a search head cluster, since last week (6.2.6?) I have noticed that any ad-hoc jobs (via REST API or WEB UI) are not expiring and quickly stack up.

I've checked the limits.conf and savedsearches.conf, and have confirmed that the ttl's are set to 600 seconds or less.
This only happens in a clustered environment. We have dev servers running the exact same searches without issue.

In the job inspector info below, I can see that the job was created yesterday. It has completed successfully and has TTLs of 600 seconds, so why is it still there?? The expiration time just updates to now whenever I refresh the jobs list.

Is there some config specific to SHC that sets the TTL for completed jobs?

This is an example from the job inspector

Search job inspector
This search has completed and has returned 1,192 results by scanning 4,986 events in 8.404 seconds.

The following messages were returned by the search subsystem:

INFO: Your timerange was substituted based on your search string
(SID: 1443331931.9_ECBEC051-E014-4F98-95CC-90307C8D43D7) search.log

Execution costs

Duration (seconds)      Component   Invocations Input count Output count
    0.16    command.addinfo 158 4,808   4,808
    0.02    command.eval    5   19,877  19,877
    0.07    command.fields  158 4,808   4,808
    0.00    command.presort 1   1,192   1,192
    0.91    command.prestats    158 4,808   4,742
    21.69   command.search  317 8,347   10,808
    6.22    command.search.rawdata  149 -   -
    0.74    command.search.kv   149 -   -
    0.46    command.search.typer    149 4,808   4,808
    0.32    command.search.filter   309 -   -
    0.15    command.search.calcfields   149 4,986   4,986
    0.15    command.search.fieldalias   149 4,986   4,986
    0.09    command.search.lookups  149 4,986   4,986
    0.08    command.search.tags 149 4,808   4,808
    0.05    command.search.summary  157 -   -
    0.00    command.search.index.usec_1_8   38,306  -   -
    0.00    command.search.index.usec_32768_262144  2   -   -
    0.00    command.search.index.usec_4096_32768    838 -   -
    0.00    command.search.index.usec_512_4096  68  -   -
    0.00    command.search.index.usec_64_512    153 -   -
    0.00    command.search.index.usec_8_64  1,053   -   -
    0.00    command.sort    1   1,192   1,192
    0.12    command.stats   1   -   3,539
    0.61    command.stats.execute_input 159 -   -
    0.15    command.stats.execute_output    1   -   -
    0.00    command.table   1   1,192   2,384
    0.00    dispatch.check_disk_usage   1   -   -
    0.08    dispatch.createdSearchResultInfrastructure  1   -   -
    0.06    dispatch.evaluate   1   -   -
    0.06    dispatch.evaluate.search    2   -   -
    0.00    dispatch.evaluate.eval  5   -   -
    0.00    dispatch.evaluate.stats 2   -   -
    0.00    dispatch.evaluate.sort  1   -   -
    0.00    dispatch.evaluate.table 1   -   -
    7.32    dispatch.fetch  159 -   -
    0.00    dispatch.localSearch    1   -   -
    0.32    dispatch.parserThread   157 -   -
    0.00    dispatch.stream.local   1   -   -
    22.39   dispatch.stream.remote  157 -   32,716,802
    0.03    dispatch.writeStatus    12  -   -
    0.26    startup.configuration   9   -   -
    3.49    startup.handoff 9   -   -
Search job properties

bundleVersion   4206439116757466412
canSummarize    1
**createTime    2015-09-27T15:32:11.000+10:00**
cursorTime  1970-01-01T10:00:00.000+10:00
defaultSaveTTL  604800
**defaultTTL    600**
delegate    None
diskUsage   188416
**dispatchState DONE**
doneProgress    1.0
dropCount   0
eai:acl 
{
    "app": "apm_snpm", 
    "can_write": "1", 
    "modifiable": "1", 
    "owner": "username", 
    "perms": {
        "read": [
            "username"
        ], 
        "write": [
            "username"
        ]
    }, 
    "sharing": "global", 
    **"ttl": "600"**
}
earliestTime    2015-09-13T00:00:00.000+10:00
eventAvailableCount 0
eventCount  4808
eventFieldCount 0
eventIsStreaming    True
eventIsTruncated    True
eventSearch search (eventtype="summary_cvc_util") eventtype=summary_sanitized earliest=1442066400 latest=1443276000 CVC_ID="CVC000000123456"
eventSorting    none
isBatchModeSearch   True
isDone  True
isFailed    False
isFinalized False
isGoodSummarizationCandidate    1
isPaused    False
isPreviewEnabled    False
isRealTimeSearch    False
isRemoteTimeline    False
isSaved False
isSavedSearch   False
isTimeCursored  1
isZombie    False
keywords    cvc_id::cvc000000123456 earliest::1442066400 eventtype::summary_cvc_util eventtype::summary_sanitized latest::1443276000 tclass::4
label   None
latestTime  2015-09-27T00:00:00.000+10:00
modifiedTime    2015-09-28T10:10:59.478+10:00
normalizedSearch    litsearch foo bar
numPreviews 0
pid 19020
priority    5
reduceSearch    foo bar
request 
{
    "namespace": "apm_snpm", 
    "search": "| savedsearch cvc_util_up_down_green cvcid=\"CVC000000123456\" startdate=\"1442066400\" enddate=\"1443276000\" | search tclass=4 | sort 0 date                     | table date, ACCESS_SEEKER_ID, CSA_ID, POI_CODE, POI_STATE, CVC_ID, tclass, bandwidth, inboundUtilizationPcnt, inboundThroughputMbps, inboundDroppedPcnt, inboundDroppedMbps, outboundUtilizationPcnt, outboundThroughputMbps, outboundDroppedPcnt, outboundDroppedMbps"
}
resultCount 1192
resultIsStreaming   False
resultPreviewCount  1192
runDuration 8.404
runtime 
{
    "auto_cancel": "0", 
    "auto_pause": "0"
}
scanCount   4986
search  | savedsearch cvc_util_up_down_green cvcid="CVC000000123456" startdate="1442066400" enddate="1443276000" | search tclass=4 | sort 0 date | table date, ACCESS_SEEKER_ID, CSA_ID, POI_CODE, POI_STATE, CVC_ID, tclass, bandwidth, inboundUtilizationPcnt, inboundThroughputMbps, inboundDroppedPcnt, inboundDroppedMbps, outboundUtilizationPcnt, outboundThroughputMbps, outboundDroppedPcnt, outboundDroppedMbps
searchCanBeEventType    0
searchEarliestTime  1442066400.000000000
searchLatestTime    1443276000.000000000
searchProviders 
[
    "indexer1-heavy", 
    "indexer2-heavy", 
    "indexer3-heavy", 
    "indexer4-heavy", 
    "indexer5-heavy", 
    "indexer6-heavy", 
    "indexer7-heavy", 
    "indexer8-heavy", 
    "searchead1-heavy"
]
sid 1443331931.9_ECBEC051-E014-4F98-95CC-90307C8D43D7
statusBuckets   0
ttl 600
Additional info search.log 
Server info: Splunk 6.2.6, foo.bar.local:8000, Mon Sep 28 10:10:59 2015 User: keithmuggleton

[kmugglet · Accepted solution]

We've recently upgraded to 6.3.1 and this issue seems to have resolved itself.