Why am I unable to delete data from a certain indexer in a cluster with the delete command in a search?

[appdev84]

I have duplicated records that I am trying to delete in Splunk.

I am using Splunk 6.5 with Search Head Clustering, and Clustered Indexers. My user has the can_delete option checked off and I am able to delete records, but for some reason I have 35 records that refuse to be deleted. They all seem to be on the same indexer. I am able to search for the records without the delete option and they come up within a few seconds, but when I put the delete option, it just searches for a long time and does not delete the records.

My search looks like this:

index=main earliest=11/30/2016:11:33:00 latest=11/30/2016:11:33:20 | eval delete_id=_cd."|".index."|".splunk_server | search (( delete_id="109:40078965|main|cp-vm3" ) OR ( delete_id="109:40084329|main|cp-vm3" ) OR ( delete_id="109:40085221|main|cp-vm3" ) OR ( delete_id="109:40088243|main|cp-vm3" ) OR ( delete_id="109:40088070|main|cp-vm3" ) OR ( delete_id="109:40085858|main|cp-vm3" ) OR ( delete_id="109:40086155|main|cp-vm3" ) OR ( delete_id="109:40088053|main|cp-vm3" ) OR ( delete_id="109:40085602|main|cp-vm3" ) OR ( delete_id="109:40066343|main|cp-vm3" ) OR ( delete_id="109:40068493|main|cp-vm3" ) OR ( delete_id="109:40073891|main|cp-vm3" ) OR ( delete_id="109:40077210|main|cp-vm3" ) OR ( delete_id="109:40069880|main|cp-vm3" ) OR ( delete_id="109:40066724|main|cp-vm3" ) OR ( delete_id="109:40067052|main|cp-vm3" ) OR ( delete_id="109:40067280|main|cp-vm3" ) OR ( delete_id="109:40070422|main|cp-vm3" ) OR ( delete_id="109:40072184|main|cp-vm3" ) OR ( delete_id="109:40067032|main|cp-vm3" ) OR ( delete_id="109:40072168|main|cp-vm3" ) OR ( delete_id="109:40086139|main|cp-vm3" ) OR ( delete_id="109:40084253|main|cp-vm3" ) OR ( delete_id="109:40084615|main|cp-vm3" ) OR ( delete_id="109:40068545|main|cp-vm3" ) OR ( delete_id="109:40087829|main|cp-vm3" ) OR ( delete_id="109:40066808|main|cp-vm3" ) OR ( delete_id="109:40067264|main|cp-vm3" ) OR ( delete_id="109:40069296|main|cp-vm3" ) OR ( delete_id="109:40079749|main|cp-vm3" ) OR ( delete_id="109:40079733|main|cp-vm3" ) OR ( delete_id="109:40081521|main|cp-vm3" ) OR ( delete_id="109:40084269|main|cp-vm3" ) OR ( delete_id="109:40086336|main|cp-vm3" )) | delete

In the search job inspector, it shows this message
This search is still running and is approximately 100% complete.
(SID: 1480601649.181) search.log

The actual search.log

12-01-2016 14:14:09.388 INFO  dispatchRunner - Search process mode: preforked (reused process)
12-01-2016 14:14:09.388 WARN  DistributedInfoSingleton - Failed to read symptoms of peer=devsh-vm
12-01-2016 14:14:09.388 INFO  dispatchRunner - registering build time modules, count=1
12-01-2016 14:14:09.388 INFO  dispatchRunner - registering search time components of build time module name=vix
12-01-2016 14:14:09.389 INFO  BundlesSetup - Setup stats for /opt/splunk/etc: wallclock_elapsed_msec=30, cpu_time_used=0.032, shared_services_generation=2, shared_services_population=1
12-01-2016 14:14:09.389 INFO  UserManager - Setting user context: splunk-system-user
12-01-2016 14:14:09.389 INFO  UserManager - Done setting user context: NULL -> splunk-system-user
12-01-2016 14:14:09.389 INFO  UserManager - Unwound user context: splunk-system-user -> NULL
12-01-2016 14:14:09.389 INFO  UserManager - Setting user context: admin
12-01-2016 14:14:09.389 INFO  UserManager - Done setting user context: NULL -> admin
12-01-2016 14:14:09.389 INFO  dispatchRunner - search context: user="admin", app="search", bs-pathname="/opt/splunk/etc"
12-01-2016 14:14:09.390 INFO  SearchParser - PARSING: search index=main earliest=11/30/2016:11:33:00 latest=11/30/2016:11:33:20 | eval delete_id=_cd."|".index."|".splunk_server | search (( delete_id="109:40078965|main|cp-vm3" ) OR ( delete_id="109:40084329|main|cp-vm3" ) OR ( delete_id="109:40085221|main|cp-vm3" ) OR ( delete_id="109:40088243|main|cp-vm3" ) OR ( delete_id="109:40088070|main|cp-vm3" ) OR ( delete_id="109:40085858|main|cp-vm3" ) OR ( delete_id="109:40086155|main|cp-vm3" ) OR ( delete_id="109:40088053|main|cp-vm3" ) OR ( delete_id="109:40085602|main|cp-vm3" ) OR ( delete_id="109:40066343|main|cp-vm3" ) OR ( delete_id="109:40068493|main|cp-vm3" ) OR ( delete_id="109:40073891|main|cp-vm3" ) OR ( delete_id="109:40077210|main|cp-vm3" ) OR ( delete_id="109:40069880|main|cp-vm3" ) OR ( delete_id="109:40066724|main|cp-vm3" ) OR ( delete_id="109:40067052|main|cp-vm3" ) OR ( delete_id="109:40067280|main|cp-vm3" ) OR ( delete_id="109:40070422|main|cp-vm3" ) OR ( delete_id="109:40072184|main|cp-vm3" ) OR ( delete_id="109:40067032|main|cp-vm3" ) OR ( delete_id="109:40072168|main|cp-vm3" ) OR ( delete_id="109:40086139|main|cp-vm3" ) OR ( delete_id="109:40084253|main|cp-vm3" ) OR ( delete_id="109:40084615|main|cp-vm3" ) OR ( delete_id="109:40068545|main|cp-vm3" ) OR ( delete_id="109:40087829|main|cp-vm3" ) OR ( delete_id="109:40066808|main|cp-vm3" ) OR ( delete_id="109:40067264|main|cp-vm3" ) OR ( delete_id="109:40069296|main|cp-vm3" ) OR ( delete_id="109:40079749|main|cp-vm3" ) OR ( delete_id="109:40079733|main|cp-vm3" ) OR ( delete_id="109:40081521|main|cp-vm3" ) OR ( delete_id="109:40084269|main|cp-vm3" ) OR ( delete_id="109:40086336|main|cp-vm3" )) | delete
12-01-2016 14:14:09.390 INFO  ISplunkDispatch - Not running in splunkd. Bundle replication not triggered.
12-01-2016 14:14:09.471 INFO  UserManager - Setting user context: admin
12-01-2016 14:14:09.471 INFO  UserManager - Done setting user context: NULL -> admin
12-01-2016 14:14:09.482 INFO  CalcFieldProcessor - Found valid eval expression for field 'chain_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),1,3))
12-01-2016 14:14:09.482 INFO  CalcFieldProcessor - Found valid eval expression for field 'store_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),4,7))
12-01-2016 14:14:09.485 INFO  SearchProcessor - Building search filter
12-01-2016 14:14:09.497 WARN  LookupOperator - Unable to find property=filename for lookup=world_timezones will attempt to use implicit filename.
12-01-2016 14:14:09.497 WARN  LookupOperator - No valid lookup found for lookup=world_timezones
12-01-2016 14:14:09.497 ERROR LookupOperator - The lookup table 'world_timezones' does not exist. It is referenced by configuration 'host::catalinavaultkafka'.
12-01-2016 14:14:09.498 INFO  StringSearchExpander - calculated_field="index" not expanded in comparison_expression="index=main". calc_field_processor!=null, negated=false (negation depth=0)
12-01-2016 14:14:09.498 INFO  StringSearchExpander - calculated_field="_time" not expanded in comparison_expression="_time>=1480523580.000". calc_field_processor!=null, negated=false (negation depth=0)
12-01-2016 14:14:09.498 INFO  StringSearchExpander - calculated_field="_time" not expanded in comparison_expression="_time<1480523600.000". calc_field_processor!=null, negated=false (negation depth=0)
12-01-2016 14:14:09.585 INFO  SearchOperator:kv - name=EXTRACT-GUID, can_use_jit=1, regex: (?i)(?!=\w)(?:objectguid|guid)\s*=\s*(?[\w\-]+)
12-01-2016 14:14:09.585 INFO  SearchOperator:kv - name=EXTRACT-SID, can_use_jit=1, regex: objectSid\s*=\s*(?\S+)
12-01-2016 14:14:09.586 INFO  SearchOperator:kv - name=ad-kv, can_use_jit=1, regex: (?<_KEY_1>[\w-]+)=(?<_VAL_1>[^\r\n]*)
12-01-2016 14:14:09.595 INFO  SearchOperator:kv - name=access-extractions, can_use_jit=1, regex: ^(?P\S+)\s++(?P\S+)\s++(?P\S+)\s++\[(?[^\]]*+)\]\s++"\s*+(?P[^\s"]++)?(?:\s++(?(?:(?\w++://[^/\s"]++))?+(?(?:/++(?(?:\\"|[^\s\?/"])++)/++)?(?:(?:\\"|[^\s\?/"])*+/++)*(?[^\s\?/]+)?)(?:\?(?[^\s]*))?)(?:\s++(?P[^\s"]++))*)?\s*+"\s++(?P\S+)\s++(?P\S+)(?:\s++"(?(?:(?\w++://[^/\s"]++))?+[^"]*+)"(?:\s++"(?[^"]*+)"(?:\s++"(?[^"]*+)")?+)?+)?(?P.*)
12-01-2016 14:14:09.595 INFO  SearchOperator:kv - name=syslog-extractions, can_use_jit=1, regex: \s([^\s\[]+)(?:\[(\d+)\])?:\s
12-01-2016 14:14:09.596 INFO  SearchOperator:kv - name=db2, can_use_jit=1, regex: ([A-Z]+) *: (.*?)(?=\n|$| +[A-Z]+ *:)
12-01-2016 14:14:09.597 INFO  SearchOperator:kv - name=EXTRACT-extract_spent, can_use_jit=1, regex: \s(?\d+(\.\d+)?)ms$
12-01-2016 14:14:09.597 INFO  SearchOperator:kv - name=EXTRACT-1, can_use_jit=1, regex: (?<_KEY_1>\S+)::(?<_VAL_1>\S+)
12-01-2016 14:14:09.598 INFO  SearchOperator:kv - name=bracket-space, can_use_jit=1, regex: \[(\S+) (.*?)\]
12-01-2016 14:14:09.599 INFO  SearchOperator:kv - name=EXTRACT-fields, can_use_jit=1, regex: (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P[^ ]*)\s+(?P[^ ]+) - (?P.+)
12-01-2016 14:14:09.600 INFO  SearchOperator:kv - name=sendmail-extractions, can_use_jit=1, regex: sendmail\[(\d+)\]: (\w+):
12-01-2016 14:14:09.600 INFO  SearchOperator:kv - name=tcpdump-endpoints, can_use_jit=1, regex: (\d+\.\d+\.\d+\.\d+):(\d+) -> (\d+\.\d+\.\d+\.\d+):(\d+)
12-01-2016 14:14:09.601 INFO  SearchOperator:kv - name=colon-kv, can_use_jit=1, regex: (?<= )([A-Za-z]+): ?((0x[A-F\d]+)|\d+)(?= |\n|$)
12-01-2016 14:14:09.620 INFO  SearchOperator:kv - name=EXTRACT-severity,logger, can_use_jit=1, regex: .*?(?[A-Z]+) ((?[^\s]+) \-)*
12-01-2016 14:14:09.627 INFO  SearchOperator:kv - name=EXTRACT-collection,category,object, can_use_jit=1, regex: collection=\"?(?P[^\"\n]+)\"?\ncategory=\"?(?P[^\"\n]+)\"?\nobject=\"?(?P<object>[^\"\n]+)\"?\n
12-01-2016 14:14:09.628 INFO  SearchOperator:kv - name=wel-message, can_use_jit=1, regex: (?sm)^(?<_pre_msg>.+)\nMessage=(?.+)$
12-01-2016 14:14:09.628 INFO  SearchOperator:kv - name=wel-col-kv, can_use_jit=1, regex: \n([^:\n\r]+):[ \t]++([^\n]*)
12-01-2016 14:14:09.629 INFO  SearchOperator:kv - name=EXTRACT-useragent, can_use_jit=1, regex: userAgent=(?P[^ (]+)
12-01-2016 14:14:09.629 INFO  SearchOperator:kv - name=splunk-service-extractions, can_use_jit=1, regex: (?i)^(?:[^ ]* ){2}(?P[^\s]*)\s+\[(?P\w+)]\s+(?P[^ ]+):(?P\d+) - (?P.+)
12-01-2016 14:14:09.630 INFO  SearchOperator:kv - name=extract_spent, can_use_jit=1, regex: \s(?P\d+(\.\d+)?)ms$
12-01-2016 14:14:09.631 INFO  SearchOperator:kv - name=weblogic-code, can_use_jit=1, regex: 
12-01-2016 14:14:09.637 INFO  SearchOperator:kv - name=colon-line, can_use_jit=1, regex: ^(\w+)\s*:[ \t]*(.*?)$
12-01-2016 14:14:09.637 INFO  SearchOperator:kv - name=was-trlog-code, can_use_jit=1, regex: ] ([a-fA-F0-9]{8})
12-01-2016 14:14:09.638 INFO  UnifiedSearch - base lispy: [ AND index::main ]
12-01-2016 14:14:09.639 INFO  UnifiedSearch - Processed search targeting arguments
12-01-2016 14:14:09.669 INFO  UnifiedSearch - Processed search targeting arguments
12-01-2016 14:14:09.669 INFO  SearchParser - PARSING: predelete
12-01-2016 14:14:09.669 INFO  SearchParser - PARSING: addinfo type=count label=prereport_events
12-01-2016 14:14:09.669 INFO  DispatchThread - BatchMode: allowBatchMode: 1, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0
12-01-2016 14:14:09.670 INFO  DispatchThread - required fields list to add to remote search = _bkt,_cd,index,splunk_server
12-01-2016 14:14:09.670 INFO  SearchParser - PARSING: fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server"
12-01-2016 14:14:09.670 INFO  DispatchCommandProcessor - summaryHash=c544ca20eeb5ac6c summaryId=5FE18509-5E6F-4E6C-80E9-176BA1EBB4AF_search_admin_c544ca20eeb5ac6c remoteSearch=litsearch index=main _time>=1480523580.000 _time<1480523600.000 | eval  delete_id=_cd."|".index."|".splunk_server  | search (( delete_id="109:40078965|main|cp-vm3" ) OR ( delete_id="109:40084329|main|cp-vm3" ) OR ( delete_id="109:40085221|main|cp-vm3" ) OR ( delete_id="109:40088243|main|cp-vm3" ) OR ( delete_id="109:40088070|main|cp-vm3" ) OR ( delete_id="109:40085858|main|cp-vm3" ) OR ( delete_id="109:40086155|main|cp-vm3" ) OR ( delete_id="109:40088053|main|cp-vm3" ) OR ( delete_id="109:40085602|main|cp-vm3" ) OR ( delete_id="109:40066343|main|cp-vm3" ) OR ( delete_id="109:40068493|main|cp-vm3" ) OR ( delete_id="109:40073891|main|cp-vm3" ) OR ( delete_id="109:40077210|main|cp-vm3" ) OR ( delete_id="109:40069880|main|cp-vm3" ) OR ( delete_id="109:40066724|main|cp-vm3" ) OR ( delete_id="109:40067052|main|cp-vm3" ) OR ( delete_id="109:40067280|main|cp-vm3" ) OR ( delete_id="109:40070422|main|cp-vm3" ) OR ( delete_id="109:40072184|main|cp-vm3" ) OR ( delete_id="109:40067032|main|cp-vm3" ) OR ( delete_id="109:40072168|main|cp-vm3" ) OR ( delete_id="109:40086139|main|cp-vm3" ) OR ( delete_id="109:40084253|main|cp-vm3" ) OR ( delete_id="109:40084615|main|cp-vm3" ) OR ( delete_id="109:40068545|main|cp-vm3" ) OR ( delete_id="109:40087829|main|cp-vm3" ) OR ( delete_id="109:40066808|main|cp-vm3" ) OR ( delete_id="109:40067264|main|cp-vm3" ) OR ( delete_id="109:40069296|main|cp-vm3" ) OR ( delete_id="109:40079749|main|cp-vm3" ) OR ( delete_id="109:40079733|main|cp-vm3" ) OR ( delete_id="109:40081521|main|cp-vm3" ) OR ( delete_id="109:40084269|main|cp-vm3" ) OR ( delete_id="109:40086336|main|cp-vm3" ))  | addinfo  type=count label=prereport_events | fields  keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 
12-01-2016 14:14:09.670 INFO  DispatchCommandProcessor - summaryHash=NSc41c4fa16f7c937e summaryId=5FE18509-5E6F-4E6C-80E9-176BA1EBB4AF_search_admin_NSc41c4fa16f7c937e remoteSearch=litsearch index=main _time>=1480523580.000 _time<1480523600.000 | eval delete_id=_cd."|".index."|".splunk_server | search (( delete_id="109:40078965|main|cp-vm3" ) OR ( delete_id="109:40084329|main|cp-vm3" ) OR ( delete_id="109:40085221|main|cp-vm3" ) OR ( delete_id="109:40088243|main|cp-vm3" ) OR ( delete_id="109:40088070|main|cp-vm3" ) OR ( delete_id="109:40085858|main|cp-vm3" ) OR ( delete_id="109:40086155|main|cp-vm3" ) OR ( delete_id="109:40088053|main|cp-vm3" ) OR ( delete_id="109:40085602|main|cp-vm3" ) OR ( delete_id="109:40066343|main|cp-vm3" ) OR ( delete_id="109:40068493|main|cp-vm3" ) OR ( delete_id="109:40073891|main|cp-vm3" ) OR ( delete_id="109:40077210|main|cp-vm3" ) OR ( delete_id="109:40069880|main|cp-vm3" ) OR ( delete_id="109:40066724|main|cp-vm3" ) OR ( delete_id="109:40067052|main|cp-vm3" ) OR ( delete_id="109:40067280|main|cp-vm3" ) OR ( delete_id="109:40070422|main|cp-vm3" ) OR ( delete_id="109:40072184|main|cp-vm3" ) OR ( delete_id="109:40067032|main|cp-vm3" ) OR ( delete_id="109:40072168|main|cp-vm3" ) OR ( delete_id="109:40086139|main|cp-vm3" ) OR ( delete_id="109:40084253|main|cp-vm3" ) OR ( delete_id="109:40084615|main|cp-vm3" ) OR ( delete_id="109:40068545|main|cp-vm3" ) OR ( delete_id="109:40087829|main|cp-vm3" ) OR ( delete_id="109:40066808|main|cp-vm3" ) OR ( delete_id="109:40067264|main|cp-vm3" ) OR ( delete_id="109:40069296|main|cp-vm3" ) OR ( delete_id="109:40079749|main|cp-vm3" ) OR ( delete_id="109:40079733|main|cp-vm3" ) OR ( delete_id="109:40081521|main|cp-vm3" ) OR ( delete_id="109:40084269|main|cp-vm3" ) OR ( delete_id="109:40086336|main|cp-vm3" )) | addinfo type=count label=prereport_events | fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 
12-01-2016 14:14:09.670 INFO  DispatchThread - Getting summary ID for summaryHash=NSc41c4fa16f7c937e
12-01-2016 14:14:09.691 INFO  DispatchThread - Did not find a usable summary_id, setting info._summary_mode=none, not modifying input summary_id=5FE18509-5E6F-4E6C-80E9-176BA1EBB4AF_search_admin_NSc41c4fa16f7c937e
12-01-2016 14:14:09.691 INFO  DispatchThread - Matches no summary
12-01-2016 14:14:09.691 INFO  DispatchThread - SrchOptMetrics check_query_matches_ra=221
12-01-2016 14:14:09.691 INFO  SearchParser - PARSING: search index=main earliest=11/30/2016:11:33:00 latest=11/30/2016:11:33:20 | eval delete_id=_cd."|".index."|".splunk_server | search (( delete_id="109:40078965|main|cp-vm3" ) OR ( delete_id="109:40084329|main|cp-vm3" ) OR ( delete_id="109:40085221|main|cp-vm3" ) OR ( delete_id="109:40088243|main|cp-vm3" ) OR ( delete_id="109:40088070|main|cp-vm3" ) OR ( delete_id="109:40085858|main|cp-vm3" ) OR ( delete_id="109:40086155|main|cp-vm3" ) OR ( delete_id="109:40088053|main|cp-vm3" ) OR ( delete_id="109:40085602|main|cp-vm3" ) OR ( delete_id="109:40066343|main|cp-vm3" ) OR ( delete_id="109:40068493|main|cp-vm3" ) OR ( delete_id="109:40073891|main|cp-vm3" ) OR ( delete_id="109:40077210|main|cp-vm3" ) OR ( delete_id="109:40069880|main|cp-vm3" ) OR ( delete_id="109:40066724|main|cp-vm3" ) OR ( delete_id="109:40067052|main|cp-vm3" ) OR ( delete_id="109:40067280|main|cp-vm3" ) OR ( delete_id="109:40070422|main|cp-vm3" ) OR ( delete_id="109:40072184|main|cp-vm3" ) OR ( delete_id="109:40067032|main|cp-vm3" ) OR ( delete_id="109:40072168|main|cp-vm3" ) OR ( delete_id="109:40086139|main|cp-vm3" ) OR ( delete_id="109:40084253|main|cp-vm3" ) OR ( delete_id="109:40084615|main|cp-vm3" ) OR ( delete_id="109:40068545|main|cp-vm3" ) OR ( delete_id="109:40087829|main|cp-vm3" ) OR ( delete_id="109:40066808|main|cp-vm3" ) OR ( delete_id="109:40067264|main|cp-vm3" ) OR ( delete_id="109:40069296|main|cp-vm3" ) OR ( delete_id="109:40079749|main|cp-vm3" ) OR ( delete_id="109:40079733|main|cp-vm3" ) OR ( delete_id="109:40081521|main|cp-vm3" ) OR ( delete_id="109:40084269|main|cp-vm3" ) OR ( delete_id="109:40086336|main|cp-vm3" )) | delete
12-01-2016 14:14:09.691 INFO  UnifiedSearch - Processed search targeting arguments
12-01-2016 14:14:09.692 INFO  UnifiedSearch - Processed search targeting arguments
12-01-2016 14:14:09.693 INFO  DispatchThread - SrchOptMetrics optimize_toJson=3
12-01-2016 14:14:09.693 INFO  PredicatePushOptimizer - searchcannot be pushed through eval. Reason='delete_id' is modified (Ref:'delete_id')
12-01-2016 14:14:09.693 INFO  DispatchThread - SrchOptMetrics optimization=1
12-01-2016 14:14:09.693 INFO  SearchPipeline - Command='search' doesnt have raw field 
12-01-2016 14:14:09.694 INFO  DispatchThread - Optimized Search = | search (index=main earliest=11/30/2016:11:33:00 latest=11/30/2016:11:33:20) | eval delete_id=_cd."|".index."|".splunk_server| search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3") | delete
12-01-2016 14:14:09.694 INFO  DispatchThread - SrchOptMetrics fromJsontoSpl=1
12-01-2016 14:14:09.694 INFO  SearchParser - PARSING: | search (index=main earliest=11/30/2016:11:33:00 latest=11/30/2016:11:33:20) | eval delete_id=_cd."|".index."|".splunk_server| search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3") | delete
12-01-2016 14:14:09.694 INFO  DispatchThread - SrchOptMetrics reparse_optimized_query=1
12-01-2016 14:14:09.704 INFO  CalcFieldProcessor - Found valid eval expression for field 'chain_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),1,3))
12-01-2016 14:14:09.704 INFO  CalcFieldProcessor - Found valid eval expression for field 'store_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),4,7))
12-01-2016 14:14:09.705 INFO  SearchProcessor - Building search filter
12-01-2016 14:14:09.707 WARN  LookupOperator - Unable to find property=filename for lookup=world_timezones will attempt to use implicit filename.
12-01-2016 14:14:09.707 WARN  LookupOperator - No valid lookup found for lookup=world_timezones
12-01-2016 14:14:09.707 ERROR LookupOperator - The lookup table 'world_timezones' does not exist. It is referenced by configuration 'host::catalinavaultkafka'.
12-01-2016 14:14:09.708 INFO  StringSearchExpander - calculated_field="index" not expanded in comparison_expression="index=main". calc_field_processor!=null, negated=false (negation depth=0)
12-01-2016 14:14:09.708 INFO  StringSearchExpander - calculated_field="_time" not expanded in comparison_expression="_time>=1480523580.000". calc_field_processor!=null, negated=false (negation depth=0)
12-01-2016 14:14:09.708 INFO  StringSearchExpander - calculated_field="_time" not expanded in comparison_expression="_time<1480523600.000". calc_field_processor!=null, negated=false (negation depth=0)
12-01-2016 14:14:09.714 INFO  SearchOperator:kv - name=EXTRACT-GUID, can_use_jit=1, regex: (?i)(?!=\w)(?:objectguid|guid)\s*=\s*(?[\w\-]+)
12-01-2016 14:14:09.714 INFO  SearchOperator:kv - name=EXTRACT-SID, can_use_jit=1, regex: objectSid\s*=\s*(?\S+)
12-01-2016 14:14:09.714 INFO  SearchOperator:kv - name=ad-kv, can_use_jit=1, regex: (?<_KEY_1>[\w-]+)=(?<_VAL_1>[^\r\n]*)
12-01-2016 14:14:09.714 INFO  SearchOperator:kv - name=access-extractions, can_use_jit=1, regex: ^(?P\S+)\s++(?P\S+)\s++(?P\S+)\s++\[(?[^\]]*+)\]\s++"\s*+(?P[^\s"]++)?(?:\s++(?(?:(?\w++://[^/\s"]++))?+(?(?:/++(?(?:\\"|[^\s\?/"])++)/++)?(?:(?:\\"|[^\s\?/"])*+/++)*(?[^\s\?/]+)?)(?:\?(?[^\s]*))?)(?:\s++(?P[^\s"]++))*)?\s*+"\s++(?P\S+)\s++(?P\S+)(?:\s++"(?(?:(?\w++://[^/\s"]++))?+[^"]*+)"(?:\s++"(?[^"]*+)"(?:\s++"(?[^"]*+)")?+)?+)?(?P.*)
12-01-2016 14:14:09.714 INFO  SearchOperator:kv - name=syslog-extractions, can_use_jit=1, regex: \s([^\s\[]+)(?:\[(\d+)\])?:\s
12-01-2016 14:14:09.715 INFO  SearchOperator:kv - name=db2, can_use_jit=1, regex: ([A-Z]+) *: (.*?)(?=\n|$| +[A-Z]+ *:)
12-01-2016 14:14:09.715 INFO  SearchOperator:kv - name=EXTRACT-extract_spent, can_use_jit=1, regex: \s(?\d+(\.\d+)?)ms$
12-01-2016 14:14:09.715 INFO  SearchOperator:kv - name=EXTRACT-1, can_use_jit=1, regex: (?<_KEY_1>\S+)::(?<_VAL_1>\S+)
12-01-2016 14:14:09.716 INFO  SearchOperator:kv - name=bracket-space, can_use_jit=1, regex: \[(\S+) (.*?)\]
12-01-2016 14:14:09.717 INFO  SearchOperator:kv - name=EXTRACT-fields, can_use_jit=1, regex: (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P[^ ]*)\s+(?P[^ ]+) - (?P.+)
12-01-2016 14:14:09.717 INFO  SearchOperator:kv - name=sendmail-extractions, can_use_jit=1, regex: sendmail\[(\d+)\]: (\w+):
12-01-2016 14:14:09.717 INFO  SearchOperator:kv - name=tcpdump-endpoints, can_use_jit=1, regex: (\d+\.\d+\.\d+\.\d+):(\d+) -> (\d+\.\d+\.\d+\.\d+):(\d+)
12-01-2016 14:14:09.717 INFO  SearchOperator:kv - name=colon-kv, can_use_jit=1, regex: (?<= )([A-Za-z]+): ?((0x[A-F\d]+)|\d+)(?= |\n|$)
12-01-2016 14:14:09.736 INFO  SearchOperator:kv - name=EXTRACT-severity,logger, can_use_jit=1, regex: .*?(?[A-Z]+) ((?[^\s]+) \-)*
12-01-2016 14:14:09.736 INFO  SearchOperator:kv - name=EXTRACT-collection,category,object, can_use_jit=1, regex: collection=\"?(?P[^\"\n]+)\"?\ncategory=\"?(?P[^\"\n]+)\"?\nobject=\"?(?P<object>[^\"\n]+)\"?\n
12-01-2016 14:14:09.736 INFO  SearchOperator:kv - name=wel-message, can_use_jit=1, regex: (?sm)^(?<_pre_msg>.+)\nMessage=(?.+)$
12-01-2016 14:14:09.737 INFO  SearchOperator:kv - name=wel-col-kv, can_use_jit=1, regex: \n([^:\n\r]+):[ \t]++([^\n]*)
12-01-2016 14:14:09.743 INFO  SearchOperator:kv - name=EXTRACT-useragent, can_use_jit=1, regex: userAgent=(?P[^ (]+)
12-01-2016 14:14:09.743 INFO  SearchOperator:kv - name=splunk-service-extractions, can_use_jit=1, regex: (?i)^(?:[^ ]* ){2}(?P[^\s]*)\s+\[(?P\w+)]\s+(?P[^ ]+):(?P\d+) - (?P.+)
12-01-2016 14:14:09.743 INFO  SearchOperator:kv - name=extract_spent, can_use_jit=1, regex: \s(?P\d+(\.\d+)?)ms$
12-01-2016 14:14:09.743 INFO  SearchOperator:kv - name=weblogic-code, can_use_jit=1, regex: 
12-01-2016 14:14:09.743 INFO  SearchOperator:kv - name=colon-line, can_use_jit=1, regex: ^(\w+)\s*:[ \t]*(.*?)$
12-01-2016 14:14:09.744 INFO  SearchOperator:kv - name=was-trlog-code, can_use_jit=1, regex: ] ([a-fA-F0-9]{8})
12-01-2016 14:14:09.744 INFO  UnifiedSearch - base lispy: [ AND index::main ]
12-01-2016 14:14:09.744 INFO  UnifiedSearch - Processed search targeting arguments
12-01-2016 14:14:09.746 INFO  UnifiedSearch - Processed search targeting arguments
12-01-2016 14:14:09.746 INFO  SearchParser - PARSING: predelete
12-01-2016 14:14:09.746 INFO  SearchParser - PARSING: addinfo type=count label=prereport_events
12-01-2016 14:14:09.746 INFO  DispatchThread - BatchMode: allowBatchMode: 1, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0
12-01-2016 14:14:09.746 INFO  DispatchThread - required fields list to add to remote search = _bkt,_cd,index,splunk_server
12-01-2016 14:14:09.746 INFO  SearchParser - PARSING: fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server"
12-01-2016 14:14:09.746 INFO  DispatchCommandProcessor - summaryHash=49572ff03ece5238 summaryId=5FE18509-5E6F-4E6C-80E9-176BA1EBB4AF_search_admin_49572ff03ece5238 remoteSearch=litsearch ( index=main _time>=1480523580.000 _time<1480523600.000 ) | eval  delete_id=_cd."|".index."|".splunk_server | search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3")  | addinfo  type=count label=prereport_events | fields  keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 
12-01-2016 14:14:09.746 INFO  DispatchCommandProcessor - summaryHash=NSc97faad8e897f32e summaryId=5FE18509-5E6F-4E6C-80E9-176BA1EBB4AF_search_admin_NSc97faad8e897f32e remoteSearch=litsearch ( index=main _time>=1480523580.000 _time<1480523600.000 ) | eval delete_id=_cd."|".index."|".splunk_server | search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3") | addinfo type=count label=prereport_events | fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 
12-01-2016 14:14:09.752 INFO  DispatchThread - Setting summary_mode=NONE after optimization
12-01-2016 14:14:09.752 INFO  DispatchThread - SrchOptMetrics FinalEval=59
12-01-2016 14:14:09.752 INFO  DispatchThread - Allow retry on peer failure
12-01-2016 14:14:09.752 INFO  UserManager - Setting user context: admin
12-01-2016 14:14:09.752 INFO  UserManager - Done setting user context: admin -> admin
12-01-2016 14:14:09.752 INFO  UserManager - Unwound user context: admin -> admin
12-01-2016 14:14:09.752 INFO  DistributedSearchResultCollectionManager - Stream search: litsearch ( index=main _time>=1480523580.000 _time<1480523600.000 ) | eval  delete_id=_cd."|".index."|".splunk_server | search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3")  | addinfo  type=count label=prereport_events | fields  keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 
12-01-2016 14:14:09.752 INFO  ExternalResultProvider - No external result providers are configured
12-01-2016 14:14:09.752 INFO  DistributedSearchResultCollectionManager - ERP_FACTORY initialized, but zero external result provider, hence disabling _isERPCollectionEnabled
12-01-2016 14:14:09.752 INFO  DistributedSearchResultCollectionManager - Default search group:*
12-01-2016 14:14:09.752 INFO  DistributedSearchResultCollectionManager - Connecting to peer cp-vm0 connectAll 0 connectToSpecificPeer 1
12-01-2016 14:14:09.752 INFO  DistributedSearchResultCollectionManager - Connecting to peer cp-vm1 connectAll 0 connectToSpecificPeer 1
12-01-2016 14:14:09.752 INFO  DistributedSearchResultCollectionManager - Connecting to peer cp-vm2 connectAll 0 connectToSpecificPeer 1
12-01-2016 14:14:09.752 INFO  DistributedSearchResultCollectionManager - Connecting to peer cp-vm3 connectAll 0 connectToSpecificPeer 1
12-01-2016 14:14:09.752 INFO  DistributedSearchResultCollectionManager - Connecting to peer cp-vm4 connectAll 0 connectToSpecificPeer 1
12-01-2016 14:14:09.752 INFO  DistributedSearchResultCollectionManager - Connecting to peer devsh-vm connectAll 0 connectToSpecificPeer 1
12-01-2016 14:14:09.763 INFO  ServerConfig - Using REMOTE_SERVER_NAME=devsh-vm
12-01-2016 14:14:09.763 INFO  KeyManagerLocalhost - Checking for localhost key pair
12-01-2016 14:14:09.763 INFO  KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
12-01-2016 14:14:09.763 INFO  KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
12-01-2016 14:14:09.763 INFO  KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
12-01-2016 14:14:09.763 INFO  KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
12-01-2016 14:14:09.763 INFO  KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
12-01-2016 14:14:09.764 INFO  DistributedSearchResultCollectionManager - Successfully created search result collector for peer=cp-vm0 in 0.012000 seconds
12-01-2016 14:14:09.765 INFO  DistributedSearchResultCollectionManager - Successfully created search result collector for peer=cp-vm1 in 0.002000 seconds
12-01-2016 14:14:09.772 INFO  DistributedSearchResultCollectionManager - Successfully created search result collector for peer=cp-vm2 in 0.007000 seconds
12-01-2016 14:14:09.774 INFO  DistributedSearchResultCollectionManager - Successfully created search result collector for peer=cp-vm3 in 0.002000 seconds
12-01-2016 14:14:09.775 INFO  DistributedSearchResultCollectionManager - Successfully created search result collector for peer=cp-vm4 in 0.002000 seconds
12-01-2016 14:14:09.775 INFO  UserManager - Setting user context: admin
12-01-2016 14:14:09.775 INFO  UserManager - Done setting user context: NULL -> admin
12-01-2016 14:14:09.775 INFO  UserManager - Setting user context: admin
12-01-2016 14:14:09.775 INFO  UserManager - Done setting user context: NULL -> admin
12-01-2016 14:14:09.775 INFO  UserManager - Setting user context: admin
12-01-2016 14:14:09.775 INFO  UserManager - Done setting user context: NULL -> admin
12-01-2016 14:14:09.775 INFO  UserManager - Setting user context: admin
12-01-2016 14:14:09.775 INFO  UserManager - Setting user context: admin
12-01-2016 14:14:09.775 INFO  UserManager - Done setting user context: NULL -> admin
12-01-2016 14:14:09.775 INFO  UserManager - Done setting user context: NULL -> admin
12-01-2016 14:14:09.775 INFO  SearchParser - PARSING: litsearch ( index=main _time>=1480523580.000 _time<1480523600.000 ) | eval  delete_id=_cd."|".index."|".splunk_server | search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3")  | addinfo  type=count label=prereport_events | fields  keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 
12-01-2016 14:14:09.775 INFO  UserManager - Setting user context: admin
12-01-2016 14:14:09.775 INFO  UserManager - Done setting user context: NULL -> admin
12-01-2016 14:14:09.782 INFO  DispatchThread - Disk quota = 10485760000
12-01-2016 14:14:09.785 INFO  CalcFieldProcessor - Found valid eval expression for field 'chain_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),1,3))
12-01-2016 14:14:09.785 INFO  CalcFieldProcessor - Found valid eval expression for field 'store_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),4,7))
12-01-2016 14:14:09.794 WARN  LookupOperator - Unable to find property=filename for lookup=world_timezones will attempt to use implicit filename.
12-01-2016 14:14:09.794 WARN  LookupOperator - No valid lookup found for lookup=world_timezones
12-01-2016 14:14:09.794 ERROR LookupOperator - The lookup table 'world_timezones' does not exist. It is referenced by configuration 'host::catalinavaultkafka'.
12-01-2016 14:14:09.795 INFO  SearchParser - PARSING: typer | tags
12-01-2016 14:14:09.812 INFO  FastTyper - found nodes count: comparisons=6, unique_comparisons=5, terms=4, unique_terms=4, phrases=12, unique_phrases=12, total leaves=22
12-01-2016 14:14:09.855 INFO  BatchSearch - Using Batch Search
12-01-2016 14:14:09.855 INFO  BatchSearch - index: main dbsize=0
12-01-2016 14:14:09.855 INFO  UnifiedSearch - Initialization of search data structures took 61 ms
12-01-2016 14:14:09.855 INFO  UnifiedSearch - Processed search targeting arguments
12-01-2016 14:14:09.857 INFO  UnifiedSearch - Processed search targeting arguments
12-01-2016 14:14:09.857 INFO  LocalCollector - Final required fields list = _bkt,_cd,_subsecond,_time,index,splunk_server
12-01-2016 14:14:09.857 INFO  UserManager - Unwound user context: admin -> NULL
12-01-2016 14:14:09.857 INFO  UserManager - Setting user context: admin
12-01-2016 14:14:09.857 INFO  UserManager - Done setting user context: NULL -> admin
12-01-2016 14:14:09.857 INFO  UserManager - Unwound user context: admin -> NULL
12-01-2016 14:14:20.271 INFO  StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted
12-01-2016 14:14:20.272 INFO  DispatchThread - Generating results preview took 1 ms
12-01-2016 14:14:30.283 INFO  StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted
12-01-2016 14:14:30.283 INFO  DispatchThread - Generating results preview took 1 ms
12-01-2016 14:14:40.285 INFO  StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted
12-01-2016 14:14:40.285 INFO  DispatchThread - Generating results preview took 1 ms
12-01-2016 14:14:50.305 INFO  StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted
12-01-2016 14:14:50.305 INFO  DispatchThread - Generating results preview took 1 ms
12-01-2016 14:15:00.312 INFO  StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted
12-01-2016 14:15:00.312 INFO  DispatchThread - Generating results preview took 1 ms
12-01-2016 14:15:10.323 INFO  StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted
12-01-2016 14:15:10.323 INFO  DispatchThread - Generating results preview took 1 ms
12-01-2016 14:15:20.327 INFO  StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted
12-01-2016 14:15:20.327 INFO  DispatchThread - Generating results preview took 1 ms
12-01-2016 14:15:30.330 INFO  StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted
12-01-2016 14:15:30.330 INFO  DispatchThread - Generating results preview took 1 ms
12-01-2016 14:15:40.333 INFO  StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted
12-01-2016 14:15:40.333 INFO  DispatchThread - Generating results preview took 1 ms
12-01-2016 14:15:50.336 INFO  StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted
12-01-2016 14:15:50.336 INFO  DispatchThread - Generating results preview took 1 ms
12-01-2016 14:16:00.351 INFO  StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted
12-01-2016 14:16:00.351 INFO  DispatchThread - Generating results preview took 1 ms
</object></object>

[coltwanger]

I realize this is an older post, but we ran into an issue with "| delete" when we upgraded to 6.5.

The issue was that the events flagged for deletion would eventually reappear in Splunk; typically after a cluster restart. Splunk would not apply the delete journals across the cluster when running | delete from the search head (single search head, not SHC).

The workaround we were given was to put the cluster into maintenance mode, then log into each indexer and run the delete command on each individual indexer. We automated this to a point using REST calls. We were monitoring delete requests using a lookup and would insert the source deleted, request number, and any other comments. I scheduled a search to look for data from these sources on an hourly basis (reloaded sources were suffixed with "_RELOAD_01". Files reappear if they are not deleted using this workaround.

Word from support is they have identified the issue, and a fix is incoming for 6.5.2.