I have duplicated records that I am trying to delete in Splunk. I am using Splunk 6.5 with Search Head Clustering, and Clustered Indexers. My user has the can_delete option checked off and I am able to delete records, but for some reason I have 35 records that refuse to be deleted. They all seem to be on the same indexer. I am able to search for the records without the delete option and they come up within a few seconds, but when I put the delete option, it just searches for a long time and does not delete the records. My search looks like this: index=main earliest=11/30/2016:11:33:00 latest=11/30/2016:11:33:20 | eval delete_id=_cd."|".index."|".splunk_server | search (( delete_id="109:40078965|main|cp-vm3" ) OR ( delete_id="109:40084329|main|cp-vm3" ) OR ( delete_id="109:40085221|main|cp-vm3" ) OR ( delete_id="109:40088243|main|cp-vm3" ) OR ( delete_id="109:40088070|main|cp-vm3" ) OR ( delete_id="109:40085858|main|cp-vm3" ) OR ( delete_id="109:40086155|main|cp-vm3" ) OR ( delete_id="109:40088053|main|cp-vm3" ) OR ( delete_id="109:40085602|main|cp-vm3" ) OR ( delete_id="109:40066343|main|cp-vm3" ) OR ( delete_id="109:40068493|main|cp-vm3" ) OR ( delete_id="109:40073891|main|cp-vm3" ) OR ( delete_id="109:40077210|main|cp-vm3" ) OR ( delete_id="109:40069880|main|cp-vm3" ) OR ( delete_id="109:40066724|main|cp-vm3" ) OR ( delete_id="109:40067052|main|cp-vm3" ) OR ( delete_id="109:40067280|main|cp-vm3" ) OR ( delete_id="109:40070422|main|cp-vm3" ) OR ( delete_id="109:40072184|main|cp-vm3" ) OR ( delete_id="109:40067032|main|cp-vm3" ) OR ( delete_id="109:40072168|main|cp-vm3" ) OR ( delete_id="109:40086139|main|cp-vm3" ) OR ( delete_id="109:40084253|main|cp-vm3" ) OR ( delete_id="109:40084615|main|cp-vm3" ) OR ( delete_id="109:40068545|main|cp-vm3" ) OR ( delete_id="109:40087829|main|cp-vm3" ) OR ( delete_id="109:40066808|main|cp-vm3" ) OR ( delete_id="109:40067264|main|cp-vm3" ) OR ( delete_id="109:40069296|main|cp-vm3" ) OR ( delete_id="109:40079749|main|cp-vm3" ) OR ( delete_id="109:40079733|main|cp-vm3" ) OR ( delete_id="109:40081521|main|cp-vm3" ) OR ( delete_id="109:40084269|main|cp-vm3" ) OR ( delete_id="109:40086336|main|cp-vm3" )) | delete In the search job inspector, it shows this message This search is still running and is approximately 100% complete. (SID: 1480601649.181) search.log The actual search.log 12-01-2016 14:14:09.388 INFO dispatchRunner - Search process mode: preforked (reused process) 12-01-2016 14:14:09.388 WARN DistributedInfoSingleton - Failed to read symptoms of peer=devsh-vm 12-01-2016 14:14:09.388 INFO dispatchRunner - registering build time modules, count=1 12-01-2016 14:14:09.388 INFO dispatchRunner - registering search time components of build time module name=vix 12-01-2016 14:14:09.389 INFO BundlesSetup - Setup stats for /opt/splunk/etc: wallclock_elapsed_msec=30, cpu_time_used=0.032, shared_services_generation=2, shared_services_population=1 12-01-2016 14:14:09.389 INFO UserManager - Setting user context: splunk-system-user 12-01-2016 14:14:09.389 INFO UserManager - Done setting user context: NULL -> splunk-system-user 12-01-2016 14:14:09.389 INFO UserManager - Unwound user context: splunk-system-user -> NULL 12-01-2016 14:14:09.389 INFO UserManager - Setting user context: admin 12-01-2016 14:14:09.389 INFO UserManager - Done setting user context: NULL -> admin 12-01-2016 14:14:09.389 INFO dispatchRunner - search context: user="admin", app="search", bs-pathname="/opt/splunk/etc" 12-01-2016 14:14:09.390 INFO SearchParser - PARSING: search index=main earliest=11/30/2016:11:33:00 latest=11/30/2016:11:33:20 | eval delete_id=_cd."|".index."|".splunk_server | search (( delete_id="109:40078965|main|cp-vm3" ) OR ( delete_id="109:40084329|main|cp-vm3" ) OR ( delete_id="109:40085221|main|cp-vm3" ) OR ( delete_id="109:40088243|main|cp-vm3" ) OR ( delete_id="109:40088070|main|cp-vm3" ) OR ( delete_id="109:40085858|main|cp-vm3" ) OR ( delete_id="109:40086155|main|cp-vm3" ) OR ( delete_id="109:40088053|main|cp-vm3" ) OR ( delete_id="109:40085602|main|cp-vm3" ) OR ( delete_id="109:40066343|main|cp-vm3" ) OR ( delete_id="109:40068493|main|cp-vm3" ) OR ( delete_id="109:40073891|main|cp-vm3" ) OR ( delete_id="109:40077210|main|cp-vm3" ) OR ( delete_id="109:40069880|main|cp-vm3" ) OR ( delete_id="109:40066724|main|cp-vm3" ) OR ( delete_id="109:40067052|main|cp-vm3" ) OR ( delete_id="109:40067280|main|cp-vm3" ) OR ( delete_id="109:40070422|main|cp-vm3" ) OR ( delete_id="109:40072184|main|cp-vm3" ) OR ( delete_id="109:40067032|main|cp-vm3" ) OR ( delete_id="109:40072168|main|cp-vm3" ) OR ( delete_id="109:40086139|main|cp-vm3" ) OR ( delete_id="109:40084253|main|cp-vm3" ) OR ( delete_id="109:40084615|main|cp-vm3" ) OR ( delete_id="109:40068545|main|cp-vm3" ) OR ( delete_id="109:40087829|main|cp-vm3" ) OR ( delete_id="109:40066808|main|cp-vm3" ) OR ( delete_id="109:40067264|main|cp-vm3" ) OR ( delete_id="109:40069296|main|cp-vm3" ) OR ( delete_id="109:40079749|main|cp-vm3" ) OR ( delete_id="109:40079733|main|cp-vm3" ) OR ( delete_id="109:40081521|main|cp-vm3" ) OR ( delete_id="109:40084269|main|cp-vm3" ) OR ( delete_id="109:40086336|main|cp-vm3" )) | delete 12-01-2016 14:14:09.390 INFO ISplunkDispatch - Not running in splunkd. Bundle replication not triggered. 12-01-2016 14:14:09.471 INFO UserManager - Setting user context: admin 12-01-2016 14:14:09.471 INFO UserManager - Done setting user context: NULL -> admin 12-01-2016 14:14:09.482 INFO CalcFieldProcessor - Found valid eval expression for field 'chain_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),1,3)) 12-01-2016 14:14:09.482 INFO CalcFieldProcessor - Found valid eval expression for field 'store_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),4,7)) 12-01-2016 14:14:09.485 INFO SearchProcessor - Building search filter 12-01-2016 14:14:09.497 WARN LookupOperator - Unable to find property=filename for lookup=world_timezones will attempt to use implicit filename. 12-01-2016 14:14:09.497 WARN LookupOperator - No valid lookup found for lookup=world_timezones 12-01-2016 14:14:09.497 ERROR LookupOperator - The lookup table 'world_timezones' does not exist. It is referenced by configuration 'host::catalinavaultkafka'. 12-01-2016 14:14:09.498 INFO StringSearchExpander - calculated_field="index" not expanded in comparison_expression="index=main". calc_field_processor!=null, negated=false (negation depth=0) 12-01-2016 14:14:09.498 INFO StringSearchExpander - calculated_field="_time" not expanded in comparison_expression="_time>=1480523580.000". calc_field_processor!=null, negated=false (negation depth=0) 12-01-2016 14:14:09.498 INFO StringSearchExpander - calculated_field="_time" not expanded in comparison_expression="_time<1480523600.000". calc_field_processor!=null, negated=false (negation depth=0) 12-01-2016 14:14:09.585 INFO SearchOperator:kv - name=EXTRACT-GUID, can_use_jit=1, regex: (?i)(?!=\w)(?:objectguid|guid)\s*=\s*(?[\w\-]+) 12-01-2016 14:14:09.585 INFO SearchOperator:kv - name=EXTRACT-SID, can_use_jit=1, regex: objectSid\s*=\s*(?\S+) 12-01-2016 14:14:09.586 INFO SearchOperator:kv - name=ad-kv, can_use_jit=1, regex: (?<_KEY_1>[\w-]+)=(?<_VAL_1>[^\r\n]*) 12-01-2016 14:14:09.595 INFO SearchOperator:kv - name=access-extractions, can_use_jit=1, regex: ^(?P\S+)\s++(?P\S+)\s++(?P\S+)\s++\[(?[^\]]*+)\]\s++"\s*+(?P[^\s"]++)?(?:\s++(?(?:(?\w++://[^/\s"]++))?+(?(?:/++(?(?:\\"|[^\s\?/"])++)/++)?(?:(?:\\"|[^\s\?/"])*+/++)*(?[^\s\?/]+)?)(?:\?(?[^\s]*))?)(?:\s++(?P[^\s"]++))*)?\s*+"\s++(?P\S+)\s++(?P\S+)(?:\s++"(?(?:(?\w++://[^/\s"]++))?+[^"]*+)"(?:\s++"(?[^"]*+)"(?:\s++"(?[^"]*+)")?+)?+)?(?P.*) 12-01-2016 14:14:09.595 INFO SearchOperator:kv - name=syslog-extractions, can_use_jit=1, regex: \s([^\s\[]+)(?:\[(\d+)\])?:\s 12-01-2016 14:14:09.596 INFO SearchOperator:kv - name=db2, can_use_jit=1, regex: ([A-Z]+) *: (.*?)(?=\n|$| +[A-Z]+ *:) 12-01-2016 14:14:09.597 INFO SearchOperator:kv - name=EXTRACT-extract_spent, can_use_jit=1, regex: \s(?\d+(\.\d+)?)ms$ 12-01-2016 14:14:09.597 INFO SearchOperator:kv - name=EXTRACT-1, can_use_jit=1, regex: (?<_KEY_1>\S+)::(?<_VAL_1>\S+) 12-01-2016 14:14:09.598 INFO SearchOperator:kv - name=bracket-space, can_use_jit=1, regex: \[(\S+) (.*?)\] 12-01-2016 14:14:09.599 INFO SearchOperator:kv - name=EXTRACT-fields, can_use_jit=1, regex: (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P[^ ]*)\s+(?P[^ ]+) - (?P.+) 12-01-2016 14:14:09.600 INFO SearchOperator:kv - name=sendmail-extractions, can_use_jit=1, regex: sendmail\[(\d+)\]: (\w+): 12-01-2016 14:14:09.600 INFO SearchOperator:kv - name=tcpdump-endpoints, can_use_jit=1, regex: (\d+\.\d+\.\d+\.\d+):(\d+) -> (\d+\.\d+\.\d+\.\d+):(\d+) 12-01-2016 14:14:09.601 INFO SearchOperator:kv - name=colon-kv, can_use_jit=1, regex: (?<= )([A-Za-z]+): ?((0x[A-F\d]+)|\d+)(?= |\n|$) 12-01-2016 14:14:09.620 INFO SearchOperator:kv - name=EXTRACT-severity,logger, can_use_jit=1, regex: .*?(?[A-Z]+) ((?[^\s]+) \-)* 12-01-2016 14:14:09.627 INFO SearchOperator:kv - name=EXTRACT-collection,category,object, can_use_jit=1, regex: collection=\"?(?P[^\"\n]+)\"?\ncategory=\"?(?P[^\"\n]+)\"?\nobject=\"?(?P<object>[^\"\n]+)\"?\n 12-01-2016 14:14:09.628 INFO SearchOperator:kv - name=wel-message, can_use_jit=1, regex: (?sm)^(?<_pre_msg>.+)\nMessage=(?.+)$ 12-01-2016 14:14:09.628 INFO SearchOperator:kv - name=wel-col-kv, can_use_jit=1, regex: \n([^:\n\r]+):[ \t]++([^\n]*) 12-01-2016 14:14:09.629 INFO SearchOperator:kv - name=EXTRACT-useragent, can_use_jit=1, regex: userAgent=(?P[^ (]+) 12-01-2016 14:14:09.629 INFO SearchOperator:kv - name=splunk-service-extractions, can_use_jit=1, regex: (?i)^(?:[^ ]* ){2}(?P[^\s]*)\s+\[(?P\w+)]\s+(?P[^ ]+):(?P\d+) - (?P.+) 12-01-2016 14:14:09.630 INFO SearchOperator:kv - name=extract_spent, can_use_jit=1, regex: \s(?P\d+(\.\d+)?)ms$ 12-01-2016 14:14:09.631 INFO SearchOperator:kv - name=weblogic-code, can_use_jit=1, regex: 12-01-2016 14:14:09.637 INFO SearchOperator:kv - name=colon-line, can_use_jit=1, regex: ^(\w+)\s*:[ \t]*(.*?)$ 12-01-2016 14:14:09.637 INFO SearchOperator:kv - name=was-trlog-code, can_use_jit=1, regex: ] ([a-fA-F0-9]{8}) 12-01-2016 14:14:09.638 INFO UnifiedSearch - base lispy: [ AND index::main ] 12-01-2016 14:14:09.639 INFO UnifiedSearch - Processed search targeting arguments 12-01-2016 14:14:09.669 INFO UnifiedSearch - Processed search targeting arguments 12-01-2016 14:14:09.669 INFO SearchParser - PARSING: predelete 12-01-2016 14:14:09.669 INFO SearchParser - PARSING: addinfo type=count label=prereport_events 12-01-2016 14:14:09.669 INFO DispatchThread - BatchMode: allowBatchMode: 1, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0 12-01-2016 14:14:09.670 INFO DispatchThread - required fields list to add to remote search = _bkt,_cd,index,splunk_server 12-01-2016 14:14:09.670 INFO SearchParser - PARSING: fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server" 12-01-2016 14:14:09.670 INFO DispatchCommandProcessor - summaryHash=c544ca20eeb5ac6c summaryId=5FE18509-5E6F-4E6C-80E9-176BA1EBB4AF_search_admin_c544ca20eeb5ac6c remoteSearch=litsearch index=main _time>=1480523580.000 _time<1480523600.000 | eval delete_id=_cd."|".index."|".splunk_server | search (( delete_id="109:40078965|main|cp-vm3" ) OR ( delete_id="109:40084329|main|cp-vm3" ) OR ( delete_id="109:40085221|main|cp-vm3" ) OR ( delete_id="109:40088243|main|cp-vm3" ) OR ( delete_id="109:40088070|main|cp-vm3" ) OR ( delete_id="109:40085858|main|cp-vm3" ) OR ( delete_id="109:40086155|main|cp-vm3" ) OR ( delete_id="109:40088053|main|cp-vm3" ) OR ( delete_id="109:40085602|main|cp-vm3" ) OR ( delete_id="109:40066343|main|cp-vm3" ) OR ( delete_id="109:40068493|main|cp-vm3" ) OR ( delete_id="109:40073891|main|cp-vm3" ) OR ( delete_id="109:40077210|main|cp-vm3" ) OR ( delete_id="109:40069880|main|cp-vm3" ) OR ( delete_id="109:40066724|main|cp-vm3" ) OR ( delete_id="109:40067052|main|cp-vm3" ) OR ( delete_id="109:40067280|main|cp-vm3" ) OR ( delete_id="109:40070422|main|cp-vm3" ) OR ( delete_id="109:40072184|main|cp-vm3" ) OR ( delete_id="109:40067032|main|cp-vm3" ) OR ( delete_id="109:40072168|main|cp-vm3" ) OR ( delete_id="109:40086139|main|cp-vm3" ) OR ( delete_id="109:40084253|main|cp-vm3" ) OR ( delete_id="109:40084615|main|cp-vm3" ) OR ( delete_id="109:40068545|main|cp-vm3" ) OR ( delete_id="109:40087829|main|cp-vm3" ) OR ( delete_id="109:40066808|main|cp-vm3" ) OR ( delete_id="109:40067264|main|cp-vm3" ) OR ( delete_id="109:40069296|main|cp-vm3" ) OR ( delete_id="109:40079749|main|cp-vm3" ) OR ( delete_id="109:40079733|main|cp-vm3" ) OR ( delete_id="109:40081521|main|cp-vm3" ) OR ( delete_id="109:40084269|main|cp-vm3" ) OR ( delete_id="109:40086336|main|cp-vm3" )) | addinfo type=count label=prereport_events | fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 12-01-2016 14:14:09.670 INFO DispatchCommandProcessor - summaryHash=NSc41c4fa16f7c937e summaryId=5FE18509-5E6F-4E6C-80E9-176BA1EBB4AF_search_admin_NSc41c4fa16f7c937e remoteSearch=litsearch index=main _time>=1480523580.000 _time<1480523600.000 | eval delete_id=_cd."|".index."|".splunk_server | search (( delete_id="109:40078965|main|cp-vm3" ) OR ( delete_id="109:40084329|main|cp-vm3" ) OR ( delete_id="109:40085221|main|cp-vm3" ) OR ( delete_id="109:40088243|main|cp-vm3" ) OR ( delete_id="109:40088070|main|cp-vm3" ) OR ( delete_id="109:40085858|main|cp-vm3" ) OR ( delete_id="109:40086155|main|cp-vm3" ) OR ( delete_id="109:40088053|main|cp-vm3" ) OR ( delete_id="109:40085602|main|cp-vm3" ) OR ( delete_id="109:40066343|main|cp-vm3" ) OR ( delete_id="109:40068493|main|cp-vm3" ) OR ( delete_id="109:40073891|main|cp-vm3" ) OR ( delete_id="109:40077210|main|cp-vm3" ) OR ( delete_id="109:40069880|main|cp-vm3" ) OR ( delete_id="109:40066724|main|cp-vm3" ) OR ( delete_id="109:40067052|main|cp-vm3" ) OR ( delete_id="109:40067280|main|cp-vm3" ) OR ( delete_id="109:40070422|main|cp-vm3" ) OR ( delete_id="109:40072184|main|cp-vm3" ) OR ( delete_id="109:40067032|main|cp-vm3" ) OR ( delete_id="109:40072168|main|cp-vm3" ) OR ( delete_id="109:40086139|main|cp-vm3" ) OR ( delete_id="109:40084253|main|cp-vm3" ) OR ( delete_id="109:40084615|main|cp-vm3" ) OR ( delete_id="109:40068545|main|cp-vm3" ) OR ( delete_id="109:40087829|main|cp-vm3" ) OR ( delete_id="109:40066808|main|cp-vm3" ) OR ( delete_id="109:40067264|main|cp-vm3" ) OR ( delete_id="109:40069296|main|cp-vm3" ) OR ( delete_id="109:40079749|main|cp-vm3" ) OR ( delete_id="109:40079733|main|cp-vm3" ) OR ( delete_id="109:40081521|main|cp-vm3" ) OR ( delete_id="109:40084269|main|cp-vm3" ) OR ( delete_id="109:40086336|main|cp-vm3" )) | addinfo type=count label=prereport_events | fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 12-01-2016 14:14:09.670 INFO DispatchThread - Getting summary ID for summaryHash=NSc41c4fa16f7c937e 12-01-2016 14:14:09.691 INFO DispatchThread - Did not find a usable summary_id, setting info._summary_mode=none, not modifying input summary_id=5FE18509-5E6F-4E6C-80E9-176BA1EBB4AF_search_admin_NSc41c4fa16f7c937e 12-01-2016 14:14:09.691 INFO DispatchThread - Matches no summary 12-01-2016 14:14:09.691 INFO DispatchThread - SrchOptMetrics check_query_matches_ra=221 12-01-2016 14:14:09.691 INFO SearchParser - PARSING: search index=main earliest=11/30/2016:11:33:00 latest=11/30/2016:11:33:20 | eval delete_id=_cd."|".index."|".splunk_server | search (( delete_id="109:40078965|main|cp-vm3" ) OR ( delete_id="109:40084329|main|cp-vm3" ) OR ( delete_id="109:40085221|main|cp-vm3" ) OR ( delete_id="109:40088243|main|cp-vm3" ) OR ( delete_id="109:40088070|main|cp-vm3" ) OR ( delete_id="109:40085858|main|cp-vm3" ) OR ( delete_id="109:40086155|main|cp-vm3" ) OR ( delete_id="109:40088053|main|cp-vm3" ) OR ( delete_id="109:40085602|main|cp-vm3" ) OR ( delete_id="109:40066343|main|cp-vm3" ) OR ( delete_id="109:40068493|main|cp-vm3" ) OR ( delete_id="109:40073891|main|cp-vm3" ) OR ( delete_id="109:40077210|main|cp-vm3" ) OR ( delete_id="109:40069880|main|cp-vm3" ) OR ( delete_id="109:40066724|main|cp-vm3" ) OR ( delete_id="109:40067052|main|cp-vm3" ) OR ( delete_id="109:40067280|main|cp-vm3" ) OR ( delete_id="109:40070422|main|cp-vm3" ) OR ( delete_id="109:40072184|main|cp-vm3" ) OR ( delete_id="109:40067032|main|cp-vm3" ) OR ( delete_id="109:40072168|main|cp-vm3" ) OR ( delete_id="109:40086139|main|cp-vm3" ) OR ( delete_id="109:40084253|main|cp-vm3" ) OR ( delete_id="109:40084615|main|cp-vm3" ) OR ( delete_id="109:40068545|main|cp-vm3" ) OR ( delete_id="109:40087829|main|cp-vm3" ) OR ( delete_id="109:40066808|main|cp-vm3" ) OR ( delete_id="109:40067264|main|cp-vm3" ) OR ( delete_id="109:40069296|main|cp-vm3" ) OR ( delete_id="109:40079749|main|cp-vm3" ) OR ( delete_id="109:40079733|main|cp-vm3" ) OR ( delete_id="109:40081521|main|cp-vm3" ) OR ( delete_id="109:40084269|main|cp-vm3" ) OR ( delete_id="109:40086336|main|cp-vm3" )) | delete 12-01-2016 14:14:09.691 INFO UnifiedSearch - Processed search targeting arguments 12-01-2016 14:14:09.692 INFO UnifiedSearch - Processed search targeting arguments 12-01-2016 14:14:09.693 INFO DispatchThread - SrchOptMetrics optimize_toJson=3 12-01-2016 14:14:09.693 INFO PredicatePushOptimizer - searchcannot be pushed through eval. Reason='delete_id' is modified (Ref:'delete_id') 12-01-2016 14:14:09.693 INFO DispatchThread - SrchOptMetrics optimization=1 12-01-2016 14:14:09.693 INFO SearchPipeline - Command='search' doesnt have raw field 12-01-2016 14:14:09.694 INFO DispatchThread - Optimized Search = | search (index=main earliest=11/30/2016:11:33:00 latest=11/30/2016:11:33:20) | eval delete_id=_cd."|".index."|".splunk_server| search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3") | delete 12-01-2016 14:14:09.694 INFO DispatchThread - SrchOptMetrics fromJsontoSpl=1 12-01-2016 14:14:09.694 INFO SearchParser - PARSING: | search (index=main earliest=11/30/2016:11:33:00 latest=11/30/2016:11:33:20) | eval delete_id=_cd."|".index."|".splunk_server| search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3") | delete 12-01-2016 14:14:09.694 INFO DispatchThread - SrchOptMetrics reparse_optimized_query=1 12-01-2016 14:14:09.704 INFO CalcFieldProcessor - Found valid eval expression for field 'chain_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),1,3)) 12-01-2016 14:14:09.704 INFO CalcFieldProcessor - Found valid eval expression for field 'store_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),4,7)) 12-01-2016 14:14:09.705 INFO SearchProcessor - Building search filter 12-01-2016 14:14:09.707 WARN LookupOperator - Unable to find property=filename for lookup=world_timezones will attempt to use implicit filename. 12-01-2016 14:14:09.707 WARN LookupOperator - No valid lookup found for lookup=world_timezones 12-01-2016 14:14:09.707 ERROR LookupOperator - The lookup table 'world_timezones' does not exist. It is referenced by configuration 'host::catalinavaultkafka'. 12-01-2016 14:14:09.708 INFO StringSearchExpander - calculated_field="index" not expanded in comparison_expression="index=main". calc_field_processor!=null, negated=false (negation depth=0) 12-01-2016 14:14:09.708 INFO StringSearchExpander - calculated_field="_time" not expanded in comparison_expression="_time>=1480523580.000". calc_field_processor!=null, negated=false (negation depth=0) 12-01-2016 14:14:09.708 INFO StringSearchExpander - calculated_field="_time" not expanded in comparison_expression="_time<1480523600.000". calc_field_processor!=null, negated=false (negation depth=0) 12-01-2016 14:14:09.714 INFO SearchOperator:kv - name=EXTRACT-GUID, can_use_jit=1, regex: (?i)(?!=\w)(?:objectguid|guid)\s*=\s*(?[\w\-]+) 12-01-2016 14:14:09.714 INFO SearchOperator:kv - name=EXTRACT-SID, can_use_jit=1, regex: objectSid\s*=\s*(?\S+) 12-01-2016 14:14:09.714 INFO SearchOperator:kv - name=ad-kv, can_use_jit=1, regex: (?<_KEY_1>[\w-]+)=(?<_VAL_1>[^\r\n]*) 12-01-2016 14:14:09.714 INFO SearchOperator:kv - name=access-extractions, can_use_jit=1, regex: ^(?P\S+)\s++(?P\S+)\s++(?P\S+)\s++\[(?[^\]]*+)\]\s++"\s*+(?P[^\s"]++)?(?:\s++(?(?:(?\w++://[^/\s"]++))?+(?(?:/++(?(?:\\"|[^\s\?/"])++)/++)?(?:(?:\\"|[^\s\?/"])*+/++)*(?[^\s\?/]+)?)(?:\?(?[^\s]*))?)(?:\s++(?P[^\s"]++))*)?\s*+"\s++(?P\S+)\s++(?P\S+)(?:\s++"(?(?:(?\w++://[^/\s"]++))?+[^"]*+)"(?:\s++"(?[^"]*+)"(?:\s++"(?[^"]*+)")?+)?+)?(?P.*) 12-01-2016 14:14:09.714 INFO SearchOperator:kv - name=syslog-extractions, can_use_jit=1, regex: \s([^\s\[]+)(?:\[(\d+)\])?:\s 12-01-2016 14:14:09.715 INFO SearchOperator:kv - name=db2, can_use_jit=1, regex: ([A-Z]+) *: (.*?)(?=\n|$| +[A-Z]+ *:) 12-01-2016 14:14:09.715 INFO SearchOperator:kv - name=EXTRACT-extract_spent, can_use_jit=1, regex: \s(?\d+(\.\d+)?)ms$ 12-01-2016 14:14:09.715 INFO SearchOperator:kv - name=EXTRACT-1, can_use_jit=1, regex: (?<_KEY_1>\S+)::(?<_VAL_1>\S+) 12-01-2016 14:14:09.716 INFO SearchOperator:kv - name=bracket-space, can_use_jit=1, regex: \[(\S+) (.*?)\] 12-01-2016 14:14:09.717 INFO SearchOperator:kv - name=EXTRACT-fields, can_use_jit=1, regex: (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P[^ ]*)\s+(?P[^ ]+) - (?P.+) 12-01-2016 14:14:09.717 INFO SearchOperator:kv - name=sendmail-extractions, can_use_jit=1, regex: sendmail\[(\d+)\]: (\w+): 12-01-2016 14:14:09.717 INFO SearchOperator:kv - name=tcpdump-endpoints, can_use_jit=1, regex: (\d+\.\d+\.\d+\.\d+):(\d+) -> (\d+\.\d+\.\d+\.\d+):(\d+) 12-01-2016 14:14:09.717 INFO SearchOperator:kv - name=colon-kv, can_use_jit=1, regex: (?<= )([A-Za-z]+): ?((0x[A-F\d]+)|\d+)(?= |\n|$) 12-01-2016 14:14:09.736 INFO SearchOperator:kv - name=EXTRACT-severity,logger, can_use_jit=1, regex: .*?(?[A-Z]+) ((?[^\s]+) \-)* 12-01-2016 14:14:09.736 INFO SearchOperator:kv - name=EXTRACT-collection,category,object, can_use_jit=1, regex: collection=\"?(?P[^\"\n]+)\"?\ncategory=\"?(?P[^\"\n]+)\"?\nobject=\"?(?P<object>[^\"\n]+)\"?\n 12-01-2016 14:14:09.736 INFO SearchOperator:kv - name=wel-message, can_use_jit=1, regex: (?sm)^(?<_pre_msg>.+)\nMessage=(?.+)$ 12-01-2016 14:14:09.737 INFO SearchOperator:kv - name=wel-col-kv, can_use_jit=1, regex: \n([^:\n\r]+):[ \t]++([^\n]*) 12-01-2016 14:14:09.743 INFO SearchOperator:kv - name=EXTRACT-useragent, can_use_jit=1, regex: userAgent=(?P[^ (]+) 12-01-2016 14:14:09.743 INFO SearchOperator:kv - name=splunk-service-extractions, can_use_jit=1, regex: (?i)^(?:[^ ]* ){2}(?P[^\s]*)\s+\[(?P\w+)]\s+(?P[^ ]+):(?P\d+) - (?P.+) 12-01-2016 14:14:09.743 INFO SearchOperator:kv - name=extract_spent, can_use_jit=1, regex: \s(?P\d+(\.\d+)?)ms$ 12-01-2016 14:14:09.743 INFO SearchOperator:kv - name=weblogic-code, can_use_jit=1, regex: 12-01-2016 14:14:09.743 INFO SearchOperator:kv - name=colon-line, can_use_jit=1, regex: ^(\w+)\s*:[ \t]*(.*?)$ 12-01-2016 14:14:09.744 INFO SearchOperator:kv - name=was-trlog-code, can_use_jit=1, regex: ] ([a-fA-F0-9]{8}) 12-01-2016 14:14:09.744 INFO UnifiedSearch - base lispy: [ AND index::main ] 12-01-2016 14:14:09.744 INFO UnifiedSearch - Processed search targeting arguments 12-01-2016 14:14:09.746 INFO UnifiedSearch - Processed search targeting arguments 12-01-2016 14:14:09.746 INFO SearchParser - PARSING: predelete 12-01-2016 14:14:09.746 INFO SearchParser - PARSING: addinfo type=count label=prereport_events 12-01-2016 14:14:09.746 INFO DispatchThread - BatchMode: allowBatchMode: 1, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0 12-01-2016 14:14:09.746 INFO DispatchThread - required fields list to add to remote search = _bkt,_cd,index,splunk_server 12-01-2016 14:14:09.746 INFO SearchParser - PARSING: fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server" 12-01-2016 14:14:09.746 INFO DispatchCommandProcessor - summaryHash=49572ff03ece5238 summaryId=5FE18509-5E6F-4E6C-80E9-176BA1EBB4AF_search_admin_49572ff03ece5238 remoteSearch=litsearch ( index=main _time>=1480523580.000 _time<1480523600.000 ) | eval delete_id=_cd."|".index."|".splunk_server | search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3") | addinfo type=count label=prereport_events | fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 12-01-2016 14:14:09.746 INFO DispatchCommandProcessor - summaryHash=NSc97faad8e897f32e summaryId=5FE18509-5E6F-4E6C-80E9-176BA1EBB4AF_search_admin_NSc97faad8e897f32e remoteSearch=litsearch ( index=main _time>=1480523580.000 _time<1480523600.000 ) | eval delete_id=_cd."|".index."|".splunk_server | search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3") | addinfo type=count label=prereport_events | fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 12-01-2016 14:14:09.752 INFO DispatchThread - Setting summary_mode=NONE after optimization 12-01-2016 14:14:09.752 INFO DispatchThread - SrchOptMetrics FinalEval=59 12-01-2016 14:14:09.752 INFO DispatchThread - Allow retry on peer failure 12-01-2016 14:14:09.752 INFO UserManager - Setting user context: admin 12-01-2016 14:14:09.752 INFO UserManager - Done setting user context: admin -> admin 12-01-2016 14:14:09.752 INFO UserManager - Unwound user context: admin -> admin 12-01-2016 14:14:09.752 INFO DistributedSearchResultCollectionManager - Stream search: litsearch ( index=main _time>=1480523580.000 _time<1480523600.000 ) | eval delete_id=_cd."|".index."|".splunk_server | search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3") | addinfo type=count label=prereport_events | fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 12-01-2016 14:14:09.752 INFO ExternalResultProvider - No external result providers are configured 12-01-2016 14:14:09.752 INFO DistributedSearchResultCollectionManager - ERP_FACTORY initialized, but zero external result provider, hence disabling _isERPCollectionEnabled 12-01-2016 14:14:09.752 INFO DistributedSearchResultCollectionManager - Default search group:* 12-01-2016 14:14:09.752 INFO DistributedSearchResultCollectionManager - Connecting to peer cp-vm0 connectAll 0 connectToSpecificPeer 1 12-01-2016 14:14:09.752 INFO DistributedSearchResultCollectionManager - Connecting to peer cp-vm1 connectAll 0 connectToSpecificPeer 1 12-01-2016 14:14:09.752 INFO DistributedSearchResultCollectionManager - Connecting to peer cp-vm2 connectAll 0 connectToSpecificPeer 1 12-01-2016 14:14:09.752 INFO DistributedSearchResultCollectionManager - Connecting to peer cp-vm3 connectAll 0 connectToSpecificPeer 1 12-01-2016 14:14:09.752 INFO DistributedSearchResultCollectionManager - Connecting to peer cp-vm4 connectAll 0 connectToSpecificPeer 1 12-01-2016 14:14:09.752 INFO DistributedSearchResultCollectionManager - Connecting to peer devsh-vm connectAll 0 connectToSpecificPeer 1 12-01-2016 14:14:09.763 INFO ServerConfig - Using REMOTE_SERVER_NAME=devsh-vm 12-01-2016 14:14:09.763 INFO KeyManagerLocalhost - Checking for localhost key pair 12-01-2016 14:14:09.763 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem 12-01-2016 14:14:09.763 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem 12-01-2016 14:14:09.763 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem 12-01-2016 14:14:09.763 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem 12-01-2016 14:14:09.763 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem 12-01-2016 14:14:09.764 INFO DistributedSearchResultCollectionManager - Successfully created search result collector for peer=cp-vm0 in 0.012000 seconds 12-01-2016 14:14:09.765 INFO DistributedSearchResultCollectionManager - Successfully created search result collector for peer=cp-vm1 in 0.002000 seconds 12-01-2016 14:14:09.772 INFO DistributedSearchResultCollectionManager - Successfully created search result collector for peer=cp-vm2 in 0.007000 seconds 12-01-2016 14:14:09.774 INFO DistributedSearchResultCollectionManager - Successfully created search result collector for peer=cp-vm3 in 0.002000 seconds 12-01-2016 14:14:09.775 INFO DistributedSearchResultCollectionManager - Successfully created search result collector for peer=cp-vm4 in 0.002000 seconds 12-01-2016 14:14:09.775 INFO UserManager - Setting user context: admin 12-01-2016 14:14:09.775 INFO UserManager - Done setting user context: NULL -> admin 12-01-2016 14:14:09.775 INFO UserManager - Setting user context: admin 12-01-2016 14:14:09.775 INFO UserManager - Done setting user context: NULL -> admin 12-01-2016 14:14:09.775 INFO UserManager - Setting user context: admin 12-01-2016 14:14:09.775 INFO UserManager - Done setting user context: NULL -> admin 12-01-2016 14:14:09.775 INFO UserManager - Setting user context: admin 12-01-2016 14:14:09.775 INFO UserManager - Setting user context: admin 12-01-2016 14:14:09.775 INFO UserManager - Done setting user context: NULL -> admin 12-01-2016 14:14:09.775 INFO UserManager - Done setting user context: NULL -> admin 12-01-2016 14:14:09.775 INFO SearchParser - PARSING: litsearch ( index=main _time>=1480523580.000 _time<1480523600.000 ) | eval delete_id=_cd."|".index."|".splunk_server | search (delete_id="109:40078965|main|cp-vm3" OR delete_id="109:40084329|main|cp-vm3" OR delete_id="109:40085221|main|cp-vm3" OR delete_id="109:40088243|main|cp-vm3" OR delete_id="109:40088070|main|cp-vm3" OR delete_id="109:40085858|main|cp-vm3" OR delete_id="109:40086155|main|cp-vm3" OR delete_id="109:40088053|main|cp-vm3" OR delete_id="109:40085602|main|cp-vm3" OR delete_id="109:40066343|main|cp-vm3" OR delete_id="109:40068493|main|cp-vm3" OR delete_id="109:40073891|main|cp-vm3" OR delete_id="109:40077210|main|cp-vm3" OR delete_id="109:40069880|main|cp-vm3" OR delete_id="109:40066724|main|cp-vm3" OR delete_id="109:40067052|main|cp-vm3" OR delete_id="109:40067280|main|cp-vm3" OR delete_id="109:40070422|main|cp-vm3" OR delete_id="109:40072184|main|cp-vm3" OR delete_id="109:40067032|main|cp-vm3" OR delete_id="109:40072168|main|cp-vm3" OR delete_id="109:40086139|main|cp-vm3" OR delete_id="109:40084253|main|cp-vm3" OR delete_id="109:40084615|main|cp-vm3" OR delete_id="109:40068545|main|cp-vm3" OR delete_id="109:40087829|main|cp-vm3" OR delete_id="109:40066808|main|cp-vm3" OR delete_id="109:40067264|main|cp-vm3" OR delete_id="109:40069296|main|cp-vm3" OR delete_id="109:40079749|main|cp-vm3" OR delete_id="109:40079733|main|cp-vm3" OR delete_id="109:40081521|main|cp-vm3" OR delete_id="109:40084269|main|cp-vm3" OR delete_id="109:40086336|main|cp-vm3") | addinfo type=count label=prereport_events | fields keepcolorder=t "_bkt" "_cd" "index" "splunk_server" | predelete 12-01-2016 14:14:09.775 INFO UserManager - Setting user context: admin 12-01-2016 14:14:09.775 INFO UserManager - Done setting user context: NULL -> admin 12-01-2016 14:14:09.782 INFO DispatchThread - Disk quota = 10485760000 12-01-2016 14:14:09.785 INFO CalcFieldProcessor - Found valid eval expression for field 'chain_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),1,3)) 12-01-2016 14:14:09.785 INFO CalcFieldProcessor - Found valid eval expression for field 'store_id' in stanza [host::catalinavaultkafka]': tonumber(substr(substr("0000000".site_id,-7),4,7)) 12-01-2016 14:14:09.794 WARN LookupOperator - Unable to find property=filename for lookup=world_timezones will attempt to use implicit filename. 12-01-2016 14:14:09.794 WARN LookupOperator - No valid lookup found for lookup=world_timezones 12-01-2016 14:14:09.794 ERROR LookupOperator - The lookup table 'world_timezones' does not exist. It is referenced by configuration 'host::catalinavaultkafka'. 12-01-2016 14:14:09.795 INFO SearchParser - PARSING: typer | tags 12-01-2016 14:14:09.812 INFO FastTyper - found nodes count: comparisons=6, unique_comparisons=5, terms=4, unique_terms=4, phrases=12, unique_phrases=12, total leaves=22 12-01-2016 14:14:09.855 INFO BatchSearch - Using Batch Search 12-01-2016 14:14:09.855 INFO BatchSearch - index: main dbsize=0 12-01-2016 14:14:09.855 INFO UnifiedSearch - Initialization of search data structures took 61 ms 12-01-2016 14:14:09.855 INFO UnifiedSearch - Processed search targeting arguments 12-01-2016 14:14:09.857 INFO UnifiedSearch - Processed search targeting arguments 12-01-2016 14:14:09.857 INFO LocalCollector - Final required fields list = _bkt,_cd,_subsecond,_time,index,splunk_server 12-01-2016 14:14:09.857 INFO UserManager - Unwound user context: admin -> NULL 12-01-2016 14:14:09.857 INFO UserManager - Setting user context: admin 12-01-2016 14:14:09.857 INFO UserManager - Done setting user context: NULL -> admin 12-01-2016 14:14:09.857 INFO UserManager - Unwound user context: admin -> NULL 12-01-2016 14:14:20.271 INFO StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted 12-01-2016 14:14:20.272 INFO DispatchThread - Generating results preview took 1 ms 12-01-2016 14:14:30.283 INFO StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted 12-01-2016 14:14:30.283 INFO DispatchThread - Generating results preview took 1 ms 12-01-2016 14:14:40.285 INFO StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted 12-01-2016 14:14:40.285 INFO DispatchThread - Generating results preview took 1 ms 12-01-2016 14:14:50.305 INFO StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted 12-01-2016 14:14:50.305 INFO DispatchThread - Generating results preview took 1 ms 12-01-2016 14:15:00.312 INFO StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted 12-01-2016 14:15:00.312 INFO DispatchThread - Generating results preview took 1 ms 12-01-2016 14:15:10.323 INFO StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted 12-01-2016 14:15:10.323 INFO DispatchThread - Generating results preview took 1 ms 12-01-2016 14:15:20.327 INFO StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted 12-01-2016 14:15:20.327 INFO DispatchThread - Generating results preview took 1 ms 12-01-2016 14:15:30.330 INFO StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted 12-01-2016 14:15:30.330 INFO DispatchThread - Generating results preview took 1 ms 12-01-2016 14:15:40.333 INFO StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted 12-01-2016 14:15:40.333 INFO DispatchThread - Generating results preview took 1 ms 12-01-2016 14:15:50.336 INFO StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted 12-01-2016 14:15:50.336 INFO DispatchThread - Generating results preview took 1 ms 12-01-2016 14:16:00.351 INFO StreamingDeleteOperator - sid:1480601649.181 0 events successfully deleted 12-01-2016 14:16:00.351 INFO DispatchThread - Generating results preview took 1 ms </object></object> ... View more