I've been trying to resolve this since October and not getting traction.  Turning to the community for help: I have seemingly contradictory information within the same log line makes me question- do we have an issue or not?   On the one hand, i think i do because the history command shows the search is cancelled... and I trust this information.  However, there are artifacts in the logs that make me question if the search is fully running (which appears to be true since "fully_completed_search=TRUE"... so I am now confused if we have a problem or not.) Why do searches show fully_completed_search=TRUE and has_error_warn=FALSE when the info field (and history command) show "cancelled" and have a tag of "error"   BOTTOM LINE QUESTION: Are my searches are running correctly and returning all results or not?    Sample _audit log search activity that I found - not sure if this gives any usable insight Audit:[timestamp=10-01-2021 16:31:40.338, user=redacted_user, action=search, info=canceled, search_id='1633105804.108286', has_error_warn=false, fully_completed_search=true, total_run_time=18.13, event_count=0, result_count=0, available_count=0, scan_count=133645, drop_count=0, exec_time=1633105804, api_et=1633104900.000000000, api_lt=1633105800.000000000, api_index_et=N/A, api_index_lt=N/A, search_et=1633104900.000000000, search_lt=1633105800.000000000, is_realtime=0, savedsearch_name="", search_startup_time="1270", is_prjob=false, acceleration_id="98DCBC55-D36C-4671-93CD-1A950D796EC4_search_redacted_user_311d202b50b71a64", app="search", provenance="N/A", mode="historical_batch", workload_pool=standard_perf, is_proxied=false, searched_buckets=53, eliminated_buckets=0, considered_events=133645, total_slices=331408, decompressed_slices=11305, duration.command.search.index=120, invocations.command.search.index.bucketcache.hit=53, duration.command.search.index.bucketcache.hit=0, invocations.command.search.index.bucketcache.miss=0, duration.command.search.index.bucketcache.miss=0, invocations.command.search.index.bucketcache.error=0, duration.command.search.rawdata=2533, invocations.command.search.rawdata.bucketcache.hit=0, duration.command.search.rawdata.bucketcache.hit=0, invocations.command.search.rawdata.bucketcache.miss=0, duration.command.search.rawdata.bucketcache.miss=0, invocations.command.search.rawdata.bucketcache.error=0, roles='redacted', search='search index=oswinsec (EventID=7036 OR EventID=50 OR EventID=56 OR EventID=1000 OR EventID=1001) | eval my_ts2 = _time*1000 | eval indextime=_indextime |table my_ts2,EventID | rename EventID as EventCode'] ... View more

Imagine the following data set: STUDENT EOY_GRADE GENDER STUDENT_STATUS Alice 96 Female ACTIVE Bob 94 Male ACTIVE Candice 92 Female FORMER Debbie 94 Female FORMER Eddie 94 Male FORMER Frank 96 Male FORMER   And I would like the produce the following output comparing current students to former: STUDENT EOY_GRADE PREV_GENDER_AVG PREV_CLASS_AVG CURRENT_CLASS_AVG Alice 96 93 94 95 Bob 94 95 94 95 Thanks in advance for consideration and thoughts ... View more

I executed the following SPL with makeresults, but the results only give me the fields for _time and _raw... i don't get parsed fields. Can this be solved? |makeresults count=100|eval _raw="Process Create: UtcTime: 2017-04-28 22:08:22.025 ProcessGuid: {a23eae89-bd56-5903-0000-0010e9d95e00} ProcessId: 6228 Image: C:\Program Files (x86)\Google\Chrome\Application\chrome.exe CommandLine: \"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe\" --type=utility --lang=en-US --no-sandbox --service-request-channel-token=F47498BBA884E523FA93E623C4569B94 --mojo-platform-channel-handle=3432 /prefetch:8 CurrentDirectory: C:\Program Files (x86)\Google\Chrome\Application\58.0.3029.81\ User: LAB\rsmith LogonGuid: {a23eae89-b357-5903-0000-002005eb0700} LogonId: 0x7EB05 TerminalSessionId: 1 IntegrityLevel: Medium Hashes: SHA256=6055A20CF7EC81843310AD37700FF67B2CF8CDE3DCE68D54BA42934177C10B57 ParentProcessGuid: {a23eae89-bd28-5903-0000-00102f345d00} ParentProcessId: 13220 ParentImage: C:\Program Files (x86)\Google\Chrome\Application\chrome.exe ParentCommandLine: \"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe\" " ... View more

I am super stoked about the potential of Schema Accelerated Event Searches- might be one of the best improvements i've seen if i could actually get it to work- but it doesn't. 😞 Don't focus on the fact that i'm only returning the count of events... performance doesn't differ if i returned the raw events (which is ultimately what i want to do).... i'm just doing the count so i can make an apples-to-apples comparison. So consider the following two searches over 15 minutes of data: SEARCH # 1 |tstats summariesonly=true count from datamodel="Web" where Web.user="dmerritt" The value returned was 25. The search itself took 2.676 seconds SEARCH # 2 |from datamodel Web|search user=dmerritt|stats count The value returned was 106. The search itself took 2 minutes, 14 seconds. QUESTIONS: 1) Why the HUGE difference in performance? 2) Why is the result count different? NOTE : Am running Splunk 7.1.5 ... View more

If a an accelerated data model is 80% complete, what does that ACTUALLY mean? Does it mean I have 80% of the events? 80% of the time? 80% of the data? Almost half my data models are not keeping up at even close to 100% but i can't figure out what is actually "missing" from the data models. ... View more

Technically, this is two questions in one with the goal of solving a single problem: I need an SPL query that returns ALL the indexes I can search and the associated retention time for each. Here is how far I've gotten: | rest /services/data/indexes | eval yr = floor(frozenTimePeriodInSecs/86400/365)| eval dy = (frozenTimePeriodInSecs/86400) % 365 | eval ret = yr . " years, " . dy . " days" | stats list(splunk_server) list(frozenTimePeriodInSecs) list(ret) by title The query above is very very close, but it only returns a subset of the indexes — technically, it only returns 32 index names to me, and I have many more than that. (Note- starting with "rest /services/admin/indexes ... " makes no difference either. My second query is this: | eventcount summarize=false index=* index=_* | dedup index | fields index That will return all 250+ index names, but I can't seem to find anyway to get back to the retention period. So my two questions are: 1) Why is the rest command only pulling a subset (<15%) of all indexes that are returned by the event count query? 2) How can I get a single query that gets to my goal to have a single SPL query that shows all 250+ indexes and their associated retention setting? ... View more