I have a question about filtering in data. We have a customer who is requesting a set of fields to be sent in from 0365. The issue is, we cant modify what we pull in because we are using  an API, not the universal forwarder. Currently I am trying to test out the search query to confirm that I am only pulling in the correct events with those fields.  The o365 data pulls in about 400+ fields. We are wanting about 40 of those events for a specific use case. My question is, what is the correct syntax for splunk to only search for those fields.  Original query that brings in about 400+ fields:   index=o365     New query for about 35 fields:   index=o365 "Operation"="*" OR "LabelAction"="*" OR "LabelAppliedDateTime"="*" OR "LabelIid"="*" OR "abelName"="*" OR "DlpAuditEventMetadata.DlpPolicyMatchId"="*" OR "DlpAuditEventMetadata.EvaluationTime"="*" OR "DlpOriginalFilePath"="*" OR "IrmContentId"="*" OR "PolicyMatchInfo.PolicyId"="*" OR "PolicyMatchInfo.PolicyName"="*" OR "PolicyMatchInfo.RuleId"="*" OR "PolicyMatchInfo.RuleName"="*" OR "ProtectionEventData.IsProtected"="*" OR "ProtectionEventData.IsProtectedBefore"="*" OR "ProtectionEventData.ProtectionEventType"="*" OR "ProtectionEventData.ProtectionOwner"="*" OR "ProtectionEventData.ProtectionType"="*" OR "ProtectionEventData.TemplateId"="*" OR "ProtectionEventType"="*" OR "RMSEncrypted"="*" OR "SensitiveInfoTypeData{}.Confidence"="*" OR "SensitiveInfoTypeData{}.Count"="*" OR "SensitiveInfoTypeData{}.SensitiveInfoTypeId"="*" OR "SensitiveInfoTypeData{}.SensitiveInfoTypeName"="*" OR "SensitiveInfoTypeData{}.SensitiveInformationDetailedClassificationAttributes{}.Confidence"="*" OR "SensitiveInfoTypeData{}.SensitiveInformationDetailedClassificationAttributes{}.Count"="*" OR "SensitivityLabelEventData.ActionSource"="*" OR "SensitivityLabelEventData.ActionSourceDetail"="*" OR "SensitivityLabelEventData.ContentType"="*" OR "SensitivityLabelEventData.JustificationText"="*" OR "SensitivityLabelEventData.LabelEventType"="*" OR "SensitivityLabelEventData.OldSensitivityLabelId"="*" OR "SensitivityLabelEventData.SensitivityLabelId"="*" OR "SensitivityLabelEventData.SensitivityLabelPolicyId"="*" OR "LabelName"="*" | fields Operation,LabelAction,LabelAppliedDateTime,LabelIid,abelName,DlpAuditEventMetadata.DlpPolicyMatchId,DlpAuditEventMetadata.EvaluationTime,DlpOriginalFilePath,IrmContentId,PolicyMatchInfo.PolicyId,PolicyMatchInfo.PolicyName,PolicyMatchInfo.RuleId,PolicyMatchInfo.RuleName,ProtectionEventData.IsProtected,ProtectionEventData.IsProtectedBefore,ProtectionEventData.ProtectionEventType,ProtectionEventData.ProtectionOwner,ProtectionEventData.ProtectionType,ProtectionEventData.TemplateId,ProtectionEventType,RMSEncrypted,SensitiveInfoTypeData{}.Confidence,SensitiveInfoTypeData{}.Count,SensitiveInfoTypeData{}.SensitiveInfoTypeId,SensitiveInfoTypeData{}.SensitiveInfoTypeName,SensitiveInfoTypeData{}.SensitiveInformationDetailedClassificationAttributes{}.Confidence,SensitiveInfoTypeData{}.SensitiveInformationDetailedClassificationAttributes{}.Count,SensitivityLabelEventData.ActionSource,SensitivityLabelEventData.ActionSourceDetail,SensitivityLabelEventData.ContentType,SensitivityLabelEventData.JustificationText,SensitivityLabelEventData.LabelEventType,SensitivityLabelEventData.OldSensitivityLabelId,SensitivityLabelEventData.SensitivityLabelId,SensitivityLabelEventData.SensitivityLabelPolicyId,LabelName     Basically, From my understanding and my research, if you just append a specific string in quotes, or outside of quotes, splunk searches all events for that string and pulls it in. Such as:   index=Test field1 field2 field3   That would bring in only events with field1 or field2 or field3 within it. Adding quotes to it, such as    index=Test "field1"="*" "field2"="*" "field3"="*"   Should filter the same way. I have tested it both way, with double quotes surrounding the field, as well as no quotes. Im also using | fields Which should only bring those fields in, but i dont know if its only showing those fields, but bringing in ALL of the events.    My question is, is this correct? With the base searches ive been testing with, searching all of the events in o365 for one day, full 24 hours, brings in  23,410,064 events. Filtering out with the query I pasted above, for the same day, same 24 hours, brings in 23,409,887 events. Ive tested this a couple of ways, and each time, searching over the same time period, the filtering query brings in about 1k less events. But I can still only view the first 1k events, 20 pages worth. But that may be another question.    My longwinded question boils down to, am I searching this data correctly? I know its a heavy index with millions of events, but filtering out to only 40 or so fields, some of which only appear .6% of the time, still brings in millions of events. Is there a way to fully validate it?    ... View more

I would like to add a label for the upper/lower 95. I was wondering how I could do that. Id like to have it the same color as the line as well, similar to --Upper|Lower95. ... View more

I am trying to create an alert or a report to track the number of deferred searches. We had an issue where the cluster captain deferred a massive amount of searches, and it messed up a few things. We are trying to create an alert to maybe mitigate that in the future. In addition to asking the best way to create a alert for this, id also like some more clarification on how to find the deferred searches.  Through the monitoring console,  either through the DMC or the Cluster Master,  I thought i had seen a panel for deferred searches, but i can not find one now. And when i run the search        index=_internal earliest=-24h "status=skipped" sourcetype=scheduler | stats count by host app | sort - count       I get results, but if i change status to deferred, which i assume is a status, I do not get anything. I was suggested to run        | rest /services/search/jobs | search status=deferred | table id, search, app, owner, earliest_time, latest_time, status, sid       but I do not get any status. Status is not a field.   The main question I have is How do I access the number of deferred searches? If i can find that, I can run stats count on it.    Thank you.  ... View more

So we have this alert set up to check to see if any hostnames that are being monitored havnt received any time monitoring data. The current search is as follows:   |inputlookup TimeServersV2.csv | search server="*" | eval HOST =lower(server) | fields HOST | where NOT [search (index=os sourcetype=test_stats*) OR (sourcetype=syslog ptp10 OR phc10sys) OR (index=windows sourcetype="Script:TimeStatus") OR (index=windows sourcetype=domtimec) OR (index=os sourcetype=time)| dedup host | eval HOST=lower(host)| fields HOST ]     The issue with this is, we believe, once it runs at 8 AM,  it takes a bit longer to run and process data, and it'll send out partial results after a min or so of running the query. We have a lot of saved reports/alerts/searches running at the top of most hours, so I think it may be sending out the incomplete search results after a bit of running, as splunk starts the next job. I moved its cron job schedule up an hour and a half to a lighter use hour, so that may help a bit, but i would also like to optimize this search so it runs faster. Currently, it runs about 40 seconds to a little over a minute: What would be the best way to optimize this search so it could possibly be run in under 30 seconds, if possible. Running it outside of the scheduled time runs in about 6 seconds, its just slow when it runs alongside all of the other searches. Itll send us an alert with a list of hostnames its found that were not on the list, yet when we run it manually, it will only spit out 4 or 5 results. That's why we think its not finishing the search when it sends out an alert. Any help would be appreciated.  ... View more

We have this dashboard that recently started alerting us on a risky command. We were using the fit command.    I followed the docs and added the following line to the newly created commands.conf that i had put in the apps local folder to push.  [fit] is_risky = false According to the docs, i assumed that this would just disable the warning for using that command. After i put it into the specified apps local folder, /export/opt/splunk/etc/shcluster/apps/<app-name>/local, I pushed the bundle and it seemed to have put it in the apps default folder on the search heads. But the biggest issue here is that once i pushed that bundle, splunk doesn't recognize the fit.py file any more. I tested putting that commands.conf in the apps default folder, same thing. I tested this a few times, and while im glad that the bundle pushes were working, im a bit confused as to why splunk no longer recognizes that command even though im only using the is_risky=false, which should only stop the warning.    Any help on this matter would be appreciated. Thank you. And if you could also answer as to why the local file in the app's directory is pushing to the apps default folder on the search heads, that would be a bonus. Thank you.  ... View more

So i am trying to compare bar graphs for event count for our indexes for two separate days. We are upgrading our environment, and I was wanting this query to show us the event count before and after we upgrade. I am have tried using the earliest=-<int>d and latest=-<int>d, but the query keeps using the time picker. I am using dbinspect, so i wasn't sure if that had something to do with it. Below is the working query that outputs the same results for both EventCount and EventCount_1     |dbinspect index=* | search index!=_* | fields bucketId eventCount index _time | stats sum(eventCount) as EventCount values(max(_time)) as Time by index | table index EventCount, | join type=outer index [| dbinspect index=* | search index!=_* | fields bucketId eventCount index | stats sum(eventCount) as EventCount_1 by index | table index EventCount_1] | table index EventCount EventCount_1       I have tried putting the the time periods in a few places, after the first index, in which the query runs, but returns the same results using the time from the time picker. If i place it after the search, I dont get any results.      |dbinspect index=* earliest=-4d latest=-3d | search index!=_* | fields bucketId eventCount index _time | stats sum(eventCount) as EventCount values(max(_time)) as Time by index | table index EventCount, | join type=outer index [| dbinspect index=* | search index!=_* earliest=2023-05-30T00:00:00 latest=2023-06-01T23:59:59 | fields bucketId eventCount index | stats sum(eventCount) as EventCount_1 by index | table index EventCount_1] | table index EventCount EventCount_1     ^  this is also a working query, but it still uses the time from time picker instead of the stated one in query ^ Am I supposed to be using a different type of time selection with the dbinspect? If i don't use dbinspect, I don't get the same results. Is there any other way to get these results? I'm just trying to get event count by index. Thank you for any help.    ... View more

I have this report that i received an error from. Ive seen the error from different searches, but i just started to look into them. In the email, it said the issue was  Error in 'where' command: The 'not' function is unsupported or undefined.   Im assuming the search ran fine before it started getting an error as it was turned into a report.  Current search:    |inputlookup X_servers.csv | search OS=*Windows* environment=Production OR environment="Disaster Recovery" | dedup host | rename host as HOST | table HOST environment OS application1 | sort +HOST |where NOT[|inputlookup Y_agent_managed.csv | table HOST]     I looked up that error, but I couldnt find anything useful out of them. We have our DMC, Both cluster Masters, and deployment servers all on Splunk 9.x.x, everything else is on 8.x.x. The posts I saw were talking about Splunk 6.x, so that one was a bit outdated and im at a loss as to the proper syntax. I tried replacing NOT with != but apparently splunk reads them both as NOT, which makes sense.      Id appreciate any help, thank you. We haev a few old reports that still use NOt like this .  ... View more

So I couldn't find anything in splunk community that answers my question about pushing an update to a lookup table. I manually updated the .csv file through the backend searchhead server. I deleted a line and replaced it with another hostname.    When i run the command:       |inputlookup dns_hosts.csv| stats count by host|eval count=0|join host type=outer [ search index="dns"|stats count by host]|fillnull|where count=0|fields host count       Im still getting the host that has a count of 0, the host that i removed in the csv file. My question is do i need to restart the searchhead to push that change? I didnt change any config files, just the lookupfile under the specific app directory's lookup file folder. I wasnt sure if splunk would automatically read the updated file after a certain amount of time, or if i needed to restart the server for it to take effect? And will that file replicate across all searchheads after I restart it?  Thank you for any guidance.  ... View more