I need to build an overall user activities report as in login activities, file accessed, file exported, application accessed, failed attempts etc., which ever potential tracking can be done using Splunk or a user. I am not sure how to approach this. Can someone help me with which index, or Data model I should start with to get this kind of information. Any help would be really appreciated. ... View more

Hi Team, I have been getting a skipped search notification in my CMC overview under Health from quite some time. It is a scheduled report Search name: ESS - Notable Events Cron: every 5 mins ( 1-59/5 * * * *) Timerange:  earliest - 48d@d  ; latest - +0s (now) Message: The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached Search query:  `notable` | search NOT `suppression` | eval timeDiff_type=case(_time>=relative_time(now(), "-24h@h"),"current", 1=1, "historical") | expandtoken rule_title | table _time,event_id,security_domain,urgency,rule_name,rule_title,src,dest,src_user,user,dvc,status,status_group,owner,timeDiff_type,governance,control | outputlookup es_notable_events | stats count It is writing the output to an output-lookup.  and takes around 8 mins as runtime when checked under job management. Can some help me understand where the issue lies, what's making this search in particular to skip. The percentage skipped it around 50% and the status is critical.   ... View more

Hi Team, I have been observing 1 skipped search error indicating on my CMC. Error is - "The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached" Percentage skipped is 33.3%.  I saw many solutions online, stating to increase the user/role search job limit or either make changes in the limits.conf (which I don't have access to) but couldn't figure out or get clear explanations, Can someone help me to solve this. Also, the saved search in question is running on a cron of 2-59/5 * * * * for time range of 5 mins.  Please suggest.   ... View more

Hi everyone! I am working on building a dashboard which captures all the firewall, Web proxy, EDR, WAF, Email, DLP blocked for last 6 months in a table format which should look like this -  I am able to write the query which will give me the count for each parameter and then I append all the single query into one which is making the final query run slower and taking forever to complete. Here is the final query- | tstats `security_content_summariesonly` count as Blocked from datamodel=Network_Traffic where sourcetype IN ("cp_log", "cisco:asa", "pan:traffic") AND All_Traffic.action="blocked" earliest=-6mon@mon latest=now by _time | eval Source="Firewall" | tstats `security_content_summariesonly` count as Blocked from datamodel=Web where sourcetype IN ("alertlogic:waf","aemcdn","aws:*","azure:firewall:*") AND Web.action="block" earliest=-6mon latest=now by _time | eval Source="WAF" | append [search index=zscaler* action=blocked sourcetype="zscalernss-web" earliest=-6mon@mon latest=now | bin _time span=1mon | stats count as Blocked by _time | eval Source="Web Proxy"] | append [| tstats summariesonly=false dc(Message_Log.msg.header.message-id) as Blocked from datamodel=pps_ondemand where (Message_Log.filter.routeDirection="inbound") AND (Message_Log.filter.disposition="discard" OR Message_Log.filter.disposition="reject" OR Message_Log.filter.quarantine.folder="Spam*") earliest=-6mon@mon latest=now by _time | eval Source="Email"] | append [search index=crowdstrike-hc sourcetype="CrowdStrike:Event:Streams:JSON" "metadata.eventType"=DetectionSummaryEvent metadata.customerIDString=* earliest=-6mon@mon latest=now | bin _time span=1mon | transaction "event.DetectId" | search action=blocked NOT action=allowed | stats dc(event.DetectId) as Blocked by _time | eval Source="EDR"] | append [search index=forcepoint_dlp sourcetype IN ("forcepoint:dlp","forcepoint:dlp:csv") action="blocked" earliest=-6mon@mon latest=now | bin _time span=1mon | stats count(action) as Blocked by _time | eval Source="DLP"] | eval MonthNum=strftime(_time, "%Y-%m"), MonthName=strftime(_time, "%b") | stats sum(Blocked) as Blocked by Source MonthNum MonthName | xyseries Source MonthName Blocked | addinfo | table Source [| makeresults count=7 | streamstats count as month_offset | eval start_epoch=relative_time(now(),"-6mon@mon"), end_epoch=now() | eval start_month=strftime(start_epoch, "%Y-%m-01") | eval month_epoch = relative_time(strptime(start_month, "%Y-%m-%d"), "+" . (month_offset-1) . "mon") | where month_epoch <= end_epoch | eval month=strftime(month_epoch, "%b") | stats list(month) as search ]   I figured out the issue is with the firewall query- | tstats `security_content_summariesonly` count as Blocked from datamodel=Network_Traffic where sourcetype IN ("cp_log", "cisco:asa", "pan:traffic") AND All_Traffic.action="blocked" earliest=-6mon@mon latest=now by _time | eval Source="Firewall" Can someone guide me how to fix this issue. I have been stuck in this issue from 2 weeks 😞 ... View more

Hi! I am creating a basic dashboard which shows the total number of firewall blocked for 3 sourcetypes using data model "network_traffic". Query is- | tstats `security_content_summariesonly` count from datamodel=Network_Traffic where sourcetype IN ("cp_log", "cisco:asa", "pan:traffic") All_Traffic.action="blocked" Now I am trying to add one more panel which will show what is causing the block activity (error message) for each sourcetype with respect to count, but I am unable to figure out the appropriate field (or query) from the data model which is related to the error message. Can someone help me understand which field to group by to get the error message. P.S I am new to splunk ... View more

Lately, my CMC is indicating that there are 20 searches exceeding 10% of system memory. But when I click on it I don't see which searches are listed as high memory searches. All it gives is search id, app, memory used and status. (refer the image below) Could anyone please suggest me how to troubleshoot this and find which searches are under High memory searches.   ... View more

Hi Team, I am a newbie in Splunk. I am using a basic IN clause in a search command to pull out the 36 windows server integration available or not in splunk. index=* NOT index=_* host IN ("host1", "host2", "host3", ...., "host36") | stats count by host It gives me the servers which are available in splunk. I want to find out the missing ones and the available ones in this format- host status host1 Available host2 Available host3 Missing I am using the below query but it only gives 6 statistics. | makeresults count=1 | eval all_hosts="host1,host2,host3....host36" | makemv delim="," all_hosts | mvexpand all_hosts | rename all_hosts AS host | append [ search index=* NOT index=_* host=* | stats count by host ] | stats values(count) AS event_count by host | where host IN ("host1", "host2",....,"host36") | eval status=if(isnull(event_count), "Missing", "Available") | table host status | sort status, host  Can anyone please help me out what I am doing wrong. Thanks in Advance!! ... View more