All Topics

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Topics

What settings, functionalities or areas would you check in a Newly installed Splunk Enterprise 8.2.3 making sure all is well? I have a large environment, planning to install ES, Apps & TAs. Planning ... See more...
What settings, functionalities or areas would you check in a Newly installed Splunk Enterprise 8.2.3 making sure all is well? I have a large environment, planning to install ES, Apps & TAs. Planning clustered in as much areas as possible. Thank u for your help & advice?
Hi Splunkers,  I have a 2 hosts i.e server1 & server2. Each host running with multiple processes. Lets say the processes are process1 & process2. I want to create a dashboard to show the latest pr... See more...
Hi Splunkers,  I have a 2 hosts i.e server1 & server2. Each host running with multiple processes. Lets say the processes are process1 & process2. I want to create a dashboard to show the latest processes status whether it is Running or Not Running in each host   index=os host IN (server1 server2)  ARGS=*process1* OR ARGS=*process2* | eval process1_status=if(like(ARGS,"%process1%"),"Running","Not Running") | eval process2_status=if(like(ARGS,"%process2%"),"Running","Not Running") | stats latest(process1_status)  latest(process2_status)  by host | fillnull value=NULL But this query is not giving correct results. Each event will have either ARGS field as process1 or ARGS field as process2.    
Dear All, So I have a Linux script that runs vmstat as a daemon and writes the output every minute to a csv file. Here is some typical output ... _time, metric_name:vmstat.procs.runwait, metric_nam... See more...
Dear All, So I have a Linux script that runs vmstat as a daemon and writes the output every minute to a csv file. Here is some typical output ... _time, metric_name:vmstat.procs.runwait, metric_name:vmstat.procs.blocking, metric_name:vmstat.memory.swapped, metric_name:vmstat.memory.free, metric_name:vmstat.memory.buffers, metric_name:vmstat.memory.cache, metric_name:vmstat.swap.in, metric_name:vmstat.swap.out, metric_name:vmstat.blocks.read, metric_name:vmstat.blocks.written, metric_name:vmstat.system.interupts, metric_name:vmstat.system.contxtswtch, metric_name:vmstat.cpu.user, metric_name:vmstat.cpu.system, metric_name:vmstat.cpu.idle, metric_name:vmstat.cpu.iowait, metric_name:vmstat.cpu.stolen 1637263961, 11, 0, 301056, 13188244, 52, 1645532, 0, 0, 258, 20, 4, 2, 2, 3, 96, 0, 0 1637264021, 3, 0, 301056, 13193028, 52, 1645648, 0, 0, 0, 37, 1480, 2090, 0, 1, 99, 0, 0 1637264081, 3, 0, 301056, 13193448, 52, 1645724, 0, 0, 0, 13, 700, 1097, 0, 0, 100, 0, 0 1637264141, 3, 0, 301056, 13192100, 52, 1645812, 0, 0, 0, 17, 756, 1154, 0, 0, 100, 0, 0 Now every so often I get an error in the message board like The metric value=metric_name:vmstat.procs.runwait provided for source=/opt/splunkforwarder/etc/apps/TA-linux-metrics/log/read_vmstat.log, sourcetype=csv, host=foo.bar.baz, index=lnx_os_metrics is not a floating point value. Using a numeric type rather than a string type is recommended to avoid indexing inefficiencies. Ensure the metric value is provided as a floating point number and not as a string. For instance, provide 123.001 rather than 123.001. This is not consistent and when I look at the file it is perfectly formed, the above is an example that just threw me an error.  The stanza from inputs.conf is [monitor:///opt/splunkforwarder/etc/apps/TA-linux-metrics/log/read_vmstat.log] index = lnx_os_metrics sourcetype = csv I tried with sourcetype csv as well as metrics_csv,  both give the same result.  What on earth could be going on here? Thanks, R.
I'm responsible for a Cisco IM & Presence system.  It can support logging of messages to an external SQL database or a 3rd party compliance server (like Verba). I'm not very familiar with Splunk and... See more...
I'm responsible for a Cisco IM & Presence system.  It can support logging of messages to an external SQL database or a 3rd party compliance server (like Verba). I'm not very familiar with Splunk and its suite of products.  I'm being asked if Splunk can be used to log Jabber instant messages but I'm not sure it can be used in that capacity.  Based on Cisco's IM compliance documentation: https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/im_presence/im_compliance/12_5_1/cup0_b_im-compliance-guide-1251/cup0_b_im-compliance-guide-1251_chapter_01.html it seems like Splunk can be used to view messages in the SQL database being used to archive messages.  Other than that, I've haven't seen any documentation showing that Splunk can be used to view or store Cisco IM & Presence instant messages between Jabber clients. Has anyone had any experience trying to use Splunk to access Cisco IMP Jabber messages?  If so, do you have any experience or documentation that you could share? Thanks,    
I am tearing my hair out trying to figure this one out... I had a powershell input on my UFs (both Win10 and Server 16) that was working fine until last week, when the events mysteriously stopped com... See more...
I am tearing my hair out trying to figure this one out... I had a powershell input on my UFs (both Win10 and Server 16) that was working fine until last week, when the events mysteriously stopped coming into my indexer. Here's the stanza from inputs.conf: [powershell://MPComputerStatus] script = get-mpcomputerstatus schedule = 5 sourcetype = Windows:MPComputerStatus (note that the schedule of 5 is just for debugging currently; normally it is set to 300) Everything appears to be functioning normally on the UF side - I look at splunk-powershell.ps1.log and I see the same lines from before and after the issue started: 11-11-2021...INFO Start executing script=get-mpcomputerstatus for stanza=MPComputerStatus 11-11-2021...INFO End of executing script..., execution time=0.0149976 seconds However, the events do not show up under sourcetype=windows:mpcomputerstatus anymore. All of the Windows event log events are still being forwarded. Here's what I have tried: updating Splunk and the forwarders from 8.1.2 to 8.2.3 running the UF service as both a domain service account (the previous setting) and Local System changing the logging config to DEBUG in log.cfg, log-cmdline.cfg Also, I found it odd that all of my Win10 workstations stopped on the same day, and my Server 2016 machine stopped on a different day. Any ideas?
I see below log in operation_install showing continuous failure to connect to https://gravity-site.kube-system.svc.cluster.local:3009/healthz. ================ Wed Nov 10 02:40:41 UTC [INFO] [DAPD0... See more...
I see below log in operation_install showing continuous failure to connect to https://gravity-site.kube-system.svc.cluster.local:3009/healthz. ================ Wed Nov 10 02:40:41 UTC [INFO] [DAPD02] Executing postInstall hook for site:6.1.48. Created Pod "site-app-post-install-125088-zqsmd" in namespace "kube-system". Container "post-install-hook" created, current state is "waiting, reason PodInitializing". Pod "site-app-post-install-125088-zqsmd" in namespace "kube-system", has changed state from "Pending" to "Running". Container "post-install-hook" changed status from "waiting, reason PodInitializing" to "running". ^[[31m[ERROR]: failed connecting to https://gravity-site.kube-system.svc.cluster.local:3009/healthz Get https://gravity-site.kube-system.svc.cluster.local:3009/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ^[[0mContainer "post-install-hook" changed status from "running" to "terminated, exit code 255". Container "post-install-hook" restarted, current state is "running". ^[[31m[ERROR]: failed connecting to https://gravity-site.kube-system.svc.cluster.local:3009/healthz Get https://gravity-site.kube-system.svc.cluster.local:3009/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ^[[0mContainer "post-install-hook" changed status from "running" to "terminated, exit code 255". Container "post-install-hook" changed status from "terminated, exit code 255" to "waiting, reason CrashLoopBackOff". ================ The gravity cluster status after the installation failure: ================ [root@DAPD02 crashreport]# gravity status Cluster name: charmingmeitner2182 Cluster status: degraded (application status check failed) Application: dsp, version 1.2.1 Gravity version: 6.1.48 (client) / 6.1.48 (server) Join token: b9b088ce63c0a703ee740ba5dfb380d Periodic updates: Not Configured Remote support: Not Configured Last completed operation: * 3-node install ID: 46614e3c-fcd1-4974-8cd7-dc404d1880b Started: Wed Nov 10 02:33 UTC (1 hour ago) Completed: Wed Nov 10 02:35 UTC (1 hour ago) Cluster endpoints: * Authentication gateway: - 10.69.80.1:32009 - 10.69.80.2:32009 - 10.69.89.3:32009 * Cluster management URL: - https://10.69.80.1:32009 - https://10.69.80.2:32009 - https://10.69.89.3:32009 Cluster nodes: Masters: * DAPD02 / 10.69.80.1 / master Status: healthy [!] overlay packet loss for node 10.69.89.3 is higher than the allowed threshold of 20% (current packet loss at 100%) [!] overlay packet loss for node 10.69.80.2 is higher than the allowed threshold of 20% (current packet loss at 100%) Remote access: online * DWPD03 / 10.69.80.2 / master Status: healthy [!] overlay packet loss for node 10.69.80.1 is higher than the allowed threshold of 20% (current packet loss at 100%) [!] overlay packet loss for node 10.69.89.3 is higher than the allowed threshold of 20% (current packet loss at 100%) Remote access: online * DDPD04 / 10.69.89.3 / master Status: healthy [!] overlay packet loss for node 10.69.80.2 is higher than the allowed threshold of 20% (current packet loss at 100%) [!] overlay packet loss for node 10.69.80.1 is higher than the allowed threshold of 20% (current packet loss at 100%) Remote access: online ================
Are there any plans to support HTTP/2 for HEC inputs?
I have panel on a dashboard that lists events in a security log.  I can list them by Event ID but I would like it listed by Event ID count so that the most frequent are at the top.  If I change "coun... See more...
I have panel on a dashboard that lists events in a security log.  I can list them by Event ID but I would like it listed by Event ID count so that the most frequent are at the top.  If I change "count by Event" to "count by count" I get an error "The output field 'count ' cannot have the same name as a group by field." <query>index="wineventlog" $Site_Token$ $Cmptr_Token$ $Type$ LogName="Security" Type=Information | stats count by Event</query> How do I get it to list them in descending order by count?
Has anybody used or currently using DB Connect to their Red hat satellite Server? 
Hello, We have an application pulling search results from a scheduled search using Splunk API periodically, but encountering an issue where there is an excess of expired jobs (5000+) which are bein... See more...
Hello, We have an application pulling search results from a scheduled search using Splunk API periodically, but encountering an issue where there is an excess of expired jobs (5000+) which are being kept for 1 month+ for some reason. Because the application has to look through each of these jobs it's taking too long and timing out.  We tried deleting the expired jobs through the UI but they keep popping back up/not going away. Some of these now say "Invalid SID" when I try to inspect them. Is there any way we can clear these bulk, preferable without resorting to UI (which only shows 50 at a time)? 
I have some passive dns data that has time stamps that look like this in JSON logs: {"timestamp":"2021-10-21 16:31:01","timestamp_s":1634833861,"timestamp_ms":973448,  So it has first conventional ... See more...
I have some passive dns data that has time stamps that look like this in JSON logs: {"timestamp":"2021-10-21 16:31:01","timestamp_s":1634833861,"timestamp_ms":973448,  So it has first conventional time stamp and then a full seconds based Unix Epoch Time Stamp in seconds followed by: timestamp_ms":990877 This has the millsecs of the time only (actually microseconds).  The more convention time would have been: timestamp_s":1634834347.990877  I have not been able to get the time to include the millisec value included so far.  I am using a TIME_PREFIX that should skip the conventional timestamp.   Most recently, I used SEDCMD to get the time stamp to look more normal for epoch time --- timestamp_s":1634834347.990877,  but maybe the SEDCMD only happens after the time stamp is determined. I have used similar to for this. TIME_PREFIX=timestamp_s": TIME_FORMAT= %s.%6N Any help appreciated !       
We just stood up a new distributed deployment with 3 indexers and a CM. I was able to connect 1 indexer to the CM successfully but when I was trying to connect the other 2 indexers to it, I was getti... See more...
We just stood up a new distributed deployment with 3 indexers and a CM. I was able to connect 1 indexer to the CM successfully but when I was trying to connect the other 2 indexers to it, I was getting the error "Could not contact manager. Check that the manager is up, the manager_uri=https://xxxxxxx:8089 and secret are specified correctly. I know the secret is right and it is the correct uri, firewalld is disabled, I am able to netcat to the host via 8089, indexer GUIDs are unique.  Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Connected to xxxxxx:8089. Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
Hello, I can estimate the # of average events SPLUNK has for an index/sourcetype using following line of query /codes. How I would estimate the average Volume of data (in MB) SPLUNK receives per our... See more...
Hello, I can estimate the # of average events SPLUNK has for an index/sourcetype using following line of query /codes. How I would estimate the average Volume of data (in MB) SPLUNK receives per our for that index.  Thank you so much, appreciate your support. Query to Estimate # of Ave Events per hour: index=win_test sourcetype=* |bucket _time span=1h|stats count by _time|stats avg(count) as "Ave Events per Hour"    
Hi I have the following command in my query    My splunk search | eval message=IF((like(source,"ABC%") OR like(source,"DEF%")) AND avg_latency>120 ,"Host with more than 2 minutes Latency","")    ... See more...
Hi I have the following command in my query    My splunk search | eval message=IF((like(source,"ABC%") OR like(source,"DEF%")) AND avg_latency>120 ,"Host with more than 2 minutes Latency","")     where avg_latency is a field with values but for some reason the above condition is not working for me.    Could someone check if there is any format issue on my eval condition and let me know how I can make it correct?
@Kenshiro70  I have just read your most brilliant answer hear https://community.splunk.com/t5/Splunk-Search/What-exactly-are-the-rules-requirements-for-using-quot-tstats/m-p/319801 I have applied i... See more...
@Kenshiro70  I have just read your most brilliant answer hear https://community.splunk.com/t5/Splunk-Search/What-exactly-are-the-rules-requirements-for-using-quot-tstats/m-p/319801 I have applied it to a one use case, but I am a little stuck now on another use case and I was hoping you might be able to give me 5 minutes, please.  The following code is working. I have used it to replace a join. The issue is when I need to add a third mstats. There are just some rules i can't see to understand or crack etc... Any help would be  great - cheers. It is when I am adding additional ""by" clause "used.by".   I supost the really question is how to handle this when there are multiple BY form different | mstats         | mstats append=t prestats=t min("mx.service.status") min(mx.service.dependencies.status) min(mx.service.resources.status) min("mx.service.deployment.status") max("mx.service.replicas") WHERE "index"="metrics_test" service.type IN (agent-based launcher-based) AND mx.env=http://mx20267vm:15000 span=10s BY "service.name" "service.type" | mstats append=t prestats=t max("mx.service.replicas") WHERE "index"="metrics_test" AND mx.env=http://mx20267vm:15000 service.type IN (agent-based launcher-based) span=10s BY service.name | eval forked="" | mstats append=t prestats=t min("mx.service.deployment.status") max("mx.service.replicas") WHERE "index"="metrics_test" service.type IN (agent-based launcher-based) AND mx.env=http://mx20267vm:15000 span=10s BY "service.name" "service.type" forked | mstats append=t prestats=t min(mx.service.dependencies.status) WHERE "index"="metrics_test" service.type IN (agent-based launcher-based) AND mx.env=http://mx20267vm:15000 span=10s | rename service.name as Service_Name,service.type as Service_Type | stats max("mx.service.replicas") as replicas min("mx.service.deployment.status") as Deployment min("mx.service.status") as Status_numeric min(mx.service.dependencies.status) as Dependencies min(mx.service.resources.status) as Resources by _time Service_Name Service_Type forked | sort 0 _time Service_Name             Working   This is the code that is not working. I added in a "used.by" in the first tstats as it is needed for min(mx.service.dependencies.status) -  However as soon as i add this i loose a lot of data           | mstats append=t prestats=t min(mx.service.dependencies.status) min("mx.service.deployment.status") max("mx.service.replicas") WHERE "index"="metrics_test" service.type IN (agent-based launcher-based) AND mx.env=http://mx20267vm:15000 span=10s BY "service.name" "service.type" "used.by" | eval forked="" | mstats append=t prestats=t min("mx.service.deployment.status") max("mx.service.replicas") WHERE "index"="metrics_test" service.type IN (agent-based launcher-based) AND mx.env=http://mx20267vm:15000 span=10s BY "service.name" "service.type" "forked" | mstats append=t prestats=t max("mx.service.replicas") WHERE "index"="metrics_test" AND mx.env=http://mx20267vm:15000 service.type IN (agent-based launcher-based) span=10s BY service.name | rename service.name as Service_Name,service.type as Service_Type | stats min("mx.service.deployment.status") as Deployment min(mx.service.dependencies.status) as Dependencies_x max("mx.service.replicas") as replicas by _time Service_Name Service_Type forked "used.by" | sort 0 - Service_Name _time           NOt working  
Hello, I have an index and 3 custom sourcetypes built in place, Suppose if the source wants to stream logs into Splunk, do i need to create 3 HEC tokens? I can see when i am trying to create HEC inp... See more...
Hello, I have an index and 3 custom sourcetypes built in place, Suppose if the source wants to stream logs into Splunk, do i need to create 3 HEC tokens? I can see when i am trying to create HEC inputs, it is asking me to select sourcetype where i can only select one sourcetype. Please help me with this situation.   Thanks
Hello.  I am running 8.2.2 on Linux.  We have four clustered indexers and are using SmartStore.  I would like to empty an index (and recover the disk space).  I have thus chosen to remove the old_dat... See more...
Hello.  I am running 8.2.2 on Linux.  We have four clustered indexers and are using SmartStore.  I would like to empty an index (and recover the disk space).  I have thus chosen to remove the old_data index from the cluster, then add it back again.  I have performed these steps: 1. Stop any data being sent to the index. 2. Edit indexes.conf and delete the index's stanza (via the CM) then apply the changes to the peer nodes (each restarts). 3. Remove the index's directories from each peer node. 4. Check on the SHC for events in the index (index=old_data); no events are returned (all time). 5. Once the cluster shows that all indexes are 'green', re-add the index as normnal (editing indexes.conf again and applying the update). However, now searching the index on the SHC returns some/most of the events.  My guess is that the cache manager / the S3 storage also needs to be purged.   If so, how is this best achieved? I have avoided using index=old_data | delete because I understand this will only mask the data from searches (and I want the disk space back too). Many thanks for your time.
Hi, I would like to count the values of a multivalue field by value. For example:   | makeresults | eval values_type=split( "value1,value2,value1,value2,value1,value2,value1,value2,value2,value2,... See more...
Hi, I would like to count the values of a multivalue field by value. For example:   | makeresults | eval values_type=split( "value1,value2,value1,value2,value1,value2,value1,value2,value2,value2,value2,",",") | eval values_count=mvcount(values_type) | eval value1=mvfilter(match(values_type,"value1")) | eval value1_count=mvcount(value1) | eval value2_count=values_count - value1_count | table values_type message_count values_count value1_count value2_count Is there another way to do this? For example, if I don't know the possible values, this way doesn't work. Thanks in advance  
Analyze yarn logs on the Hadoop cluster by using Splunk yarn logs are stored different nodes in the Hadoop cluster. For this requirement what are the configuration required  should install splunk ... See more...
Analyze yarn logs on the Hadoop cluster by using Splunk yarn logs are stored different nodes in the Hadoop cluster. For this requirement what are the configuration required  should install splunk forwarder on all the nodes or edge node ? what are the configurations required. Thanks 
Hello All, I have a use case to consume alerts from a tool called dataminr into splunk. Can someone suggest us the best approach for this integration?     Thanks