All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

If anyone knows, could you please let me know the following? Our Splunk Enterprise system is based on AWS EC2.We use AWS S3 for Splunk SmartStore.We take backup of EBS and S3(smartstore) everyday.Bu... See more...
If anyone knows, could you please let me know the following? Our Splunk Enterprise system is based on AWS EC2.We use AWS S3 for Splunk SmartStore.We take backup of EBS and S3(smartstore) everyday.But the cost of S3 backup is very high.So we are planning to stop backup of  S3.Because we have Smart S3 versioning turned on. If we had to restore the EC2 , could we restore from S3 versioning? Since the timing of EBS backup and S3 versioning are different, I think there may be a problem if I restore EBS and restore S3 from a previous version at a different slice. Since I want to prioritize cost reduction, it is not a problem if some data on the EBS side is missing as long as it can be read without any problems. Please let me know if you have any ideas on how to reduce backup costs in a similar environment.
Hello Splunk Community,   We are currently transitioning a live dashboard from Splunk Enterprise to Splunk Cloud. The original dashboard was built using HTML, CSS, and JavaScript to monitor three c... See more...
Hello Splunk Community,   We are currently transitioning a live dashboard from Splunk Enterprise to Splunk Cloud. The original dashboard was built using HTML, CSS, and JavaScript to monitor three critical KPIs: Success, Response, and Availability. In our Splunk Enterprise version, we implemented: A custom alarm beep and red colour flash when any KPI drops below 99%. Pagination to manage and display data effectively.   However, while recreating this dashboard in Splunk Cloud Studio, we’ve encountered limitations: It seems that Studio doesn’t support custom pagination features. We're unable to add the alarm beep and visual alerts as we did use HTML/CSS/JS in Enterprise. 3. Federated searches limitation and how we can use for this requirement and how to their license usage Could someone please clarify:  What are the limitations of using HTML, CSS, and JavaScript in Splunk Cloud and Studio dashboards? Are there any workarounds or supported methods to implement these types of visual and audio alerts in Splunk Cloud? Any suggestions or examples on how to handle pagination and threshold-based alerts within Studio? Your insights or best practices would be greatly appreciated.  Thank you in advance for your support!    
Hi @KishoreSrini  I think the collectd and runsvc.sh logs are not Splunk related, these look like they might be associated with VstsAgentService - Is this a VM running on Azure / Azure Pipelines? R... See more...
Hi @KishoreSrini  I think the collectd and runsvc.sh logs are not Splunk related, these look like they might be associated with VstsAgentService - Is this a VM running on Azure / Azure Pipelines? Regarding the Splunk error failed to open file - Can you confirm if the file actually exists in the filesystem? And if so, what events are in the splunkd.log? Are there any warnings/errors? Please could you confirm the ownership on /opt/splunkforwarder/var/log/splunk/splunkd.log and also confirm the user which Splunk is running as: ps -a | grep -i splunk  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
No, I am not using a Splunk add-on I am using the Splunk forwarder to send the logs
@KishoreSrini  Can you check if there is any permission issue?  collectd: processmon plugin: Error reading /proc/3605381/stat collectd failed to read process stats, likely because the process with... See more...
@KishoreSrini  Can you check if there is any permission issue?  collectd: processmon plugin: Error reading /proc/3605381/stat collectd failed to read process stats, likely because the process with PID 3605381 ended or permissions were insufficient "/opt/splunkforwarder/var/log/splunk/splunkd.log": No such file or directory - Splunk couldn't access it's main splunkd.log file this also indicates about file unavailablity or permission issue Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!
Thank you for your reply. I am using Splunk 9.4.2 which is the latest version as of now.
I am newbie to this env and I'm trying to understand some logs regrading a linux server troubleshoot. A server stopped sending metrics to Splunk (eventlogs are fine). To troubleshoot, I searched the ... See more...
I am newbie to this env and I'm trying to understand some logs regrading a linux server troubleshoot. A server stopped sending metrics to Splunk (eventlogs are fine). To troubleshoot, I searched the error logs on that time stamp. These are the logs I got, 15:02:44.000: collectd[909]: processmon plugin: Error reading /proc/3605381/stat 15:12:53.000: runsvc.sh[968]: Error reported in diagnostic logs. Please examine the log for more details. 15:12:53.000: runsvc.sh[968]: 2025-06-13 19:12:53Z: Agent connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected. 15:31:07.000: splunk[3844643]: ERROR - Failed opening "/opt/splunkforwarder/var/log/splunk/splunkd.log": No such file or directory Please help to understand the issue and troubleshooting steps for the issue(If possible) Thank you in advance.
Hello @_joe, If it is mentioned on the Splunkbase, then the TA would be compatible with the Splunk version. However, we will need more info on the ERROR log that you're receiving to understand why t... See more...
Hello @_joe, If it is mentioned on the Splunkbase, then the TA would be compatible with the Splunk version. However, we will need more info on the ERROR log that you're receiving to understand why the input won't run.  Check if you can enable the DEBUG logging and what ERROR does the python script log and we can take it from there. Thanks, Tejas.
@ITWhisperer So, if I remove the depends attributes, will it start working for the users?
@bakeery  Are you using sysmon add-on? #https://splunkbase.splunk.com/app/5709 Also refer below #https://community.splunk.com/t5/Getting-Data-In/Why-isn-t-the-Sysmon-Technology-Add-on-Parsing-my... See more...
@bakeery  Are you using sysmon add-on? #https://splunkbase.splunk.com/app/5709 Also refer below #https://community.splunk.com/t5/Getting-Data-In/Why-isn-t-the-Sysmon-Technology-Add-on-Parsing-my-Sysmon-Logs/m-p/370757   Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!  
@ITWhisperer  Below is my complete code. So do you any any possibility which is causing an issue ? What Can i fix here so it will work for all the users ? <form script="common:common.js, common:re... See more...
@ITWhisperer  Below is my complete code. So do you any any possibility which is causing an issue ? What Can i fix here so it will work for all the users ? <form script="common:common.js, common:remove_elements.js, customer_reports:attrchange.js, customer_reports:kpi_dashboard.js" stylesheet="common:vanderlande.css, kpi_dashboard.css, common:project_specific.css" hideEdit="true" version="1.1"> <label>KPI Dashboard</label> <search id="thresholds"> <query>| savedsearch report_kpi_dashboard_thresholds</query> <finalized> <!-- Availability --> <eval token="availability_min_value">$result.system_dashboard_availability_min_value$-0.0001</eval> <eval token="availability_max_value">$result.system_dashboard_availability_max_value$+0.0001</eval> <set token="availability_max_range_threshold">$result.system_dashboard_availability_max_range_threshold$</set> <!-- Completeness --> <eval token="completeness_min_value">$result.system_dashboard_completeness_min_value$-0.0001</eval> <eval token="completeness_max_value">$result.system_dashboard_completeness_max_value$+0.0001</eval> <set token="completeness_max_range_threshold">$result.system_dashboard_completeness_max_range_threshold$</set> <!-- Group Coherence --> <eval token="group_coherence_min_value">$result.system_dashboard_group_coherence_min_value$-0.0001</eval> <eval token="group_coherence_max_value">$result.system_dashboard_group_coherence_max_value$+0.0001</eval> <set token="group_coherence_max_range_threshold">$result.system_dashboard_group_coherence_max_range_threshold$</set> <!-- Lead Time --> <eval token="lead_time_min_value">$result.system_dashboard_lead_time_min_value$-0.0001</eval> <eval token="lead_time_max_value">$result.system_dashboard_lead_time_max_value$+0.0001</eval> <set token="lead_time_max_range_threshold">$result.system_dashboard_lead_time_max_range_threshold$</set> <!-- On-Time --> <eval token="on-time_min_value">$result.system_dashboard_on-time_min_value$-0.0001</eval> <eval token="on-time_max_value">$result.system_dashboard_on-time_max_value$+0.0001</eval> <set token="on-time_max_range_threshold">$result.system_dashboard_on-time_max_range_threshold$</set> <!-- Partitioning --> <eval token="partitioning_min_value">$result.system_dashboard_partitioning_min_value$-0.0001</eval> <eval token="partitioning_max_value">$result.system_dashboard_partitioning_max_value$+0.0001</eval> <set token="partitioning_max_range_threshold">$result.system_dashboard_partitioning_max_range_threshold$</set> <!-- Productivity --> <eval token="productivity_min_value">$result.system_dashboard_productivity_min_value$-0.0001</eval> <eval token="productivity_max_value">$result.system_dashboard_productivity_max_value$+0.0001</eval> <set token="productivity_max_range_threshold">$result.system_dashboard_productivity_max_range_threshold$</set> <!-- Throughput --> <eval token="throughput_min_value">$result.system_dashboard_throughput_min_value$-0.0001</eval> <eval token="throughput_max_value">$result.system_dashboard_throughput_max_value$+0.0001</eval> <set token="throughput_max_range_threshold">$result.system_dashboard_throughput_max_range_threshold$</set> <!-- Utilization --> <eval token="utilization_min_value">$result.system_dashboard_utilization_min_value$-0.0001</eval> <eval token="utilization_max_value">$result.system_dashboard_utilization_max_value$+0.0001</eval> <set token="utilization_max_range_threshold">$result.system_dashboard_utilization_max_range_threshold$</set> </finalized> </search> <search id="operational_hours"> <query>| savedsearch set_operational_hours</query> <finalized> <set token="operational_start_time">$result.operational_start_time$</set> <set token="operational_end_time">$result.operational_end_time$</set> <eval token="time.earliest_epoch">if($form.time.earliest$="",0,if(isnum($form.time.earliest$),$form.time.earliest$,relative_time(now(),$form.time.earliest$)))+($operational_start_time$*3600)</eval> <eval token="time.latest_epoch">if(isnum($form.time.latest$),$form.time.latest$,relative_time(now(),$form.time.latest$))+($operational_start_time$*3600)</eval> </finalized> </search> <fieldset autoRun="true" submitButton="false"> <input id="time" type="time" token="time" depends="$operational_start_time$, $operational_end_time$" searchWhenChanged="true"> <label>Time</label> <default> <earliest>-7d@h</earliest> <latest>now</latest> </default> <change> <eval token="time.earliest_epoch">if('earliest'="",0,if(isnum('earliest'),'earliest'+($operational_start_time$*3600),relative_time(now(),'earliest')))+($operational_start_time$*3600)</eval> <eval token="time.latest_epoch">if('earliest'="",0,if(isnum('earliest'),'earliest'+(86400)+($operational_start_time$*3600),relative_time(now(),'earliest')))+(86400)+($operational_start_time$*3600)</eval> </change> </input> </fieldset> <row> <panel> <title>Throughput</title> <chart id="throughput"> <search> <query>| savedsearch report_kpi_dashboard_throughput operational_start_time=$operational_start_time$ operational_end_time=$operational_end_time$ earliest_epoch=$time.earliest_epoch$</query> <earliest>$time.earliest$</earliest> <latest>$time.latest$</latest> </search> <option name="charting.chart">radialGauge</option> <option name="charting.chart.rangeValues">[$throughput_min_value$,$throughput_max_range_threshold$,$throughput_max_value$]</option> <option name="charting.chart.showMinorTicks">1</option> <option name="charting.chart.style">minimal</option> <option name="charting.gaugeColors">["0x3FC77A","0xB44441"]</option> <option name="refresh.display">progressbar</option> </chart> <html> <center>Cases</center> </html> </panel> <panel> <title>Availability</title> <chart id="availability"> <search> <query>| savedsearch report_kpi_dashboard_availability operational_start_time=$operational_start_time$ operational_end_time=$operational_end_time$ earliest_epoch=$time.earliest_epoch$</query> <earliest>$time.earliest$</earliest> <latest>$time.latest$</latest> </search> <option name="charting.chart">radialGauge</option> <option name="charting.chart.rangeValues">[$availability_min_value$,$availability_max_range_threshold$,$availability_max_value$]</option> <option name="charting.chart.showMinorTicks">1</option> <option name="charting.chart.style">minimal</option> <option name="charting.gaugeColors">["0xB44441","0x3FC77A"]</option> <option name="refresh.display">progressbar</option> </chart> <html> <center>Percentage</center> </html> </panel> <panel> <title>Completeness</title> <chart id="completeness"> <search> <query>| savedsearch report_kpi_dashboard_completeness operational_start_time=$operational_start_time$ operational_end_time=$operational_end_time$ earliest_epoch=$time.earliest_epoch$</query> <earliest>$time.earliest$</earliest> <latest>$time.latest$</latest> </search> <option name="charting.chart">radialGauge</option> <option name="charting.chart.rangeValues">[$completeness_min_value$,$completeness_max_range_threshold$,$completeness_max_value$]</option> <option name="charting.chart.showMinorTicks">1</option> <option name="charting.chart.style">minimal</option> <option name="charting.gaugeColors">["0xB44441","0x3FC77A"]</option> <option name="refresh.display">progressbar</option> </chart> <html> <center>Percentage</center> </html> </panel> <panel> <title>On-Time</title> <chart id="on_time"> <search> <query>| savedsearch report_kpi_dashboard_on_time operational_start_time=$operational_start_time$ operational_end_time=$operational_end_time$ earliest_epoch=$time.earliest_epoch$</query> <earliest>$time.earliest$</earliest> <latest>$time.latest$</latest> </search> <option name="charting.chart">radialGauge</option> <option name="charting.chart.rangeValues">[$on-time_min_value$,$on-time_max_range_threshold$,$on-time_max_value$]</option> <option name="charting.chart.showMinorTicks">1</option> <option name="charting.chart.style">minimal</option> <option name="charting.gaugeColors">["0xB44441","0x3FC77A"]</option> <option name="refresh.display">progressbar</option> </chart> <html> <center>Percentage</center> </html> </panel> <panel> <title>Lead Time</title> <chart id="lead_time"> <search> <finalized> <eval token="lead_time">now()</eval> </finalized> <query>| savedsearch report_kpi_dashboard_lead_time operational_start_time=$operational_start_time$ operational_end_time=$operational_end_time$ earliest_epoch=$time.earliest_epoch$</query> <earliest>$time.earliest$</earliest> <latest>$time.latest$</latest> </search> <option name="charting.chart">radialGauge</option> <option name="charting.chart.majorUnit">1800</option> <option name="charting.chart.rangeValues">[$lead_time_min_value$,$lead_time_max_range_threshold$,$lead_time_max_value$]</option> <option name="charting.chart.showMinorTicks">1</option> <option name="charting.chart.style">minimal</option> <option name="charting.gaugeColors">["0x3FC77A","0xB44441"]</option> <option name="refresh.display">progressbar</option> </chart> <html> <center>Duration (hh:mm)</center> </html> </panel> </row> <row> <panel> <title>Utilization</title> <chart id="utilization"> <search> <query>| savedsearch report_kpi_dashboard_lfl_utilisation operational_start_time=$operational_start_time$ operational_end_time=$operational_end_time$ earliest_epoch=$time.earliest_epoch$</query> <earliest>$time.earliest$</earliest> <latest>$time.latest$</latest> </search> <option name="charting.chart">radialGauge</option> <option name="charting.chart.rangeValues">[$utilization_min_value$,$utilization_max_range_threshold$,$utilization_max_value$]</option> <option name="charting.chart.showMinorTicks">1</option> <option name="charting.chart.style">minimal</option> <option name="charting.gaugeColors">["0xB44441","0x3FC77A"]</option> <option name="refresh.display">progressbar</option> </chart> <html> <center>Percentage</center> </html> </panel> <panel> <title>Partitioning</title> <chart id="partitioning"> <search> <query>| savedsearch report_kpi_dashboard_lfl_partitioning operational_start_time=$operational_start_time$ operational_end_time=$operational_end_time$ earliest_epoch=$time.earliest_epoch$</query> <earliest>$time.earliest$</earliest> <latest>$time.latest$</latest> </search> <option name="charting.chart">radialGauge</option> <option name="charting.chart.rangeValues">[$partitioning_min_value$,$partitioning_max_range_threshold$,$partitioning_max_value$]</option> <option name="charting.chart.showMinorTicks">1</option> <option name="charting.chart.style">minimal</option> <option name="charting.gaugeColors">["0xB44441","0x3FC77A"]</option> <option name="refresh.display">progressbar</option> </chart> <html> <center>Percentage</center> </html> </panel> <panel> <title>Group Coherence</title> <chart id="group_coherence"> <search> <query>| savedsearch report_kpi_dashboard_lfl_group_coherence operational_start_time=$operational_start_time$ operational_end_time=$operational_end_time$ earliest_epoch=$time.earliest_epoch$</query> <earliest>$time.earliest$</earliest> <latest>$time.latest$</latest> </search> <option name="charting.chart">radialGauge</option> <option name="charting.chart.rangeValues">[$group_coherence_min_value$,$group_coherence_max_range_threshold$,$group_coherence_max_value$]</option> <option name="charting.chart.showMinorTicks">1</option> <option name="charting.chart.style">minimal</option> <option name="charting.gaugeColors">["0xB44441","0x3FC77A"]</option> <option name="refresh.display">progressbar</option> </chart> <html> <center>Percentage</center> </html> </panel> <panel> <title>Productivity</title> <chart id="productivity"> <search> <query>| savedsearch report_kpi_dashboard_productivity operational_start_time=$operational_start_time$ operational_end_time=$operational_end_time$ earliest_epoch=$time.earliest_epoch$</query> <earliest>$time.earliest$</earliest> <latest>$time.latest$</latest> </search> <option name="charting.chart">radialGauge</option> <option name="charting.chart.rangeValues">[$productivity_min_value$,$productivity_max_range_threshold$,$productivity_max_value$]</option> <option name="charting.chart.showMinorTicks">1</option> <option name="charting.chart.style">minimal</option> <option name="charting.gaugeColors">["0xB44441","0x3FC77A"]</option> <option name="refresh.display">progressbar</option> </chart> <html> <center>Cases/Hour</center> </html> </panel> <panel></panel> </row> </form>   
@msatish - As mentioned by @dionrivera you can use SC4S CEF for parsing.   But if you want to parse the already ingested CEF formatted data in Splunk then you can use this App's custom search comma... See more...
@msatish - As mentioned by @dionrivera you can use SC4S CEF for parsing.   But if you want to parse the already ingested CEF formatted data in Splunk then you can use this App's custom search command to do that. https://splunkbase.splunk.com/app/7701  
what will be the log file name? As I mentioned i did not do any configuration for logging. helper.log_info was already present in the default python script.
Have you started your instance(s) every time after you have applied a new version? This is needed to make a needed conversions e.g. from 8.2.8 -> 9.1.0 etc.! Without those starts it’s almost same to d... See more...
Have you started your instance(s) every time after you have applied a new version? This is needed to make a needed conversions e.g. from 8.2.8 -> 9.1.0 etc.! Without those starts it’s almost same to do it directly 8.2.8 -> 9.3.4 especially if you are using tar.gz package. With rpm and deb installing a new, removing some old unneeded files too. But all conversion tasks have done only when you are starting the instance.
You could use dedup with sortby parameter, as I previously show.
Indeed lookups often end up with multivalue.  You need to make sure that every field to include have equal number of values. Usually I am in favor of JSON like in @livehybrid 's suggestion, although... See more...
Indeed lookups often end up with multivalue.  You need to make sure that every field to include have equal number of values. Usually I am in favor of JSON like in @livehybrid 's suggestion, although it should not be that complex; especially, one should not compose JSON with string.  More on JSON later.  There is an even simpler approach if you can enumerate fields to include: multikv.  No mvexpand needed.  Here is how: | eval _raw = mvappend("FunctionGroup,MsgNr,alarm_severity,area,equipment", mvzip(mvzip(mvzip(mvzip(FunctionGroup,MsgNr, ","), alarm_severity, ","), area), equipment, ",") ) | multikv forceheader=1 | fields - _raw linecount The idea is to compose a CSV table with mvzip, then extract from this table.  If composing nested mvzip is too much, or if you cannot easily enumerate fields to include, you can add foreach to your arsenal: | rename FunctionGroup as _raw | eval header = "FunctionGroup" | foreach MsgNr,alarm_severity,area,equipment [ eval _raw = mvzip(_raw, <<FIELD>>, ","), header = header . "," . "<<FIELD>>"] | eval _raw = mvappend(header, _raw) | multikv forceheader=1 | fields - _raw header linecount  Now, back to JSON - in this use case, it is more involved than multikv.  Again, with help of foreach and provided that your Splunk version is 8.1 or later, this is a semantic way to do it: | eval jcombo = json_object() | eval idx = mvrange(0, mvcount(FunctionGroup)) | foreach FunctionGroup MsgNr alarm_severity area equipment [ eval jcombo = json_set(jcombo, "<<FIELD>>", mvindex(<<FIELD>>, idx))] | fields - FunctionGroup MsgNr alarm_severity area equipment | mvexpand jcombo | fields - idx jcombo Of course, you can also do this without foreach.
Is this the result you are looking for? ID billing_date code latest(cost) _time 10001 2025-05-01 product2 135.75 2025-05-02 10:15:00 10001 2025-05-01 product3 155.00 2025-05-02 ... See more...
Is this the result you are looking for? ID billing_date code latest(cost) _time 10001 2025-05-01 product2 135.75 2025-05-02 10:15:00 10001 2025-05-01 product3 155.00 2025-05-02 13:30:00 10001 2025-06-01 product1 102.50 2025-06-01 08:10:00 10001 2025-06-01 product2 130.75 2025-06-02 10:15:00 10001 2025-06-01 product3 150.00 2025-06-02 13:30:00 dedup with perfect sort as @PickleRick suggests should work.  Another way is to simply use stats as I originally suggested: | stats latest(cost) max(_time) as _time by ID billing_date code  
Hello @new ,  Can you try to directly run search with log file name or an keyword around logs of that custom add-on on Splunk Cloud and check how it goes?
Hi @bakeery  Please can you confirm which UF version you are running on? There is a known issue (SPL-217199) in < 9.0.1 relating to WinEventLog sourcetype having encoded broken fields appended and I... See more...
Hi @bakeery  Please can you confirm which UF version you are running on? There is a known issue (SPL-217199) in < 9.0.1 relating to WinEventLog sourcetype having encoded broken fields appended and I'm wondering if this could be related?  See https://splunk.my.site.com/customer/s/article/Special-characters-in-sourcetype-for-windows-data-in-UF for more info. If you are on <9.0.1 I would recommend upgrading to see if this resolves the issue.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @Mirza_Jaffar1  What was the previous version and current version you are on now? Did you get a clean start when starting after upgrading to the previous version from the version before it?  D... See more...
Hi @Mirza_Jaffar1  What was the previous version and current version you are on now? Did you get a clean start when starting after upgrading to the previous version from the version before it?  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing