All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Have events like below 1) date-Timestamp Server - hostname Status - host is down Threshold - unable to ping   2)  Date-Timestamp Db - dbname Status- database is down Instance status- DB ins... See more...
Have events like below 1) date-Timestamp Server - hostname Status - host is down Threshold - unable to ping   2)  Date-Timestamp Db - dbname Status- database is down Instance status- DB instance is not available    I would need to write Eval condition and create new field description that if field status is " database is down" , I need to add date, dB, status, Instances status fields to description field   And if status is host down, need to add date,server, status, threshold to description field.
Thanks @livehybrid  The issue was at the firewall side. It was dropping packets, so the policies/rules had to applied again. Also added some useful link which i came across. https://community.splun... See more...
Thanks @livehybrid  The issue was at the firewall side. It was dropping packets, so the policies/rules had to applied again. Also added some useful link which i came across. https://community.splunk.com/t5/Security/ERROR-TcpOutputFd-Connection-to-host-c-9997-failed-sock-error/m-p/487895    https://splunk.my.site.com/customer/s/article/Splunk-Universal-Forwarder-is-not-sending-events-to-Splunk-Cloud  
it was an issue with firewall rules, as it dropped packets.
Hi @Ciccius  This is likely to be an issue with permissions, please could you validate that the permissions within the rest_ta app in $SPLUNK_HOME/etc/apps/rest_ta is the same across your SHC?   ... See more...
Hi @Ciccius  This is likely to be an issue with permissions, please could you validate that the permissions within the rest_ta app in $SPLUNK_HOME/etc/apps/rest_ta is the same across your SHC?   Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi asimit, thank you very much, it was a permission issue. I don't know why the user/group for app rest_ta was root/root, once I reset to splunk/splunk it worked. Thanks!
Hi @StephenD1 , Yes, Splunk Enterprise 9.4.0 (build 6b4ebe426ca6) is affected by CVE-2024-7264. This vulnerability affects libcurl versions between 7.32.0 and prior to 8.9.1, and as you confirmed,... See more...
Hi @StephenD1 , Yes, Splunk Enterprise 9.4.0 (build 6b4ebe426ca6) is affected by CVE-2024-7264. This vulnerability affects libcurl versions between 7.32.0 and prior to 8.9.1, and as you confirmed, your installation includes libcurl 7.61.1, which falls within this range. ## Official Fix According to the latest information: - The Splunk fix is identified as SPL-270280 - The fix has been included in Splunk Enterprise 9.4.2 - The fix has also been backported to supported older versions: 9.3.4, 9.2.6, and 9.1.9 ## Recommended Actions ### Option 1: Upgrade to a Patched Version The most comprehensive solution is to upgrade to one of the fixed versions: - Splunk Enterprise 9.4.2 (preferred for your current version) - Or one of the other patched versions (9.3.4, 9.2.6, or 9.1.9) ### Option 2: Disable KVStore (MongoDB) Temporarily If you cannot upgrade immediately, you can consider disabling the KVStore service, which uses MongoDB: 1. Check if any critical apps depend on KVStore: ``` splunk list kvstore -collections ``` 2. Disable KVStore: ``` splunk disable kvstore splunk restart ``` 3. Verify MongoDB is no longer running: ``` ps -ef | grep mongo ``` Note that disabling KVStore will impact any apps that rely on it, including: - Enterprise Security - ITSI - Splunk App for Infrastructure - Some custom apps that use KVStore collections ### Option 3: Mitigate Risk Through Network Controls If you can't upgrade or disable KVStore: - Ensure MongoDB is properly configured to only listen on localhost - Implement additional network controls to restrict access to the MongoDB port (typically 8191) - Monitor for potential exploitation attempts ## Additional Information You can find more details in the Splunk article regarding this vulnerability: https://splunk.my.site.com/customer/s/article/Splunk-vulnerability-libcurl-7-32-0-8-9-1-DoS-CVE-2024-7264 The CVE-2024-7264 is a denial-of-service vulnerability in libcurl that could allow a malicious server to cause a denial of service by sending specially crafted responses that trigger excessive memory consumption. ## Long-term Recommendation For a more permanent solution, plan to upgrade to the patched version as soon as your change management process allows. This is especially important if you have internet-facing Splunk components that might be vulnerable to this exploitation vector. Please give  for support  happly splunking .... 
Hi @Ciccius  Based on the error message you're receiving, this appears to be a permissions issue with the rest_ta app. The specific error "Could not find writer for: /nobody/rest_ta/app/install/stat... See more...
Hi @Ciccius  Based on the error message you're receiving, this appears to be a permissions issue with the rest_ta app. The specific error "Could not find writer for: /nobody/rest_ta/app/install/state" suggests that Splunk doesn't have the proper permissions to update the app's state. ## Troubleshooting steps: 1. **Check permissions on the app directory**: ``` sudo ls -la /opt/splunk/etc/apps/rest_ta/ ``` Make sure the directory and files are owned by the Splunk user and group. 2. **Fix permissions if needed**: ``` sudo chown -R splunk:splunk /opt/splunk/etc/apps/rest_ta/ sudo chmod -R 755 /opt/splunk/etc/apps/rest_ta/ ``` 3. **Try disabling the app manually before deployment**: - On the deployment server, edit `/opt/splunk/etc/apps/rest_ta/default/app.conf` - Set `state = disabled` in the `[install]` section - Or completely remove the app if it's not needed: `sudo rm -rf /opt/splunk/etc/apps/rest_ta/` 4. **Check for file system issues**: - The error might indicate file system corruption or disk issues - Run `df -h` to check disk space (you mentioned this is fine) - Run `sudo touch /opt/splunk/etc/test.txt` to verify write permissions to the directory 5. **Validate the deployment server's configuration**: ``` sudo /opt/splunk/bin/splunk show shcluster-bundle-status ``` 6. **Restart Splunk on both servers**: ``` sudo /opt/splunk/bin/splunk restart ``` 7. **Deploy without the problematic app**: - Temporarily move the app out of the deployment directory - Try the deployment again - If successful, the issue is definitely with the app itself If the issue persists, you may need to check Splunk logs for more details: ``` sudo cat /opt/splunk/var/log/splunk/splunkd.log | grep rest_ta ``` Let me know if any of these steps help resolve the issue!
Hi all, I am trying to deploy my apps from the deployment server with the command:  /opt/splunk/bin/splunk apply shcluster-bundle -target https://splunksrc:8089 -preserve-lookups true It never fai... See more...
Hi all, I am trying to deploy my apps from the deployment server with the command:  /opt/splunk/bin/splunk apply shcluster-bundle -target https://splunksrc:8089 -preserve-lookups true It never failed to do the task but now I am getting this error: Error while deploying apps to first member, aborting apps deployment to all members: Error while deleting app=rest_ta on target=https://splunksrc:8089: Non-200/201 status_code=500; {"messages":[{"type":"ERROR","text":"\n In handler 'localapps': Cannot update application info: /nobody/rest_ta/app/install/state = disabled: Could not find writer for: /nobody/rest_ta/app/install/state [0] [/opt/splunk/etc]"}]} Both the nodes (deployment and splunksrc) have enough disk space. Any ideas? Thanks Francesco
Hi @sankardevarajan  The configuration you provided is for the OnBase application to send logs to Splunk, not for Splunk configuration itself. You need to configure the OnBase application's .config ... See more...
Hi @sankardevarajan  The configuration you provided is for the OnBase application to send logs to Splunk, not for Splunk configuration itself. You need to configure the OnBase application's .config file to send logs to Splunk. The configuration snippet you provided is for the Hyland.Logging component, which is part of the OnBase application. You need to modify the .config file ( likely Application-Server-Web.config or another relevant config file) on the OnBase Application Server to include the specified route. <Route name="Logging_Local_Splunk" > <add key="Splunk" value="http://your-splunk-heavy-forwarder-or-indexer:8088"/> <add key="SplunkToken" value="your-splunk-http-event-collector-token"/> <add key="DisableIPAddressMasking" value="false" /> </Route>   To receive these logs in Splunk Cloud, you need to: Set up an HTTP Event Collector (HEC) token in your Splunk Cloud instance. Configure the OnBase application to send logs to the HEC endpoint. In Splunk Cloud, you will need to create an HEC token and get the HEC endpoint URL. You can then use this token and endpoint URL in the OnBase application's .config file. The http://localhost:SplunkPort in the configuration should be replaced with the URL of your Splunk HEC endpoint (typically https://http-inputs-<stackName>.splunkcloud.com ) and SplunkTokenNumber should be replaced with the actual HEC token. For more information on configuring HEC in Splunk Cloud, refer to https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/UsetheHTTPEventCollector. For reference the current instructions for creating HEC tokens for Splunk Cloud are: Click Settings > Add Data. Click monitor. Click HTTP Event Collector. In the Name field, enter a name for the token. (Optional) In the Source name override field, enter a name for a source to be assigned to events that this endpoint generates. (Optional) In the Description field, enter a description for the input. (Optional) If you want to enable indexer acknowledgment for this token, click the Enable indexer acknowledgment checkbox. Click Next. (Optional) Make edits to source type and confirm the index where you want HEC events to be stored. See Modify input settings. Click Review. Confirm that all settings for the endpoint are what you want. If all settings are what you want, click Submit. Otherwise, click < to make changes. (Optional) Copy the token value that Splunk Web displays and paste it into another document for reference later. (Optional) Click Track deployment progress to see progress on how the token has been deployed to the rest of the Splunk Cloud Platform deployment. When you see a status of "Done", you can then use the token to send data to HEC. Ensure that the Splunk HEC endpoint is accessible from the OnBase Application Server. If it's not, you may need to set up a Heavy Forwarder to act as an intermediary.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
i want to onboard application logs into splunk cloud.  Hyland.Logging can be configured to send information to Splunk as well as the Diagnostics Console by modifying the .config file of the server. ... See more...
i want to onboard application logs into splunk cloud.  Hyland.Logging can be configured to send information to Splunk as well as the Diagnostics Console by modifying the .config file of the server. To configure Hyland.Logging to send information to Splunk: <Route name="Logging_Local_Splunk" > <add key="Splunk" value="http://localhost:SplunkPort"/> <add key="SplunkToken" value="SplunkTokenNumber"/> <add key="DisableIPAddressMasking" value="false" /> </Route> Configuring Hyland.Logging for Splunk • Application Server • Reader • Product Documentation  i am not understanding where we need to configure above config in Splunk. Much appreciated anyone guide me.
Yes. If you don't have "holes" in your firewall to send data directly from the other components to Qradar, it won't work. You might try to use RULESET in props.conf on indexers instead of TRANSFORMS.
@PickleRick SH, CM & LM don't have connectivity to the remote Qradar, only Indexer is configured the send the syslogs to the remote Qradar, so no point to configure syslog in SH, CM and LM right?
You got a lot of hints already. What have you compiled from them?
Thank you so much! How did you all figure it out? Life saver!
The new CSS uses a flex attribute, which breaks the old definitions. I  had used the syntax below (without the flex:unset) , but since the breaking change, the flex:unset fixes the problem. ... See more...
The new CSS uses a flex attribute, which breaks the old definitions. I  had used the syntax below (without the flex:unset) , but since the breaking change, the flex:unset fixes the problem. #header_row .dashboard-cell { flex:unset; } #header_row .dashboard-cell:nth-child(1) { width:52% !important; } #header_row .dashboard-cell:nth-child(2) { width:24% !important; } #header_row .dashboard-cell:nth-child(3) { width:24% !important; }  
I'm assuming you have the following sort of CSS #header_row .dashboard-cell:nth-child(1) { width:52% !important; } #header_row .dashboard-cell:nth-child(2) ... See more...
I'm assuming you have the following sort of CSS #header_row .dashboard-cell:nth-child(1) { width:52% !important; } #header_row .dashboard-cell:nth-child(2) { width:24% !important; } #header_row .dashboard-cell:nth-child(3) { width:24% !important; } which has stopped working with Splunk 9.4. You need to add the following for each of your row definitions  #header_row .dashboard-cell { flex:unset; } It's the flex attribute that is present in 9.4 variants that breaks things, so this fixes it.
Any working solution for 9.4.x Doesn't seem to be working thus far any of the suggestions.
Hi, I would like to resize the panels that I have in a Splunk row. So I have 3 panels and I referred to some previous posts on doing the panel width resize using CSS. I remember this used to work? B... See more...
Hi, I would like to resize the panels that I have in a Splunk row. So I have 3 panels and I referred to some previous posts on doing the panel width resize using CSS. I remember this used to work? But I can't seem to get this working on my current Splunk dashboard. Due to some script dependencies, I am not able to use Dashboard Studio hence still stuck with the classic XML dashboard. I referred to previous question on this and did exactly like what was mentioned but the panels still appear equally spaced at 33.33% each. <form version="1"> <label>Adjust Width of Panels in Dashboard</label> <fieldset submitButton="false"> <input type="time" token="tokTime" searchWhenChanged="true"> <label>Select Time</label> <default> <earliest>-24h@h</earliest> <latest>now</latest> </default> </input> </fieldset> <row> <panel depends="$alwaysHideCSS$" id="CSSPanel"> <html> <p/> <style> #CSSPanel{ width:0% !important; } #errorSinglePanel{ width:25% !important; } #errorStatsPanel{ width:30% !important; } #errorLineChartPanel{ width:45% !important; } </style> </html> </panel> <panel id="errorSinglePanel"> <title>Splunkd Errors (Single Value)</title> <single> <search> <query>index=_internal sourcetype=splunkd log_level!=INFO | timechart count</query> <earliest>$tokTime.earliest$</earliest> <latest>$tokTime.latest$</latest> <sampleRatio>1</sampleRatio> </search> <option name="colorBy">trend</option> <option name="colorMode">block</option> <option name="drilldown">none</option> <option name="numberPrecision">0</option> <option name="showSparkline">1</option> <option name="showTrendIndicator">1</option> <option name="trellis.enabled">0</option> <option name="trellis.scales.shared">1</option> <option name="trellis.size">medium</option> <option name="trendColorInterpretation">inverse</option> <option name="trendDisplayMode">absolute</option> <option name="unitPosition">after</option> <option name="useColors">1</option> <option name="useThousandSeparators">1</option> </single> </panel> <panel id="errorStatsPanel"> <title>Top 5 Error (Stats)</title> <table> <search> <query>index=_internal sourcetype=splunkd log_level!=INFO | top 5 component showperc=false</query> <earliest>$tokTime.earliest$</earliest> <latest>$tokTime.latest$</latest> <sampleRatio>1</sampleRatio> </search> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> <panel id="errorLineChartPanel"> <title>Splunkd Errors (Timechart)</title> <chart> <search> <query>index=_internal sourcetype=splunkd log_level!=INFO | timechart count</query> <earliest>$tokTime.earliest$</earliest> <latest>$tokTime.latest$</latest> <sampleRatio>1</sampleRatio> </search> <option name="charting.chart">line</option> <option name="charting.drilldown">none</option> <option name="trellis.enabled">0</option> <option name="trellis.scales.shared">1</option> <option name="trellis.size">medium</option> </chart> </panel> </row> </form>
Hello @dshpritz looks like this is "officially" documented at https://splunk.my.site.com/customer/s/article/How-To-Use-Wildcards-with-Sourcetype
Could you help me with guiding me for setting up these whole thing.