All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Good points @PickleRick  So I guess further to my previous reply @L_Petch  - Is there a firewall between Site1 and Site2, and if so are you able to verify that this isnt causing an issue here!  D... See more...
Good points @PickleRick  So I guess further to my previous reply @L_Petch  - Is there a firewall between Site1 and Site2, and if so are you able to verify that this isnt causing an issue here!  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
To be precise, "connection reset by peer" means that the other end sent a packet with a RST flag. This might happen when: 1) The traffic is not filtered but the port is not open on the other side. ... See more...
To be precise, "connection reset by peer" means that the other end sent a packet with a RST flag. This might happen when: 1) The traffic is not filtered but the port is not open on the other side. In this case a server would simply respond with RST to the initial SYN packet and no session would be established at all. 2) The session is established (there is normal three-way handshake) but either: 2a) There is some low-level problem with the connection and the IP stack on the other end decides that it's unrecoverable and decides to close the session abruptly. 2b) There is some intermediate solution monitoring/inspecting/whatevering the traffic and breaking the connection in case it finds anything "wrong" or "suspicious". In my experience I encountered IPS solutions which would send spoofed RST both ways (to the client and server) because it was seeing unknown TLS certificates. 2c) The connection on the TCP level is working OK but there is some problem with a higher layer protocol and the protocol doesn't have signaling for that and doesn't allow for graceful shutdown and instead just closes the connection. Typical example is again TLS-oriented - when the server doesn't like client's crypto proposals (or client's certificate if you're using mTLS), it will send a TLS alert within the TLS negotiation session and then simply close the connection. It should be reflected in logs on the server's side in such case. It's often good to get a network dump (tcpdump/wireshark) from both ends of the connection to see who's sending the RST and at which moment.
Hi @Leonardo1998  I see you found the SC4SNMP before i was able to reply   Yes you can install this on a server with other Splunk components without issue as long as it meets the hardware require... See more...
Hi @Leonardo1998  I see you found the SC4SNMP before i was able to reply   Yes you can install this on a server with other Splunk components without issue as long as it meets the hardware requirements.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @Lien  I assume your user has been assigned to a relevant group in Okta that is relevant to the Splunk application?  I have seen an issue before with users who having 100+ groups where the SAML ... See more...
Hi @Lien  I assume your user has been assigned to a relevant group in Okta that is relevant to the Splunk application?  I have seen an issue before with users who having 100+ groups where the SAML response doesnt send the groups, I wonder if that could be the case here - does your user have a high count of groups in Okta? I dont know if you have seen this but it may be useful? https://splunk.my.site.com/customer/s/article/SAML-user-unable-to-login  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @tgulgund @PrewinThomas  You can set a default refresh time which will apply automatically to all data sources (unless a specific datasource is overwritten, edit the source of your dashboard and ... See more...
Hi @tgulgund @PrewinThomas  You can set a default refresh time which will apply automatically to all data sources (unless a specific datasource is overwritten, edit the source of your dashboard and find the "defaults" section, under defaults->dataSources->ds.search->options create a new "refresh" key with a value containing your intended refresh interval, such as this: { "title": "testing", "description": "", "inputs": {}, "defaults": { "dataSources": { "ds.search": { "options": { "queryParameters": { "earliest": "-24h@h", "latest": "now" }, "refresh": "60s" } } } }, "visualizations": { ... ... } }  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @L_Petch  The "Connection reset by peer" error indicates that the TCP connection was established, but then abruptly closed by the remote side (the License Manager or a network device in between).... See more...
Hi @L_Petch  The "Connection reset by peer" error indicates that the TCP connection was established, but then abruptly closed by the remote side (the License Manager or a network device in between). This often happens during the SSL/TLS handshake or if the License Manager itself encounters an issue processing the request from the Site2 indexers. Verify License Master URI Configuration: Ensure the master_uri in $SPLUNK_HOME/etc/system/local/server.conf on the affected indexers (Site2) correctly points to the License Manager using HTTPS and port 8089. [license] master_uri = https://<LM_hostname_or_IP>:8089 Replace <LM_hostname_or_IP> with the actual hostname or IP address of your License Manager. Confirm this is resolvable from the Site2 indexers.   Check SSL/TLS Certificate and Configuration: Certificate Trust: The SSL certificate used by the License Manager on port 8089 must be trusted by the Site2 indexers. If using custom certificates, ensure the CA chain is correctly installed on the indexers. Test SSL Connection: From one of the problematic Site2 indexers, use openssl to test the SSL handshake directly: openssl s_client -connect <LM_hostname_or_IP>:8089 -servername <LM_hostname_if_SNI_used> This command can help identify SSL handshake failures and certificate issues. You may wish to try passing with your certificate files: openssl s_client -connect <LM_hostname_or_IP>:8089 -servername <LM_hostname_if_SNI_used> \ -cert /path/to/your/client-cert.pem \ -key /path/to/your/client-key.pem \ -CAfile /path/to/your/ca-cert.pem​    Check yuor logs: On affected indexers (Site2): Examine splunkd.log for more detailed messages around the "failed to send rows" error. Look for messages related to SSL, TLS, or connection failures. On the License Manager (Site1): Examine splunkd.log for any errors corresponding to connection attempts from the affected indexers' IPs. Look for SSL errors, resource issues, or licensing-specific messages. Inspect Splunk Logs: On your SH: Search in _internal, something like this would be a good starting point index=_internal host= sourcetype=splunkd (log_level=ERROR OR log_level=WARN) ("SSL" OR "handshake" OR "connection from" OR "LicenseManager  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
@L_Petch  Check this https://community.splunk.com/t5/Splunk-Enterprise-Security/Unable-to-connect-to-license-master-since-certification/m-p/488156    https://community.splunk.com/t5/Security/Getti... See more...
@L_Petch  Check this https://community.splunk.com/t5/Splunk-Enterprise-Security/Unable-to-connect-to-license-master-since-certification/m-p/488156    https://community.splunk.com/t5/Security/Getting-an-issue-where-Splunk-hosts-can-t-reach-the-License/m-p/455396 
Hello,   I am getting the below error on two of my indexers. The indexers in question are on a different site (Site2) to the other two indexers & license manager in the cluster (site1). Site 1is wo... See more...
Hello,   I am getting the below error on two of my indexers. The indexers in question are on a different site (Site2) to the other two indexers & license manager in the cluster (site1). Site 1is working correctly with the same configuration as the indexers for site 2. My guess is networking but both indexers can connect to the LM on this port and there are no issues showing on the firewall between the two. All troubleshooting I have tried shows doesn't show any connectivity issues. Anyone come across this problem and have a solution? ####################################################################### HttpClientRequest [2156984 LMTrackerExecutorWorker-0] - Returning error HTTP/1.1 502 Error connecting: Connection reset by peer ERROR LMTracker [2156984 LMTrackerExecutorWorker-0] - failed to send rows, reason='Unable to connect to license manager=https://****:8089 Error connecting: Connection reset by peer' #######################################################################
I have a base search and multiple chain search. can u add refresh only to base search ?  Will that refresh other panels ?
@tgulgund  Unfortunately i dont think dashboard studio can set auto refresh for entire dashboard in a single config, auto-refresh is set per data source. You must define the refresh interval for... See more...
@tgulgund  Unfortunately i dont think dashboard studio can set auto refresh for entire dashboard in a single config, auto-refresh is set per data source. You must define the refresh interval for each relevant data source in the dataSources section of your dashboard JSON Eg: "dataSources": { "myDataSource": { "type": "ds.search", "options": { "query": "your search here", "queryParameters": { "earliest": "-48h@h", "latest": "@h" }, "refresh": "5m", "refreshType": "delay" } } } #https://docs.splunk.com/Documentation/Splunk/9.4.2/DashStudio/dsOpt Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks!
I am using splunk 9.3.2. I have visualisation panels added my dashboard with multiple queries. I use a base search with global time picker default value to 48 hours and subsequently use the chain ... See more...
I am using splunk 9.3.2. I have visualisation panels added my dashboard with multiple queries. I use a base search with global time picker default value to 48 hours and subsequently use the chain searches. I need my entire dashboard to refresh after every 5 mins. I tried "refresh":300 but it doesn't work. Not sure what am I missing here. { "visualizations": { }, "dataSources": { }, "defaults": { }, "inputs": { }, "layout": { "type": "absolute", "options": { "height": 2500, "backgroundColor": "#000000", "display": "fit-to-width", "width": 1550 }, }, "description": "", "title": "My Dashboard", "refresh": 300 }
@ww9rivers  The message File too small to check seekcrc, probably truncated. Will re-read entire file indicates that Splunk detected that /var/log/messages was truncated. Would you mind checking yo... See more...
@ww9rivers  The message File too small to check seekcrc, probably truncated. Will re-read entire file indicates that Splunk detected that /var/log/messages was truncated. Would you mind checking your logrotate setup? Splunk interprets file truncation as a reduction in size and will restart reading from the start of the file. Troubleshooting 1-Review your logrotate config or Temporarily disable log rotation for this file to verify whether it is causing the issue. 2-Add a test message to /var/log/messages and verify whether Splunk is ingesting the new entry. Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks!
I am using Okta to configure SAML for splunk. Following the step of introduction, I created a SAML group in Splunk and same group name in Okta. Made a role mapping.  https://saml-doc.okta.com/SAML_... See more...
I am using Okta to configure SAML for splunk. Following the step of introduction, I created a SAML group in Splunk and same group name in Okta. Made a role mapping.  https://saml-doc.okta.com/SAML_Docs/How-to-Configure-SAML-2.0-for-Splunk-Cloud.html When finished the setup, the logon page is through Okta but it got below error message after filled in user email and password in Okta logon page. Saml response does not contain group information. Attached the output of saml-tracer addon.  Did I miss something?      
thanks a lot
Hi @ayomotukoya  Can I check which method you are using to ingest the SNMP data? Is it using Splunk Connect for SNMP? (https://splunk.github.io/splunk-connect-for-snmp/main/) - If so this sends the ... See more...
Hi @ayomotukoya  Can I check which method you are using to ingest the SNMP data? Is it using Splunk Connect for SNMP? (https://splunk.github.io/splunk-connect-for-snmp/main/) - If so this sends the data to Splunk with HEC so would require a different approach to this problem. Or is this monitoring local files which are saved to disk from SNMP? If so, is this on a Universal Forwarder (UF) or heavy forwarder (HF)?  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @raushank26  As @richgalloway has mentioned, there are a number of apps on Splunkbase which give the capability to check certificates, such as SSL Certificate Checker .  Once the app is setup to... See more...
Hi @raushank26  As @richgalloway has mentioned, there are a number of apps on Splunkbase which give the capability to check certificates, such as SSL Certificate Checker .  Once the app is setup to pull in data about your certificates then you can use the events to generate an appropriate dashboard, highlighting things like upcoming expiries etc.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
I'm stuck deploying the RUM app provided by the Guided Onboarding (https://github.com/signalfx/microservices-demo-rum). Is anyone else having the same issue? I tried it with both quickstart method o... See more...
I'm stuck deploying the RUM app provided by the Guided Onboarding (https://github.com/signalfx/microservices-demo-rum). Is anyone else having the same issue? I tried it with both quickstart method or local build method. I'm using minikube on M1 Max Mac, and containers won't run.  Since adoptopenjdk/openjdk8:alpine-slim doesn't exist anymore, I'm using openjdk:8-jdk-alpine as the base, and I'm stuck with the gradle error.    
Hi! If the logs produced by AME can not be sent to the index, you will not get any alert data when expanding events. It would be easiest if you could open a support case in our support portal and p... See more...
Hi! If the logs produced by AME can not be sent to the index, you will not get any alert data when expanding events. It would be easiest if you could open a support case in our support portal and provide the output of the following search as a CSV export. index=_internal source=*ame* ERROR | table _time host source _raw   Regards, Simon
Thanks a lot for that details....I am also looking to create a dashboard for all certificate where we can see the expiry date. it helps management in further decision making. if possible kindly... See more...
Thanks a lot for that details....I am also looking to create a dashboard for all certificate where we can see the expiry date. it helps management in further decision making. if possible kindly let me know the steps.
There are several apps in splunkbase to help with that.  See https://splunkbase.splunk.com/apps?page=1&keyword=certificate