All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

@christophenet  I just cloned a sample dashboard that uses Event Timeline Viz, but I see the following error in Dashboard Studio    
@kiran_panchavat@jondukehds   I've just tried a few commands: - “shcluster status --verbose” doesn't return any useful additional info - No error in splunkd.log - splunk resync shcluster-r... See more...
@kiran_panchavat@jondukehds   I've just tried a few commands: - “shcluster status --verbose” doesn't return any useful additional info - No error in splunkd.log - splunk resync shcluster-replicated-config” changes nothing - splunk restart” changes nothing - Switching from dynamic to static captain and back to dynamic + bootstrap, changes nothing apart from changing the captain.   Each search head is given the same ID in the cluster as the captain, the shcluster-status or the shclustering stanza.   It's really only in the GUI that I see another bundle. I don't get it...   Not having done the integration, I don't know if scripts interfere with the built-in Splunk startup but I don't think so, given the rest of the project.
it shows this.. but everything is running    
yes we upgraded to 9.3 on windows server 2019
Hey there & Daniel  Simple quick question : Does Event Timeline Viz works with Dashboard Studio too ? Thanks & regards, Christophe
@Treize  On each SHC member, run: /opt/splunk/bin/splunk show shcluster-status -verbose The -verbose flag provides additional details, such as replication status, member health, and any pending ac... See more...
@Treize  On each SHC member, run: /opt/splunk/bin/splunk show shcluster-status -verbose The -verbose flag provides additional details, such as replication status, member health, and any pending actions. Look for discrepancies (e.g., a member marked as “Pending” or “Out of Sync”). Check the splunkd.log on all four SHC members for errors related to clustering. Focus on: SHCMaster, SHCMember, or ConfReplication components. Errors like “failed to proxy call” or “replication failure.” The GUI might be stuck due to a caching issue. Restart Splunk Web (not the full splunkd process) on all members: /opt/splunk/bin/splunk restart If the GUI persists in showing incorrect data, resynchronize the replicated configuration across the cluster: On the captain, run: /opt/splunk/bin/splunk resync shcluster-replicated-config  https://docs.splunk.com/Documentation/Splunk/9.2.1/DistSearch/HowconfrepoworksinSHC . This command pulls the latest configuration from the captain and pushes it to all members, which might realign the GUI’s view. Rolling Restart (If Needed): If the above steps don’t resolve the issue, perform a controlled rolling restart of the cluster: /opt/splunk/bin/splunk rolling-restart shcluster-members This ensures all members restart cleanly and re-register with the captain, potentially fixing any GUI misalignment. Monitor the GUI and CLI status post-restart. Splunk 7.3.5 is several years old, and while it’s stable for many environments, there have been reported bugs in SHC management and GUI rendering in earlier 7.x versions. Check Splunk’s Known Issues documentation  https://docs.splunk.com/Documentation/Splunk/7.3.5/ReleaseNotes/KnownIssues 
Hi @Treize  It is generally advised that SHC should comprise of an odd number of nodes, this is to prevent a split-brain situation - however because you have 3 which have maintained their cluster I ... See more...
Hi @Treize  It is generally advised that SHC should comprise of an odd number of nodes, this is to prevent a split-brain situation - however because you have 3 which have maintained their cluster I am not sure if this is a split-brain situation. You may need to stop the single SH, clean and re-join it to the cluster. Check out https://community.splunk.com/t5/Deployment-Architecture/SHC-New-Member-reverts-to-down-after-restart/m-p/505409 which has some similar conversation covering this.  When you run show shcluster-status on the single SH, does it show the same cluster details as the other 3? Do you have any custom startup scripts running on the host that might interfere with the built-in Splunk startup? e.g. running any commands to join clusters, setup captains etc. Please let me know how you get on and consider adding karma to this or any other answer if it has helped. Regards Will  
Hi @ccWildcard  I have heard of this issue before for others using Splunk 8.x on Windows - please could you confirm which version of Splunk you are running? Also, are there any unusual file permissi... See more...
Hi @ccWildcard  I have heard of this issue before for others using Splunk 8.x on Windows - please could you confirm which version of Splunk you are running? Also, are there any unusual file permissions on that path? Please let me know how you get on and consider adding karma to this or any other answer if it has helped. Regards Will
Hello, I have a problem that I can't solve. I have a shcluster with 4 members (including the Captain) and splunk version 7.3.5.   We are in a multisite configuration. We wanted to do a test to pu... See more...
Hello, I have a problem that I can't solve. I have a shcluster with 4 members (including the Captain) and splunk version 7.3.5.   We are in a multisite configuration. We wanted to do a test to put a Search Head in stand-alone mode and simulate a power cut with the 3 others. Everything worked, then we returned to normal. ALL CLEAR.   But recently we realized that we had a problem (bug?). Our 4 SHC members are in the same cluster, checked on the servers directly in CLI. But on the GUI we have two different SHcluster: the first with 3 members, the second with only 1.   show shcluster-status shows the cluster, its 4 members and its ID (starting with EDF6) The [shclustering] stanza in the server.conf file for the 4 search heads has the ID EDF6[...].   I remind you that despite this, everything works normally. We've tried a lot of solutions with no results. Is this a bug or do you have any ideas?   Attached are some screenshots, to make things easier. Thank you very much
Hi @okumar1 , this means that there's something in the middle between UD and IDX that block the connection. probably an intermediate firewall or a local firewall on the IDX. Ciao. Giuseppe
hi @yuanliu For me it is straight forward ). If you have multiselect filters in dashboard studio, please open it and check the feature which i shared in screenshot. Then try to do the same in multi... See more...
hi @yuanliu For me it is straight forward ). If you have multiselect filters in dashboard studio, please open it and check the feature which i shared in screenshot. Then try to do the same in multiselect filter in classic dashboard :-). 
hi @gcusello , here is the telnet test telnet: connect to address 13.233.165.44: Connection refused and splund.log please suggest.
here is the output [root@ip-172-31-13-139 log]# nc -vz -w1 13.233.165.44 9997 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connection refused.   no  my outputs.conf is below: [tcpout] de... See more...
here is the output [root@ip-172-31-13-139 log]# nc -vz -w1 13.233.165.44 9997 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connection refused.   no  my outputs.conf is below: [tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = 13.233.165.44:9997 and when i debug splunkd.log The TCP output processor has paused the data flow. Forwarding to host_dest=13.233.165.44 inside output group default-autolb-group from host_src=ip-172-31-13-139.ap-south-1.compute.internal has been blocked for blocked_seconds=100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data please suggest
Hi @nithys , what are the results of your search? what is your issue? You shared a search that seems to be correct, does it give you results? Ciao. Giuseppe
Hi @okumar1 , what about telnet test? Ciao. Giuseppe
Hi @okumar1  Were you able to check the netcat/nc commands?  Are there any other logs around mentioning tcpOutput in your UF? Please let me know how you get on and consider adding karma to this or... See more...
Hi @okumar1  Were you able to check the netcat/nc commands?  Are there any other logs around mentioning tcpOutput in your UF? Please let me know how you get on and consider adding karma to this or any other answer if it has helped. Regards Will
I would like to understand how to perform a lookup on multiple values separated by a delimiter. As I explained in my initial comment, there is no such match. (I mean, you CAN perform complicat... See more...
I would like to understand how to perform a lookup on multiple values separated by a delimiter. As I explained in my initial comment, there is no such match. (I mean, you CAN perform complicated maneuvers with such a table to accomplish matching.  And I have.  But don't call that a lookup.)  Please explain exactly how you are going to use this lookup (using data illustration), AND what exact "problems" does mvexpand cause (again, using data illustration).
But I need select All Matches: It's about time you explain exactly this sentence means.  Make sure you illustrate data, desired output based on illustrated data, and explain the use case without S... See more...
But I need select All Matches: It's about time you explain exactly this sentence means.  Make sure you illustrate data, desired output based on illustrated data, and explain the use case without SPL.
Like @bowesmana says, if your data is as you illustrated, and if your search is exactly like you have shown, the search should give you the correct results.  So, my speculation is that in your real s... See more...
Like @bowesmana says, if your data is as you illustrated, and if your search is exactly like you have shown, the search should give you the correct results.  So, my speculation is that in your real search, spelling of myfiled in spath and in stats are different.  For example, maybe your actual search was spelled like |spath output=myfiled path=audit.addBy | stats count by myfield By the way, there should be no need for spath as @bowesmana says.  This search should give you exactly the same result kubernetes_cluster="abc*" index="aaaa" sourcetype = "kubernetes_logs" source = *pub-sub* | stats count by audit.addBy  
Use  | outputlookup mycsv.csv output_format=splunk_mv_csv That keeps the lookup file with proper MV fields.