@gcusello  ... View more

Well on further investigation everything seems to be ok. But tcpdump shows splunk is not acknowledging the incoming syncs somehow. Iptables are ok,inputs are there, ports are listening. All looks good. No errors in logs. ... View more

SElinux is disabled, i think something has changed at firewall level. going to check that... thanks @PickleRick  i will come back and let you know how it went..     ... View more

And I am getting internal logs from hf. just not the syslog inputs that are configured there. ... View more

this started happening after the migration of these boxes which were centos7 to rhel8. but telnet seem to be working just fine. i can telnet to the indexers on 9997. ... View more

so you mean to say in splunkforwarder inputs.conf i should set renderXML = false and that should fix it or will i have to do that everywhere like on SF,HF,IX all 3 places? sorry doing this for the first time. Getting windows logs is easiest of all but the way i am trying its looking very difficult.   thanks in advance. ... View more

I think I know what went wrong chmod splunk:splunk  -R /opt/splunk you forgot to add change permissions on directory level you need to add -R to the command to add permissions recursively. hope that helps.   ... View more

Okay got it, I have changed it to splunktcp on forwarder and even I can read the logs in splunk SH but still sourcetype of logs is coming in as xmlWinEventlog instead of WinEventLog. I have the SpluK_TA_Windows on indexers as well as on HF and SF. But this I think is close to resolving the issue if i can just sort this small thing. ... View more

Splunk forwarder: outputs.conf [tcpout] defaultGroup=xxxhf   [tcpout:xxxhf] autoLBFrequency=40 server=x.x.x.x:xxxx useACK=true indexandforward=false   Heavy forwarder : inputs.conf [tcp://xxxxx] sourcetype=WinEventLog index=xxxxx disabled = 0 inputs on indexers :  [splunktcp:xxxx] ... View more

Yes ... View more

@dstuder-- For this Knowledge objects have to be shared globally and not with in the app.  As you mentioned users are not splunk admins. There are couple of approaches here: 1. You can let users play with searches and dashboards and then if some searches or dashboards. Use the search below to check for the new searches and dashboards and update deployer monthly or weekly "| rest /servicesNS/admin/-/saved/searches | table author title eai:acl.app | eval Type="SavedSearch/Report" | append [| rest /servicesNS/admin/-/data/ui/views | table author title eai:acl.app | eval Type="Dashboard"] | rename author as Owner title as Name eai:.acl.app as AppName"                      OR Let them create their own dashboards and searches in search app. Which is default for all users and what will happen is when they create a dashboard it will have permissions as private and they will be the only ones that can see that dashbaord or search. So unless the dashboard is needed for a team or all the users let them create the dashboards in "search and reporting". Still you will have to make Knowledge objects like macros etc. to be with global permissions.    hope that helps. ... View more

I will suggest to develop  the app in dev environment and then move it to /opt/splunk/etc/shcluster/apps/ once its all done/complete. do not install it deployer. Normally cluster captain will be responsible for replicating changes to apps and config on cluster. It will replicate the changes if its made on one of the nodes to other nodes but this way you will lose control on changes made to app and just in case of  an issue you will not have the actual config on the nodes. so best is to make changes to deployer and push them to cluster nodes. hope that helps. ... View more