Hello Everyone,  looking for a little guidance on our Splunk deployment for a system.  Currently, we have a few different sites that span across the US with Universal forwarders deployed to all of the systems and reporting back to one main Splunk instance individually.   Id like to see about splitting the Splunk system up into two separate parts to improve integrity and reduce latency, but have never delt with deploying a heavy forwarder in an instance like this.  My thought is to have all of the western universal forwarders sending their events to a dedicated Western Heavy forwarder, and have all of the eastern universal forwards send their data to the eastern heavy forwarder, and have both of the heavy forwarders send their data to our main Splunk instance.  (crude visio below) Any guidance is greatly appreciated!    ... View more

Hello everyone! I have a bit of a challenge I have run into. Long story short, we are required to show our customer that we are completing audits each time period. I saw another sites dashboard recently, and am trying to replicate it, only issue is they didn't do it themselves so they don't know how its done.   I am going to explain it as best as I can. Basically, when they open a dashboard, a table populates with a comment, recorded by, and recorded on field.  The have the option to add a record to it, a comment box pops up, and they add whatever the comment is, and then click save.  The table then displays their comment, what their user name is, and a time stamp.  It saves week after week, and has a search records bar to find specific events as well.   Does anyone have any idea how to create something like this?     ... View more

Hello fellow Splunkers. I need a little help with an issue I am having with one of my dashboards.  Im sure its a simple fix but am having a tough time figuring out the correct way to do it.  A little background, we created a dashboard to check the status of connected forwarders for our auditing purposes. With the new infrastructure we have a VDI setup that spins up a new hostname when a new user logs in.  This results in our dashboard having a bunch of different forwarders showing as offline. I want to add in to the search to get everything that hasnt reported into the dashboard in the past 5 days to not be pulled into the chart. the search string is below, the time I am would like to filter off of is the last_phone_home. Thanks for any help you can provide!   | inputlookup hosts.csv | table * | join max=0 [| rest splunk_server=local /services/deployment/server/clients | fields - applications.* serverClasses.* eai* splunk_server author id title | collect index=summary addtime=true marker="dataset=deployment_server_clients" | eval diff=now()-lastPhoneHomeTime | eval status=if(diff>120, "Connection Failed", "Connection Successful") | rename hostname as host] | rename utsname as platform | eval last_phone_home=strftime(lastPhoneHomeTime, "%F - %T") | eval hostname=lower(hostname) | eval last_hourly_check=strftime(last_hourly_check, "%F - %T") | table host platform ip splunkVersion last_phone_home status | sort status | dedup host ... View more

Hey all, I am stumped and need some help, I am configuring a system stack with Splunk enterprise on it. It is relatively small, only 11 systems. I have the web interface installed with a license, forwarders and apps pushed out to systems, and port listeners open on 9997 for the forwarder to talk back to in the forwarding and receiving tab. I know there is some communication because I can see all of the systems in the forwarder management tab, however I cannot get any data into our dashboards. The only system data I can find and search is that of the server where the main instance is located. I have indexes made for all the different types of data, (linux_audit, Win_security, ETC). No data from the forwarders themselves is coming through. My only other thought is a firewall issue and that the correct port isn't open but beyond that I had no idea.  I am sorry for the ignorance, this is my first real time setting this up and the Splunk documentation isn't super helpful for troubleshooting. Thanks in advance!  ... View more