Hmmm.....back to the drawing board... How bout this... Rebuild your inputs.conf to accept from any host, rather than specifying the sender. Let's see if wide open UDP changes anything.... ... View more

what is the output of this search, obviously changing your values accrodingly: index=_internal source=*metrics.log host="<yourindexers>" group=per_sourcetype_thruput series="<yourSourcetype>" Also if you go to settings > indexes .....is your main index growing??? ... View more

yep, behaving as per your outputs.conf. all is looking well there my friend. So, lets take a step back. If you do directly to one of your indexers and search for these logs....anything??? ... View more

NP at all. That's why I asked. If you are in a smallish environment with lots of standalone, you could hack something together, but in a growing, traditionally architected Splunk deployment, the most scalable solution is to manage the searchheads and indexers with the cluster master and the deployer, and to have an app ready for when you need to up the ante on your logging. ... View more

Ah, ok. I recommend starting here: http://docs.splunk.com/Documentation/Splunk/6.5.0/RESTTUT/RESTbasicexamples Generally the API is used by other systems to run searches against Splunk to bring the data back for further manipulation or visualization, or to actually interact and manage Splunk. The dev guide should have all you need to get started. ... View more

When you deploy the 3 apps, you are likely overriding the 4th app's outputs.conf https://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Wheretofindtheconfigurationfiles Can you share the outputs.conf of the 3 apps vs the 4th app so we can help you reach the config you are looking for? ... View more

Awesome, probably just old logs.. Ok, so everything is looking good. Are you certain that your checkpoint is sending events? How often? ... View more

"Connection to host=10.37.129.12:9997 failed" "Connect to 10.37.129.12:9997 failed. Connection refused" "Connection to host=10.37.129.13:9997 failed" "Connect to 10.37.129.13:9997 failed. Connection refused" "Applying quarantine to ip=10.37.129.13 port=9997 _numberOfFailures=2" "Applying quarantine to ip=10.37.129.12 port=9997 _numberOfFailures=2" Can you telnet to your indexers from the forwarders on 9997? whats the timestamp on them things? splunker@n00b-splkufwd-01:/opt/splunkforwarder/var/log/splunk$ cat splunkd.log | grep TcpOutputProc ... View more

looks like you. need ntp! time is a VERY important aspect when working with Splunk. I suggest getting NTP set up on all your nodes! The rest of the config looks good at first glance, will look again closely and let you know if I find anything While we are at it best practicing, you'll probably want to file these away for a rainy day 😉 http://www.georgestarcher.com/splunk-ulimits-and-you/ https://answers.splunk.com/answers/188875/how-do-i-disable-transparent-huge-pages-thp-and-co.html ... View more

you set up looks fine to me...I am pretty sure these messages can be disregarded as they are simply verbose debug logs. Your timestamping is working correctly, right? ... View more

Hey! Ok, so you logged in as the other user and they can see the lookup file? Thats good, how about the events?? Does the other user see anything when running source="dhcpd.log" Host=H* DHCPACK lease-duration OR RENEW Slowly build up the search, pipe by pipe, with the other account till you find the problem... You can share a link to http://pastebin.com/ Get a free account and paste it there. ... View more

You likely arent...what does your inputs.conf look like for this heavy forwarder? ... View more

looks like ur all good! This is just a debug message telling you how splunk is setting the timestamp. Are you running debug log level? ... View more

Hi chaitu_kranthi, Are you the one that would be configuring the Splunk side? I would recommend starting here: https://docs.splunk.com/Documentation/Splunk/6.5.0/Data/SendSNMPeventstoSplunk ... View more

Splunk cloud instances are built in single regions only. They do not span multiple aws regions, unless you buy multiple instances. I recently worked with a client with the same requirements and they chose to host their indexers and searchheads in Dublin. I'll see if I can rustle up documentation to that effect, but I speak from experience. ... View more

The number matters because it helps to dictate the solution you build. Answering this type of question without understanding your environment is careless. If you are talking about a few standalone SH/indexers...then adding another process to do a job the existing process can do seems overkill. If you are talking a cluster of 40 indexers then you should simply be using the cluster master. SHC should use deployer... Plus... you can change input monitors without re-starting splunk at all... https://answers.splunk.com/answers/5838/can-inputs-conf-be-reloaded-without-restarting-splunkd.html Regardless, well built splunk architectures have many built in redundancies to allow admins to restart processes when needed...I guess it all depends on what your priorities are and how its been built... that can be the hardest part of being a splunk admin....everything is possible...most times its not CAN you, its SHOULD you. I say test it out and see of it provides the streamlining you are looking for, cause while your theory is based on this particular scenario, it adds much more complexity, and possibly performance concerns, to administrating your environment. ... View more

It is completely possible, but I would not recommend it, in general. Tell us more about your Searcheads and Indexers. How many? How are they set up (standalone, clustered)? ... View more

Hey Andresito123! Nice Lab setup! The experience of working through these items will serve you well in the exam and beyond! Your config looks good, and the fact that _internal logs are making it to the indexers means your forwarding/receiving setup looks good! Searching index=_internal and making sure all your hosts are present is a great place to start all your forwarder troubleshooting. How about searching index=_internal host=<yourforwarder> error OR warn anything interesting? Splunk indexers have syslog as a default props, so you should be good there. Now, Lets work from the forwarder and see what we can discover: I noticed that your input lacks an index. Are you just trying to send to default index? out of the box, that would be the 'main' index. if you search index=main over all time....you definitely aren't receiving? The forwarder has some really great debug commands. Try these from /opt/splunkforwarder/bin: ./splunk list foraward-server this will confirm your active forwards (you already did this by checking _internal, but figured I'd share anyhow as it is very useful) ./splunk list inputstatus You should see your UDP input there...how does it look? Can I assume you are running the forwarder as root? You would need root to listen on ports lower than 1024. How about the output of netstat -tulpn on your forwarder ( I assume you are on *nix)? Is splunkd listening on 514? When you pushed the app, did you configure it to restart the forwarder? In the Deployment Server, it should show, "after installation - Enable app, restart splunkd". Have you tried restarting the forwarder already manually? If you check all these and still not seeing anything, lemme know and we'll move along... ... View more

Hi Yanivdutt! I would strongly recommend you run one of your sample CSV through the Add Data wizard on one of your searchheads. http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Howdoyouwanttoadddata It will allow you to test these settings and allow you to preview how your data looks once they have been applied! http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Setsourcetype I think you may have a couple conflicting settings.... Is the sample you pasted exactly what the file looks like? Can you share one with the preamble? I can run it through the add data wiz and show you what I come up with. ... View more

you need to be under /opt/splunkforwarder/bin if it is a universal forwarder. also watch the typos! ./splunk cmd btool outputs list --debug ... View more

I would suggest proving your assumptions about access, just to be sure. 11/5/16 2:00:00.000 PM – 11/5/16 3:00:36.929 PM did not return any data. Possible solutions are to: relax the primary search criteria widen the time range of the search check that the default search indexes for your account include the desired indexes This search is an instance of the saved search: Test+Assetscanning+TH. Just like the job is telling you...relax the search criteria for that user...can they see output after running this search??? : source="dhcpd.log" Host=H* DHCPACK lease-duration OR RENEW *Make sure those examples above are a typo....your host field needs to be host=H* not host=H (im sure they are...probably just answers removing the asteriks) INFO: Assuming implicit lookup table with filename 'qualys_hostlist.csv' means the lookup table isn't defined explicitly...might be a good idea to set the lookup table as a shared knowledge object. have the user try this search: | intputlookup qualys_hostlist.csv what do they get??? ... View more

can you login as the user who is running the search and see of they can see the index with the dhcp events...also can they run the lookup u are using? job inspector output probably has owner info...just share on pastebin... one working one failing ... View more