I'm sending logs from the another ip. I can see in my tcpdump,But I can't see in my browser.How can I fix?
Last update 2 days ago. I'm using free splunk.
I wrote up some general debugging and troubleshooting steps for a input a few weeks ago that may be of use. The difference is there data had never been coming in and yours used to but I think they're similar enough that it should still be a good double-check of things. My guess is a firewall got turned on during a reboot or something.
In addition if that didn't help:
I assume these logs are syslog sent to udp 514? Have you confirmed Splunk is still listening on that port (try tunning
sudo netstat -pan | more and look at the top few lines)?
I think the other link mentions it but can you run a search over all indexes for that data in the past day or two?
index=* and see what happens - maybe something bizarre happened and it's going to the wrong index.
Lastly, are there errors either in the top menu under "Messages" or if you start searching around
Now, I search with netstat -pan | more and I saw 2 tcp 514 and I saw 1 udp 514( 0.0.0.0:514 ) in bottom.
after I tried index=* and index=_internal. But I seen my old logs.
My splunk not updating.
Please provide a sample of an event (the very latest one you have would be great) so we can check how the timestamp looks.
Also a few quick questions:
1) When these logs were coming in can you estimate about how many per second came in? Hundreds per second? Dozens per minute? (Generalities like those are fine, I'm just using this information to help narrow down exactly when it stopped).
2) Is there any way to "create" a special log entry that you can identify WITHOUT using the time? For instance if it's a firewall log can you try going to certain IP you haven't visited before to generate a log entry for, like, http://188.8.131.52/ ? (Note I have NO idea what's at that site if anything!) Maybe try a known site with a silly string it in http://amazon.com/ROYROGERSHADAHORSE/
If you can do #2, please describe what you did, then pop into Splunk and search for that special string or IP you should have created over all time and see if it shows up?
IF SO please include that event here too!
Make sure you're specifying the right index, sourcetype, etc.
This can also happen if timestamps are not interpreted correctly, resulting in events appearing to occur in the future or otherwise outside the search window. Search using All Time to see if this is the case.
I'm with the most excellent richgalloway on this - the more I think about it the more I think timestamps are messed up. It's too coincidental with the change of the year.
This is another huge advantage to running syslog-ng or rsyslog to grab syslog inputs and drop them on to disk (from where Splunk picks them up) - you have the original logs actually sitting there that you could test with to see if the timestamping mechanism in place for those still works.
One thing you could try would be to set up syslog-ng on another machine (or this one) even if it's just temporary, then a) closely compare the raw events with those that worked OK to check for changes - perhaps the sending device changed its format due to an update or something and b) run those through the "add data" wizard and see what Splunk thinks of them. I'll bet one of those two things will point out what's going on.