Getting Data In

I can't see my logs in splunk(linux)

omeryirmibes
New Member

I'm sending logs from the another ip. I can see in my tcpdump,But I can't see in my browser.How can I fix?

Last update 2 days ago. I'm using free splunk.

Thanks.

Tags (1)
0 Karma

Richfez
SplunkTrust
SplunkTrust

Hi, omeryirmibes.

I wrote up some general debugging and troubleshooting steps for a input a few weeks ago that may be of use. The difference is there data had never been coming in and yours used to but I think they're similar enough that it should still be a good double-check of things. My guess is a firewall got turned on during a reboot or something.

In addition if that didn't help:

I assume these logs are syslog sent to udp 514? Have you confirmed Splunk is still listening on that port (try tunning sudo netstat -pan | more and look at the top few lines)?

I think the other link mentions it but can you run a search over all indexes for that data in the past day or two? index=* and see what happens - maybe something bizarre happened and it's going to the wrong index.

Lastly, are there errors either in the top menu under "Messages" or if you start searching around index=_internal?

0 Karma

omeryirmibes
New Member

Now, I search with netstat -pan | more and I saw 2 tcp 514 and I saw 1 udp 514( 0.0.0.0:514 ) in bottom.

after I tried index=* and index=_internal. But I seen my old logs.

My splunk not updating.

Thanks.

0 Karma

Richfez
SplunkTrust
SplunkTrust

Please provide a sample of an event (the very latest one you have would be great) so we can check how the timestamp looks.

Also a few quick questions:

1) When these logs were coming in can you estimate about how many per second came in? Hundreds per second? Dozens per minute? (Generalities like those are fine, I'm just using this information to help narrow down exactly when it stopped).

2) Is there any way to "create" a special log entry that you can identify WITHOUT using the time? For instance if it's a firewall log can you try going to certain IP you haven't visited before to generate a log entry for, like, http://199.2.2.4/ ? (Note I have NO idea what's at that site if anything!) Maybe try a known site with a silly string it in http://amazon.com/ROYROGERSHADAHORSE/

If you can do #2, please describe what you did, then pop into Splunk and search for that special string or IP you should have created over all time and see if it shows up?

IF SO please include that event here too!

0 Karma

nmohammed
Contributor

You're seeing the old data from that IP ? what kind of data is it , syslog and Is it sending data continuously ?

0 Karma

omeryirmibes
New Member

Yes, I can see my old data. Sending data continuously

0 Karma

omeryirmibes
New Member

my prob;
splunk is not updating.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Make sure you're specifying the right index, sourcetype, etc.
This can also happen if timestamps are not interpreted correctly, resulting in events appearing to occur in the future or otherwise outside the search window. Search using All Time to see if this is the case.

---
If this reply helps you, Karma would be appreciated.
0 Karma

omeryirmibes
New Member

I'm searching with date_year ="2017" but the result is the same.

0 Karma

Richfez
SplunkTrust
SplunkTrust

I'm with the most excellent richgalloway on this - the more I think about it the more I think timestamps are messed up. It's too coincidental with the change of the year.

This is another huge advantage to running syslog-ng or rsyslog to grab syslog inputs and drop them on to disk (from where Splunk picks them up) - you have the original logs actually sitting there that you could test with to see if the timestamping mechanism in place for those still works.

One thing you could try would be to set up syslog-ng on another machine (or this one) even if it's just temporary, then a) closely compare the raw events with those that worked OK to check for changes - perhaps the sending device changed its format due to an update or something and b) run those through the "add data" wizard and see what Splunk thinks of them. I'll bet one of those two things will point out what's going on.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...