topic Re: Data being indexed but unable to search in Getting Data In

Data being indexed but unable to search

rmcdougal — Thu, 06 Dec 2012 20:42:33 GMT

Ok so here is the issue, I have installed a forwarder on my Snort box to forward over the data to Splunk. It appears to be sending the data over and it appears to be getting indexed but, I am not able to search the information.

This is my search summary page

Notice the last update time.

Now I am going to click on the source type and search for the events.

Notice that the latest event showing up has a timestamp of 12/6/2012 at 12:56 AM. This contradicts the search summary page.

One last thing, from the deployment monitor, this is the status of the forwarder on my snort box.

Re: Data being indexed but unable to search

lguinn2 — Mon, 10 Dec 2012 01:49:41 GMT

What index does the forwarder specify for the snort data in inputs.conf?

Re: Data being indexed but unable to search

miteshvohra — Mon, 10 Dec 2012 04:01:59 GMT

And, why 'Heavy Forwarder' and not 'Universal'?

Re: Data being indexed but unable to search

Ayn — Mon, 10 Dec 2012 10:22:55 GMT

Also, if I recall correctly the time on the summary page is when the last event was INDEXED, not necessarily when it was actually generated.

Re: Data being indexed but unable to search

rmcdougal — Mon, 10 Dec 2012 14:12:15 GMT

It is configured to forward to the "security" index. It is using a heavy forwarder because that is what my system admin felt most comfortable installing.

Re: Data being indexed but unable to search

Drainy — Mon, 10 Dec 2012 15:35:30 GMT

The summary page only displays data in the main index by default, so it won't register detail on other indexes.

Re: Data being indexed but unable to search

lguinn2 — Mon, 10 Dec 2012 22:20:40 GMT

My guess is that your Splunk admin did NOT set up the security index to be searched by default. That setting is under Manager -> Access Control -> Roles. For each role, the admin can determine which indexes are visible and which indexes are searched by default.

If the security index is NOT one of your default indexes, you may be able to search it explicitly:

index=security sourcetype=snort

If that doesn't work, perhaps the Splunk admin has not given you access to the security index at all.

Re: Data being indexed but unable to search

rmcdougal — Tue, 11 Dec 2012 13:54:51 GMT

I found the solution and it wasn't very intuitive. The timestamp was not being indexed properly by splunk so the events were getting indexed but there was an invalid timestamp associated with them preventing them from showing when searching for them. (I still haven't been able to find them).

After changing the TIME_FORMAT in props.conf the events started to display.