topic Can't index new data..? :S in Getting Data In

Can't index new data..? :S

gelica — Wed, 24 Jul 2013 09:10:15 GMT

I have some files that I want to index, I have created a new very simple sourcetype that fits my log format, and in the preview it looks fine. When I index the files I can see the event count changing in the search summary, and my source type and sources are showing up as well.

But when I run a search these events never show up! Here are some of the searches I tried, and none of my events from this source type is showing up:

sourcetype=my_source_type
*
sourcetype=*
source=path_to_one_of_the_files

My source type looks like this, and is generated by Splunk, I want to break at every timestamp(I've also tried setting SHOULD_LINEMERGE and LINE_BREAKER to break at every new line to see if that made any difference):

[my_source_type]
NO_BINARY_CHECK = 1
pulldown_type = 1

And my files look like this:

2013-03-18 03:51:28,616  INFO  [22] Deleting id=100188304
2013-03-18 03:51:28,631  INFO  [22] Deleting id=100188314
2013-03-18 03:51:28,631  INFO  [22] Deleting id=100188313
2013-03-18 08:37:51,728  INFO  [46] Checking access to 'path'

I'm using a free license for now, and after I've been trying to index these files I exceeded my limit, but this issue occured before exceeding the limit.

Does anyone know why I get this weird problem? :S

UPDATE:
I tried the splunk clean eventdata command in CLI, and then reindex some files with other custom source types that worked before, and I see the event count changing, saying that 133 events are indexed.
Then I run a search for * and Splunk says it has found 133 events, but no events is showing :S

The difference with these events compared to the ones with my new source type is that now Splunk tells me it found 133 events but I can't see them, with the new source type Splunk doesn't find any events at all of that source type...

UPDATE #2:
In case anyone wonders, I checked splunkd.log when I tried to index my files, but no errors, only a warning on two of my files(I tried to index more than two files):

WARN  LineBreakingProcessor - Truncating line because limit of 10000 has been exceeded

Re: Can't index new data..? :S

linu1988 — Wed, 24 Jul 2013 09:37:53 GMT

What's the time interval chosen for the search? as you can see the time in the log will be the index time, it will not be recent data. All other configuration is correct.

Re: Can't index new data..? :S

gelica — Wed, 24 Jul 2013 09:39:02 GMT

I'm searching over all time, so that shouldn't be a problem..

Re: Can't index new data..? :S

dariusz_kwasny — Wed, 24 Jul 2013 09:47:22 GMT

Try to add index=* at the beggining of your search. By default, Search App is searching default index only. Maybe, somehow, your events went to different index.

Re: Can't index new data..? :S

gelica — Wed, 24 Jul 2013 09:49:46 GMT

I tried your suggestion, unfortunately that wasn't the issue 😕

Re: Can't index new data..? :S

linu1988 — Wed, 24 Jul 2013 10:28:44 GMT

How did you add the log? from splunk UI? If the option is not selected to continuously collect data, then it will be monitored only once and you will not get the data anymore if it's deleted. Need to add it again.

Re: Can't index new data..? :S

gelica — Wed, 24 Jul 2013 10:48:33 GMT

I tried both uploading a log once from the web ui, and adding a monitor in the config files. I tried with different log files.
I know that the monitors doesn't index already indexedd files, but if that was the case, the event count wouldn't change in the search summary..

Re: Can't index new data..? :S

gelica — Wed, 24 Jul 2013 11:25:47 GMT

Here is a screenshot of what happens when I'm searching for * (cropped in the middle), as you can see, Splunk claims it finds 410 events but they aren't showing :S

Re: Can't index new data..? :S

jtworzydlo — Wed, 24 Jul 2013 12:37:00 GMT

Do you know to which index this sourcetype belongs? Do you have rights to view the events of this index? What role are you using?

Re: Can't index new data..? :S

gelica — Wed, 24 Jul 2013 13:00:21 GMT

All of my sourcetypes belong to the main index. I should have rights since I'm running everything locally and I am admin.

Re: Can't index new data..? :S

gelica — Wed, 24 Jul 2013 13:10:52 GMT

I figured out the reason to my problem, but I'm not sure of how I fixed it 😛

For some reason, Splunk created empty timestamps for my events, and because of that the events didn't show.