Splunk Search

Why am I unable to search previously indexed data?

cykuan
New Member

Hi All,

My splunk has indexed some data today. However, I am not able to search the previously indexed data anymore. For example, I am doing a search source="log.2015-05-31", it didn't show up any events, but it was able to show events on my previous report. When I change a search to source="log.2015-06-01", it does show the events, but not in my report. Thus my report can only show the result until 31-05-2015.

Is there any permission issue during search? I only made changes to admin role to inherit can_delete.

Tags (2)
0 Karma
1 Solution

woodcock
Esteemed Legend

Given this screenshot:
alt text

The problem is clear, Splunk assumes the date format is day/month/year until it realizes that this cannot be correct because the month is greater than 12 so it swaps and uses month/day/year.

You need to add this to props.conf

[YourSourcetypeHere]
TIME_FORMAT = %m/%d/%Y %H:%M:%S

Then all will be well for FUTURE events (events in the past will stay broken).

View solution in original post

cykuan
New Member

I understand, I only deleted source="log.2015-05-22", but other source likesource="log.2015-05-23"or source="log.2015-06-01" should not be deleted and able to display the event, am I right?

If I want to re-index back, what should I do? I have already tried to re-index the source="log.2015-05-22", but there is no event showing anymore for this source.

0 Karma

woodcock
Esteemed Legend

If you edit the file and swap the first 2 lines (move the top line down 1 line), it should re-index the file. The rest of what you are saying makes no sense unless you accidentally deleted more than you think you did.

0 Karma

cykuan
New Member

I know it sound weird, but it actually happen to me. For example, I put in a new log file(/home/user/cdr/chat.log.2015-06-02), when I try to do a search source="/home/user/cdr/chat.cdr.2015-06-02", there is no result at all. Any comments?

0 Karma

woodcock
Esteemed Legend

Do this search for "All Time" just to make sure the events are not timestamped "in the future" or something way off from what you expect:

... | eval lagSecs=(_indextime - _time) | stats count avg(lagSecs) BY source
0 Karma

cykuan
New Member

Hi Woodcock,

I have tried the command you provided, and it's able to show some of the index files. The result only show log.2015-05-22 until log.2015-05-31. Since my oldest log file is log.2015-05-22, hence the result display is correct. However, my latest indexed file should display log.2015-06-02, unfortunately, it doesn't show up.

0 Karma

woodcock
Esteemed Legend

Did you run it for "All Time"? This is very important (otherwise "future" events will not be found).

0 Karma

cykuan
New Member

Hi Woodcock,

Yes, after I did a "All Time", it does show all my logs with the latest log display(log.2015-06-02). But it is weird when I look on the lagSecs column, for the log from 2015-05-22 until 2015-05-31 (legSec2 is around 200000~1000000) but lagSecs for log 2015-06-01 until 2015-06-02 is very huge (12000000~10000000). Is this the reason that caused the Splunk can't show the event of 2015-06-01 onward?

0 Karma

cykuan
New Member

Yes, after I run the command for "All time", the source display all the log which start from log.2015-05-22 until log.2015-06-02. Since the log file of 2015-06-02 has been indexed, why I can't see the statistic display on my report? My report only show the statistic start from 2015-05-22 until 2015-05-31 only.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...