Splunk Search

Cannot find log messages for -1d@d but -7d@d works

baoctac
New Member

I have a Splunk alert that has been sending false emails. The alert is sent when a string is absent from the application's log. The search itself is actually not finding the log message, which I assume is the reason for the log not being triggered.
This search in Splunk UI: "finished importing data" earliest=-1d@d
Results: no messages found.

But when I search back 7 days, the expected log message appears.
Ex: "finished importing data" earliest=7d@d
Results: messages are returned as expected

Furthermore, I am able to see other messages from the application logs from the last 24 hours in Splunk, but the specific text "finished importing data" only appears when I search for 7 days or greater.

Any ideas why 7-day search would return results but not a 1-day search? Your help is greatly appreciated.

Currently running Splunk version 7.0.1.

Tags (1)
0 Karma

baoctac
New Member

hrm...this is interesting. Yesterday's log is somehow added to an exception from earlier this week. Let's close this one for now. I'll investigate further on my end.

Thanks all.

0 Karma

Sukisen1981
Champion

hmm intriguing indeed, from the status "Finished Importing data" I can probably guess that this is an indication of some sort of batch run. Check with the developer of the batch job, it has to be an issue with how he is logging the current day's (last 24 hours) log.
Meanwhile please accept my answer if you found it useful

0 Karma

baoctac
New Member

"Finished Importing data" | earliest=-d@d latest=@d

With the pipe, I get an error: Search Factory: Unknown search command 'earliest'.

"Finished Importing data" earliest=-d@d latest=@d

Without the pipe, I get: No results found. Try expanding the time range.

0 Karma

Sukisen1981
Champion

'Furthermore, I am able to see other messages from the application logs from the last 24 hours in Splunk, but the specific text "finished importing data" only appears when I search for 7 days or greater.'
Question is - when is the latest occurrence of this event when you search using

earliest=-7d@d?
If there are no "Finished Importing data" event in the last 24 hours searching for the same events will not yield any results

0 Karma

Sukisen1981
Champion

what happens if you try this?

<your search>|earliest=-d@d latest=@d
0 Karma

baoctac
New Member

I don't follow. Are you saying that most of my application log(at least >90% of it) is present in my data, and that single log message is not present in data? I'm just trying to understand how this would happen, and understand how to prevent it.

If my Splunk query is wrong, how would one search and/or create an alert that checks for existence of a log message for the last 24 hours?

Thanks!

0 Karma

493669
Super Champion

On what criteria you can say application log is present in your data?
if "finished importing data" this is contained in your data then you are considering as application log is present?

0 Karma

baoctac
New Member

I assumed that 'present in my data' means that I'm able to search and find those logs in Splunk. But perhaps it's better for me to ask for your criteria as well 🙂

Is that assumption correct? or did you mean something else?

0 Karma

493669
Super Champion

your assumption is correct i.e. if "finished importing data" present means you can search and find logs

0 Karma

baoctac
New Member

Thanks. I just posted this message below:
Yesterday's log is somehow added to an exception from earlier this week. Let's close this one for now. I'll investigate further on my end.

0 Karma

493669
Super Champion

when you search : index=<indexname> "finished importing data" earliest=-1d@d
Results: no messages found. it means "finished importing data" string is not present in your data from yesterday.
and when you search : index=<indexname>"finished importing data" earliest=-7d@d
Results: messages are returned . it means "finished importing data" string is present in data from last 7 days till day before yesterday

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...