Splunk Search

Why is newly configured ingestion not showing older logs?

anandhalagarasa
Path Finder

My search is that I have to log in the client machine, which needs to be ingested into Splunk Cloud- so I have deployed the inputs from deployment master and the app has successfully reached the client machine.

Actually the log which I have ingested is last updated on 8th Sep 2019, but I have done the configurations on today (11th Sep 2019) so when I search the data in Splunk Cloud I wasn't able to see the logs into Splunk Cloud.

So my search is so that the old data (8th Sep 2019) logs will be also ingested into Splunk Cloud?
If yes, then in my case why it's not getting ingested?

If no, then this is how the mechanism of Splunk works.

Kindly confirm the same.

Input Stanza:

[monitor:///ijk/lmn/otp.log]
sourcetype = xyz
index = abc
disabled = 0

So kindly help on the same.

0 Karma

thesplunkmonkey
Path Finder

Depending on how you set up your inputs, and how much historical data is on the source system, what you are probably seeing is the older data being indexed and splunk gradually moving it's way forward in time through the files.

If you continue to search periodically, you will probably see new data coming in as time goes on as splunk reads those older files and gets it indexed in. Unless you have a truly enormous amount of historical data, or you have issues with bandwidth from the UF, you should see it start to catch up.

You could also try running a real-time search to watch as new events are coming in, if you have access to run that type of search. I wouldn't recommend running it long term or across a large amount of data, but it can be helpful for watching these types of ingestion situations to ensure that you're still getting data in, and that it hasn't stalled.

If you want to have your input stop looking at the older files, you can set a limit on the input. For example, if you want it to only go back a maximum of 5 hours, you can do something like this.

inputs.conf
[monitor:///path/to/files]
ignoreOlderThan = 5h

https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Inputsconf

0 Karma

anandhalagarasa
Path Finder

@thesplunkmonkey ,

Thanks for your response.

I just want to know whether can i able to index the old data logs into Splunk. For example the log file has been lastly updated two days before and I have done the configuration today so will it be getting ingested into Splunk?

My requirement is that I want to index those log data into Splunk as well so based on that kindly help on the input stanza.

0 Karma

thesplunkmonkey
Path Finder

Yes. If you want those older files, then you would leave off the ignoreOlderThan attribute that I mentioned. It will read in and index all files, old and new, that match your monitor stanza path, generally reading in the oldest files first and moving forward through the newer files.

0 Karma

anandhalagarasa
Path Finder

@thesplunkmonkey,

my question is i use the below mentioned stanza but the old data is not getting ingested into Splunk Cloud.

[monitor:///ijk/lmn/otp.log]
sourcetype = xyz
index = abc
disabled = 0

Here i didn't use anything like ignoreOlderThan command itself but still the last updated log is not getting ingested into Splunk Cloud.

So should i need to add any command in the stanza so that it will pull the old data as well.

0 Karma

thesplunkmonkey
Path Finder

As long as the path you are putting in your monitor stanza matches the path and pattern that would include those older files, it should read them in.

So, for example, if you have files in /ijk/lmn/ called opt.log, opt1.log, opt2.log qrs.log, qrs1.log, etc, then your monitor stanza should be something like...

[monitor:///ijk/lmn/]

...in order to pick up all of the files inside of the lmn directory. In that example of files I list above, and the example of the monitor stanza you stated in your last comment, you would be missing all of the older files because you have it watching only a specific filename (/ijk/lmn/opt.log).

If you have files that are perhaps log rotated to a different filename, like adding a .1, .2, .3, etc to the end, or have a dated file that rotates on a regular basis, then you'll need to account for those file name differences.

something like

[monitor:///ijk/lmn/opt*.log]

may be appropriate.

The default for ignoreOlderThan is to be unset, meaning there is no threshold and no files are ignored for modification time reasons. So otherwise, your stanza settings should be fine.

You may also need to consider if the older files may appear to be identical to the new files as far as their content goes. By default, the input only performs CRC checks against the first 256 bytes of a file. This behavior prevents the input from indexing the same file twice, even though you might have renamed it, such as log rotation, for example. So if your log files happen to have a long 'header' in them that is the same for every file, they may be seen as 'identical'. in that case you may need to set a crcSalt attribute on your stanza to have it consider the files different if the filenames are different. in that case you'd use

crcSalt = <source>     (literally the string '<source>').

Just be careful if your filenames start out with otp.log and when rolled end up becoming a file called opt1.log, for example, because then your files will be indexed a second time when they role.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi anandhalagarasan,
at first check again your logs, maybe your Indexer is still indexing!
Then could you share an example of your logs to see the timstamp format? maybe there's an error in timestamp recognition.
Bye.
Giuseppe

0 Karma

anandhalagarasa
Path Finder

Kindly help on the request.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!