Hi,
most likely, Splunks own logs are also indexed in the _internal index. You can search it just like the other indexes. However, there is a possibility that whatever happened to your log parsing/indexing has not been logged by Splunk.
You may have to change the logging level in order to see this, e.g. from WARN to INFO or DEBUG. This is done in Manager -> System Settings -> System Logging. Unfortunately I don't know just which of the 400+ items should be changed.
On a side note, did you specify a sourcetype in your inputs.conf (or via the GUI), or did Splunk auto-assign it?
Also, a bit more information regarding the sourcetypes involved, along with some sample data would be good.
UPDATE:
As nick points out, if the new sourcetype is ...-too-small, then the file in question is too short for Splunks auto-sourcetyping to work properly.
If the new sourcetype is a "numbered" version of the original sourcetype, e.g. iis-2 or iis-3, means that Splunk thinks that it's the same format, but slightly different. This can happen for CSV log files where the header row changes. By default, I believe that Splunk expects a header row for CSV files.
I guess that this problem of yours only occurs on a per-file basis and not in the middle of a file, i.e. some of your files get indexed as the "wrong" sourcetype, but most do not.
Please provide the first three rows of
a) a correctly sourcetyped file and
b) an incorrectly sourcetyped file.
Don't forget to mask IP/usernames/hostnames as needed.
Hope this helps,
Kristian
... View more