Getting Data In

How can Splunk index a file without a file extension?

mookiie2005
Communicator

I have several servers with SQL logs that are in the format:

sqlerror
sqlerror.1
sqlerror.2

I have tried all kinds of wildcard variations, but I cannot get Splunk to index these files. I can see from the internal logs that Splunk says it is watching the file, yet it never indexes any data from it. Anyone face a similar issue?

1 Solution

Nicolo_Figiani
Path Finder

Hi,
Splunk definitely doesn't care about file extension in general. You should be able to collect logs with no extension without troubles (I did it many times exactly with sql related files).

I guess your problem is somewhere else, aka the watch on path is correct but there is something else wrong going on. Take a look at this page of the Splunk wiki for troubleshooting the input monitoring process: http://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs

Hope it helps,
regards

View solution in original post

0 Karma

mookiie2005
Communicator

Wow I feel really embarrassed, turns out the logfile path provided had "Microsoft" spelled wrong! Always the little things. Thanks to all that replied with assistance!

0 Karma

lycollicott
Motivator

No worries, dude! We've all had those experiences!
Glad you found it.

0 Karma

mookiie2005
Communicator

I gave everyone who responded 25 points! Thanks again guys!

lycollicott
Motivator

Please post your inputs.conf stanza and short sample of what the sqlerror file looks like.

0 Karma

sowings
Splunk Employee
Splunk Employee

It sounds like Splunk believes it has already indexed the file (as in, it read it when it was still called just 'sqlerror'). You can query the Splunk fishbucket (the record of what's been indexed) on the local forwarder, or the TailingProcessor's File Status endpoint to see why Splunk won't index it.

The easiest way is splunk _internal call /admin/inputstatus/TailingProcessor:FileStatus and redirect that to a file. You'll find an entry in the XML like <s:key name="<YOUR_FILENAME_HERE>"; following that (indented, so that it's clearly part of the details for that entry) would be <s:key name="type">finished reading</s:key>. Other options are possible, of course, but I suspect that's the reason.

eking
Engager

I had exactly the same problem, the tailing processor led me to the answer... Splunk was ignoring the ERRORLOG file because I had recursion disabled. Thank you!

0 Karma

mookiie2005
Communicator

I should point out that teh logs in question are SQL server logs if that makes a difference.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Point of clarification: When you say "SQL Server Logs" - are we referring to plain text log files about the running of the SQL Server? Or are we talking about Transaction log files?

0 Karma

Nicolo_Figiani
Path Finder

Hi,
Splunk definitely doesn't care about file extension in general. You should be able to collect logs with no extension without troubles (I did it many times exactly with sql related files).

I guess your problem is somewhere else, aka the watch on path is correct but there is something else wrong going on. Take a look at this page of the Splunk wiki for troubleshooting the input monitoring process: http://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs

Hope it helps,
regards

0 Karma

jkat54
SplunkTrust
SplunkTrust

Does the Splunk user have permission to read the files? Maybe if you check the windows security and application logs, you will find more details such as an access denied, etc.

0 Karma

mookiie2005
Communicator

It should, it is running under my user and I am able to open the files in question.

0 Karma

lycollicott
Motivator

I would think that if the internal logs indicate that it is being watched then permissions "should" be correct - I really hate the word "should" - but Windows permissions could be very funky.

You need some special privileges to run a Windows service as your own userid (and not really recommended anyway), so perhaps those are not all set up. Can you run the service a Local System (or an MSA) to test for different behaviour?

0 Karma

jkat54
SplunkTrust
SplunkTrust

I hate to ask again, but did you check the windows event logs? I know it's cliche but they've seriously saved my butt on more than one occasion only after I said... hmphh there wont be anything in THOSE logs about THIS issue.

0 Karma

mookiie2005
Communicator

Nothing in the windows logs.

0 Karma

somesoni2
Revered Legend

Can you try something like this (assuming files with extension .1/.2 are rollover files and doesn't have to be monitored)

[monitor://D:\Your Path\To Log File\Till TheFolder\Containing The\SQL Logs]
whitelist = sqlerror$
index = yourindex
sourcetype = yoursourcetype
0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...