Getting Data In

Log Monitor Stops from 30-09-2020 to 01-10-2020

bgstein
Path Finder

I have two monitored logs for which no new events are being collected.  The Splunk logs don't show any (new) issues or errors - although I did spend quite some time trying to understand if the encoding is the problem.  Seing many:  "Using charset UTF-16LE, as the monitor is believed over the raw text which may be UTF-8". But these appear for logs that are working and logs that are not.

More digging shows the logs stopped being collected *exactly* after midnight.  The first two were collected and the second two were not.  

[I 06/00000040/T06BC/P0AA0] 30-09-20 23:47:13 - Client Rules: rule 'Internal Access Policy' matched. 
[I 06/0000000A/T06BC/P0AA0] 30-09-20 23:47:13 -Server RAS-SH3:3389 is available
[I 0E/00000000/T1920/P0AA0] 01-10-20 00:15:28 - Session login for userwas successful.
[I 06/00000040/T1920/P0AA0] 01-10-20 00:15:28 - Client Rules: rule 'External Copy and Paste and Printer 

I'm guessing I'll need to use the props.conf to set the TIME_FORMAT for these particular files in the app that is collecting them.  I've not had to dig this deep into Splunk props before.  Wishing myself luck...

Labels (1)
0 Karma
1 Solution

soutamo
SplunkTrust
SplunkTrust
Hi
You must add TIME_FORMAT to props.conf https://docs.splunk.com/Documentation/Splunk/8.0.6/Admin/Propsconf with correct parameters. Here it should be %d-%m-%y %H:%M:%S
The easiest way to test these is your dev instance with Settings - Add data - monitor - files and directories. Then just test parameters with correct sourcetype with your sample data.
When it’s working, just copy props.conf and in needed also transforms.conf to your production environment.
r. Ismo

View solution in original post

bgstein
Path Finder

the debug info shows the file being monitored without errors. 

closer check within the splunk console and parsing the log source shows the logs 01-10-20 getting inserted with the Jan 01 logs. 

So having never verifed that these logs were getting parsed correctly in the first place I now see that I've got a mess.

0 Karma

soutamo
SplunkTrust
SplunkTrust
Hi
You must add TIME_FORMAT to props.conf https://docs.splunk.com/Documentation/Splunk/8.0.6/Admin/Propsconf with correct parameters. Here it should be %d-%m-%y %H:%M:%S
The easiest way to test these is your dev instance with Settings - Add data - monitor - files and directories. Then just test parameters with correct sourcetype with your sample data.
When it’s working, just copy props.conf and in needed also transforms.conf to your production environment.
r. Ismo

View solution in original post

bgstein
Path Finder

Yes - that was indeed what was needed.

It didn't seem to work initially - after restarting the Splunk server and after restarting the Splunk UF agent on the client.  But checking back a day later it was correctly parsing.  Most likely I was impatient.

Thank you.

bgstein
Path Finder

That didn't work...

Moving on to seeing what setting DEBUG in log.cfg shows:

category.TailingProcessor=INFO
category.WatchedFile=INFO
category.ArchiveProcessor=INFO
category.TailReader=INFO

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.