Hi,
I am attempting to set up the Sophos Add-On (App 1854) and have encountered a quandary.
I am setting it up using a forwarder on the Sophos Enterprise Console. The Reporting Interface is already there and working fine. LogWriter is putting logs out as expected. The logs closely match the ones included with the Add-On with the exception that mine do not have quotes (") around the data. I used all the default settings (but specified my own index to send data into) and found that while all the data was ingested, the EventTime field was not recognized as the time the event occurred so all the events were imported and stamped as happening "now". I reviewed the props.conf and modified these entries for the relevant types:
TIME_PREFIX = EventTime=" (changed to remove the '=' and '"'
TIME_FORMAT = %Y-%m-%d %H:%M:%S (verified)
MAX_TIMESTAMP_LOOKAHEAD = 25 (changed to 75 to match actual log files)
however that did not appear to help.
I did raise it with Sophos just in case it was a "quote" issue and I have found that the output from Reporting Interface/LogWriter does not have quotes and isn't easy to change to use quotes.
Any thoughts as to what I should look at ?
Thankyou
Try this please:
TIME_PREFIX = EventTime= #<- added equals back
TIME_FORMAT = "%Y-%m-%d %H:%M:%S" #<- added quotes because of the space
MAX_TIMESTAMP_LOOKAHEAD = 78 #<- 78 as we discussed
Try this please:
TIME_PREFIX = EventTime= #<- added equals back
TIME_FORMAT = "%Y-%m-%d %H:%M:%S" #<- added quotes because of the space
MAX_TIMESTAMP_LOOKAHEAD = 78 #<- 78 as we discussed
Thankyou, sadly already had the "=" in there. I have this set in the local\props.conf
[sophos:firewall]
TIME_PREFIX = EventTime=
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 78
So, interestingly, I tried again this morning and I think I found the issue, or possibly it's a fluke. I wanted to try again today as the indexers/search head do a staged restart early in the morning. I am thinking that perhaps the changes on the forwarder were ok but the same props.conf changes on the indexer/search head had not been picked up. When I tried with a different set of data today the EventTime and _Time were correct.
I will try again later with another sourcetype just to make sure. If that is the case, then it was the MAX_TIMESTAMP_LOOKAHEAD = 78 that most likely resolved it. Il will let you know later today or tomorrow.
Thankyou !!!
Just added another sourcetype. Events are coming in correctly now.
Thankyou again. Much appreciated.
You're very welcome. Thanks for staying tuned and reporting back!
Can you please provide an example _raw event?
Raw records below (raw view in Splunk Search) with appropriate fields obsfucated. The _time entry for all 3 of these is :
1/6/16
11:56:20.000 AM .
InsertedAt=2016-01-06 00:52:05; EventID=1247; EventTime=2016-01-06 00:52:04; EventTypeID=; EventType=; Name=; ReportingName=; UserName=NT AUTHORITY\SYSTEM; ActionID=; Action=; SubTypeID=; SubType=; Role=; FileName=; FilePath=; FileVersion=; FileChecksum=; CommandLine=; Session=; Desktop=; Location=; ProtocolID=; Protocol=; DirectionID=; Direction=; LocalAddress=; RemoteAddress=; LocalPort=; RemotePort=; ComputerName=COMPNAM; ComputerDomain=THEDOM; ComputerIPAddress=99.99.99.99
InsertedAt=2016-01-04 02:17:47; EventID=1246; EventTime=2016-01-04 02:17:47; EventTypeID=; EventType=; Name=; ReportingName=; UserName=NT AUTHORITY\SYSTEM; ActionID=; Action=; SubTypeID=; SubType=; Role=; FileName=; FilePath=; FileVersion=; FileChecksum=; CommandLine=; Session=; Desktop=; Location=; ProtocolID=; Protocol=; DirectionID=; Direction=; LocalAddress=; RemoteAddress=; LocalPort=; RemotePort=; ComputerName=COMPNAM; ComputerDomain=THEDOM; ComputerIPAddress=99.99.99.99
InsertedAt=2016-01-04 02:17:41; EventID=1245; EventTime=2016-01-04 02:17:41; EventTypeID=; EventType=; Name=; ReportingName=; UserName=NT AUTHORITY\SYSTEM; ActionID=; Action=; SubTypeID=; SubType=; Role=; FileName=; FilePath=; FileVersion=; FileChecksum=; CommandLine=; Session=; Desktop=; Location=; ProtocolID=; Protocol=; DirectionID=; Direction=; LocalAddress=; RemoteAddress=; LocalPort=; RemotePort=; ComputerName=COMPNAM; ComputerDomain=THEDOM; ComputerIPAddress=99.99.99.99
If you copy everything up to the semicolon ";" after EventTime, you'll get to the number of 76..
This means your MAX_TIMESTAMP_LOOKAHEAD should be 76 not 75. However, you have these EventID fields which equal a number in the thousands... it could just as easily report an eventID of 65535 (the max). Therefore you probably want to add another digit to MAX_TIMESTAMP_LOOKAHEAD. Bringing my final recommendation of 77 or 78.
Please try both MAX_TIMESTAMP_LOOKAHEAD = 77 and MAX_TIMESTAMP_LOOKAHEAD = 78 and let me know the results.
I may have another try tomorrow. I altered the lookahead to 78 and the following was the result (forwarder restarted to send data on 19th Jan at around 10:30am local time):
1/18/16
8:17:11.000 AM
InsertedAt=2016-01-17 21:12:44; EventID=1277; EventTime=2016-01-17 21:12:43; EventTypeID=; EventType=; Name=; ReportingName=; UserName=NT AUTHORITY\SYSTEM; ActionID=; Action=; SubTypeID=; SubType=; Role=; FileName=; FilePath=; FileVersion=; FileChecksum=; CommandLine=; Session=; Desktop=; Location=; ProtocolID=; Protocol=; DirectionID=; Direction=; LocalAddress=; RemoteAddress=; LocalPort=; RemotePort=; ComputerName=MYCOMP; ComputerDomain=MYDOM; ComputerIPAddress=99.99.99.99
1/18/16
8:17:11.000 AM
InsertedAt=2016-01-17 21:11:38; EventID=1276; EventTime=2016-01-17 21:11:38; EventTypeID=; EventType=; Name=; ReportingName=; UserName=NT AUTHORITY\SYSTEM; ActionID=; Action=; SubTypeID=; SubType=; Role=; FileName=; FilePath=; FileVersion=; FileChecksum=; CommandLine=; Session=; Desktop=; Location=; ProtocolID=; Protocol=; DirectionID=; Direction=; LocalAddress=; RemoteAddress=; LocalPort=; RemotePort=; ComputerName=MYCOMP; ComputerDomain=MYDOM; ComputerIPAddress=99.99.99.99
1/18/16
8:17:11.000 AM
InsertedAt=2016-01-17 21:11:34; EventID=1275; EventTime=2016-01-17 21:11:33; EventTypeID=; EventType=; Name=; ReportingName=; UserName=NT AUTHORITY\SYSTEM; ActionID=; Action=; SubTypeID=; SubType=; Role=; FileName=; FilePath=; FileVersion=; FileChecksum=; CommandLine=; Session=; Desktop=; Location=; ProtocolID=; Protocol=; DirectionID=; Direction=; LocalAddress=; RemoteAddress=; LocalPort=; RemotePort=; ComputerName=MYCOMP; ComputerDomain=MYDOM; ComputerIPAddress=99.99.99.99
Your time prefix should have the equal sign too.
Can you repost the props we have now?
Will do, thankyou !
Hi,
Thankyou. Will do next week when I have access again. I'm also going to have another go in my test environment.
Just to make sure: you did verify your settings by restarting splunk and checking newly indexed data? Existing data will not be affected.
Hi. Absolutely, yes. Part of my paranoia process 🙂