I have a file that has multiple multi line events. Each event is broken up into "INFO: ---" or "ERROR: ---"
ERROR: --- blahNewsLetter: N birthdate: 1947-10-25 countryId: 1 createdOn: 2011-05-31 13:40:46 ... INFO: --- blahNewsLetter: ~ birthdate: 0000-00-00 countryId: ~ createdOn: 2011-05-31 13:40:48 ...
[dblog] SHOULD_LINEMERGE = true # force splunk to detec multiline events BREAK_ONLY_BEFORE = (.*)(INFO|ERROR):
[monitor:///var/log/dblog.log] disabled = false index = blah sourcetype = dblog blacklist = (\.(gz|bz2|z|zip)$) followTail = 1
This does not work. It is splitting the file up into events that have lines similar to the "createdOn: 2011-05-31 13:40:48" lines.
modifiedOn: 2011-05-31 13:40:48 postalCode: 1111 promoCode: ~ requestDetail: |- <request> <billToEmail>null</billToEmail> <billToFirstName>Name</billToFirstName> <billToLastName>Name</billToLastName> <billToStreet1>null</billToStreet1> <billToStreet2>null</billToStreet2> ...
createdOn: 2011-05-31 13:40:48 email: email@example.com
Any help would be appreciated. And I have also tried using BREAKONLYBEFORE_DATE = false with same result. This is currently on the forwarder.
I would recommend using LINE_BREAKER.
[dblog] SHOULD_LINEMERGE = false # force splunk to detec multiline events LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
Update: These configurations need to be on your indexer. I would also recommend modifying Splunk's date/time properties since there is not a timestamp in first 150 characters of your event.
[dblog] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR): TIME_PREFIX = createdOn:\s+ TIME_FORMAT = %Y-%m-%d %H:%M:%S
Made your recommended change, restarted the forwarder, and got the same result.
5/31/11 3:31:17.000 PM modifiedOn: 2011-05-31 15:31:17 postalCode: 11111 promoCode: '' requestDetail: |- <request> <email>firstname.lastname@example.org</email> <userName>username</userName> <firstName>firstname</firstName> <lastName>lastname</lastName> <language>en</language> Show all 18 lines
These changes have been on the Universal Forwarder where the log file is hosted. I have other settings on the forwarder for other log files, mostly setting sourcetypes in the props.conf file. Should this all be done on the indexer?
Line breaking and time parsing are done @ the indexer or Full forwarder, not Universal Forwarder/Light Forwarder. Please move these configurations to your indexer(s).
Thanks !!! That was it. It needed to be done on the indexer and not forwarder. So is there a list of things that needs to be done on the indexer vs forwarder regarding props.conf ?
Not sure if there is a specific list, but any props related to line breaking, time parsing, or of type TRANSFORMS- are performed @ the indexing/full forwarder layer of your Splunk infrastructure.