I have a data source that looks like this:
I0908 09:35:18.395637 3109 vdisk_micro_migrate_egroup_op.cc:1075] ...
I0908 09:35:18.395697 3109 vdisk_micro_migrate_egroup_op.cc:77] ...
I0908 09:35:18.395843 3146 egroup_delete_op.cc:52] ...
I0908 09:35:18.399770 3146 disk_manager.cc:1624] ...
I0908 09:35:18.504919 3106 vdisk_distributed_oplog_slave_write_op.cc:516] ...
After forwarding to the indexer, my events contain multiple lines of data - it seems to break after approximately 4000 characters.
I added this in the props.conf for the application that deals with the data:
[storage:log-Info]
SHOULD_LINEMERGE = false
LINE_BREAKER=([\r\n]+)
I know that this is the default setting anyway but it does not seem to be working. How would you go about debugging or is there any log that shows me how the events are being formed? BTW, I also tried changing the LINE_BREAKER regex to
LINE_BREAKER=([\r\n]+[I,W,E,F][0-1][0-9][0-3][0-9]\s[0-2][0-9]:[0-5][0-9]:[0-5][0-9].[0-9]{6})
No luck.
As Omid said, the props.conf should be on the indexer, not on the forwarder. To fix the problem, I (with the help of Splunk support) edited props.conf in the $SPLUNK_HOME/etc/system/local/ directory and added :
[storage:log-Info]
SHOULD_LINEMERGE = false
Thank you, Omid, for your help.
As Omid said, the props.conf should be on the indexer, not on the forwarder. To fix the problem, I (with the help of Splunk support) edited props.conf in the $SPLUNK_HOME/etc/system/local/ directory and added :
[storage:log-Info]
SHOULD_LINEMERGE = false
Thank you, Omid, for your help.
To clarify, it should be on the heavy forwarder, not the universal forwarder.
Are you not using back slashes?
I test this regex and it seemed to work:
LINE_BREAKER=([\r\n]+)[IWEF]\d+\s[0-2][0-9]:[0-5][0-9]:[0-5][0-9].[0-9]{6}
A good site to test is regexr.com.
Omid
Thank you, Omid. I will read through your link and make changes accordingly. I will post the result as soon as I can.
any update on this loui3b3?
Hi slebbie, yes. Please look at the accepted answer at the top of this thread.
The bottom line is that think of UF as kind of just a simple forwarding mechanism that sends chunks of data to the indexers. It generally doesn't do anything at the event level. There are some exceptions around csv files and things on windows but for general file monitors this is the case.
Louie: I assume you are forwarding using a universal forwarder which is good because most of the time that is the right choice. However, when you forward using a universal forwarder the parsing and indexing happens on the indexer and not the forwarder.
The props.conf configuration you are making is considered part of the parsing pipeline and so it is not done at UF but at the indexer. This link might help: http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
Omid: I tried your regex and got the same result as I was getting.
No such things as silly questions to someone who is still learning. Sourcetype is correct.
I do have props.conf on the forwarder. So you are saying that it needs to be on the indexer? I don't understand why so could you explain please? Thank you.
oh also, just to check but is props.conf on the indexer or forwarder? It should be on the indexer.
did you try my regex? If you notice they are slightly different. What splunk finds in the first capture group is discarded so if you have the whole timestamp in there it will discard that.
If you see your props.conf settings in btool it is being picked up. Silly question but is the sourcetype correct?
hortonew: I am not too familiar with btool but googling showed me that I can do:
splunk cmd btool --app=
I ran this and it shows me the contents of the props.conf of my app. Is there another way to check to see if my props.conf is being overridden?
louieb3: did you use btool to verify that props.conf isn't being overridden by another props.conf?
Thanks, Omid. Yes, the regex works. I use a tool called RegexBuddy to test regular expressions. However, after I put it into props.conf, it does not seem to do anything. It almost seems to me like props.conf is being bypassed if that makes any sense.