props.conf not applying

[irwinj_125]

Hello,

I have a log file with dates occurring inside the lines (not just at the beginning of the line). Splunk is creating a separate event each time the date/timestamp is encountered, not just at the beginning of the line. I've done a lot of research on these forums and have tried playing extensively with props.conf inside my etc/system/local directory (which I believe is highest priority). I've tried using "LINE_BREAKER" with a regular expression (date/time stamp at the beginning of the line) and "SHOULD_LINEMERGE" set to false, have also tried "BREAK_ONLY_BEFORE", "TIME_PREFIX", "TIME_FORMAT", etc. Anytime I've made these changes and re-started Splunk, I am able to see them when I use the btool command to check for props settings, so they do seem to be picking up. However, in my GUI, my log files continue to break at any date/timestamp encountered.

Perhaps there is something else wrong with my settings. Here's what my input.conf looks like and one thing I've tried for props.conf in the same folder.

input.conf entry:
[monitor:///path_to_log/log_file_name*.log]
disabled = 0
sourcetype = log_file_name

props.conf entry (just one of many settings I've tried):
[log_file_name]
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}
sourcetype = log_file_name

Any suggestions would be appreciated.

[ekenne06]

should be LINEMERGE, not LINE_MERGE

[ekenne06]

I finally got mine to work. It was actually due to me not linebreaking properly on the right sourcetype. I would try testing your props.conf by making a LINE_BREAKER to something super simple, so if it works, you know it's just your config. If it doesn't work that means the sourcetype isn't being recognized. Once I found the right sourcetype I did:

LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}
TIME_PREFIX = ^
LINE_MERGE = FALSE

Since my timestamp is the start of every event, that was the best think to line break on.

[irwinj_125]

Excellent!

I finally got mine working too (details below).  Good to go into the weekend with problems solved.

[richgalloway]

A couple of thing to note since they're not mentioned in the question.

1. Changes to config files don't take effect until Splunk restarts.
2. Changes to props.conf only affect NEW data.  Events already indexed never change.

[irwinj_125]

Thanks.

Yes, I 've restarted the splunk forwarder each time I've made changes.

To test, I create a new log file in the log directory containing the required data.  I see the new data in the GUI, but not with the expected breaks.

[richgalloway]

To apply props.conf changes, it's the indexer that must be restarted rather than the universal forwarder.

If you use a heavy forwarder then the props.conf changes go there as well (and the HF must be restarted).

[irwinj_125]

Thanks, that is good to know.  I can stop/start the forwarder at anytime, but probably not the indexer as its heavily in use.  

Finally found a solution based on the feedback here: https://community.splunk.com/t5/Getting-Data-In/Universal-Forwarder-and-props-conf-and-transforms-co...

Once I added force_local_processing = true into my local props.conf, the data appears as I expect it.  

One thing I didn't fully understand in the above is this quote: "Note that if the Universal Forwarder does the indexing, the Splunk instances won't: all of the index-time work must be done on the Universal Forwarder."  Does this basically mean that any further indexing laid out on the indexer itself will not take place for this specific sourcetype?

Splunk documentation also says regarding this: 

Note that switching this property potentially increases the cpu
  and memory consumption of the forwarder.

Not sure how concerned I should be about this.

Thanks Rich for your guidance.

[richgalloway]

Good find, but I would consider that a temporary fix.  Restart the indexer in the next maintenance window and then turn off that flag in the UF.

You read it correctly, the UF is now doing the work of the indexer (except for the write-to-disk part).  It's causing the UF to use more CPU, memory, and network bandwidth.

[irwinj_125]

Thanks Rich,

I'll arrange that.

Just to confirm - the props.conf stays located on the forwarder server, just the way it is (minus the "force_local_processing" flag).  Once I re-start the indexer, the changes in the props.conf on the forwarder server will take effect, correct?

[irwinj_125]

Hi @richgalloway 

Would you be able to guide me on the above question?  Just want to be sure.  With the force local processing flag set to true, and the inputs.conf and props.conf in the  etc/system/local directory on the UF, things work as expected.  If I turn off the force local processing flag and re-cycle the indexer, should the other settings in the props.conf (located on the UF) come into play?  Or would I need to create the props.conf in the  etc/system/local directory on the indexer server (rather than UF)?

Normally I would just experiment and see, but as mentioned its not as easy for me to re-start the indexer as it is to re-start the UF.

[irwinj_125]

hi @richgalloway ,

I was able to figure this out as I was able to re-cycle the indexer (enterprise) today.  Initially it did not work, having the props.conf just on the UF side.  I then copied the props.conf into /etc/system/local on the Indexer and re-cycled, after this it worked as expected.

Thanks for all your guidance on this.

[ekenne06]

i'm having this same exact issue. Here is my post:

https://community.splunk.com/t5/Getting-Data-In/XML-Data-Line-Breaking-on-DateTime-tag/m-p/537715#M9...

Given a suggestion to set the TIM_PREFIX = ^ That should only search for the timestamp at the beginning of the data. However this isn't working for me. Can you give it a go and let me know how it works? 

[ekenne06]

Sorry, TIME_PREFIX = ^

[irwinj_125]

Yep I tried that one as well...seemed to make sense but no luck.  I've thinking I have some other configuration issue at play.

[ekenne06]

Here is something else I found, just haven't been able to test it yet. https://community.splunk.com/t5/Getting-Data-In/Timestamp-and-line-not-properly-break/m-p/262342

will let you know if it helps at all