line_breaker for config files

[EricPartington]

I am trying to index configuration files for a secure web gateway device (surfing appliance).
The configuration can get quite long (1,000 lines)
I would like to index the file and split on the sub headings in the file so that the chunks are manageable and easier to extract data from.
I have two problems
1) - there is a date in the header of the file (line #3) that causes the indexing to stop on the file after line #2

!- Version: ABC 1.2.3 Proxy Edition
!- Serial number: abcde12345
!- Local time: 2013-02-26 04:17:01-00:00UTC
!- BEGIN networking

2) - I think i am setting my props properly to split the event based on the occurance of " !- BEGIN " but that isnt working wither (i get a chunk of 257 lines then the events are split by line which is not what i would like).

the inputs and props are sent to a windows forwarder to monitor a directory for new files
here is the props.conf

[abcd_config]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LEARN_MODEL = false
LINE_BREAKER = (!- BEGIN\s)

the headings i want to break the event on look like this ...
!- BEGIN networking
!- BEGIN ssl
!- BEGIN authentication

etc

any thoughts on what I am missing ?

[lguinn2 · Accepted solution]

Use BREAK_ONLY_BEFORE instead of LINE_BREAKER and add in MAX_EVENTS. By default, Splunk will allow only 256 lines per event, but you can change that...

I would do this:

[abcd_config]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LEARN_MODEL = false
BREAK_ONLY_BEFORE=!- BEGIN\s
MAX_EVENTS=1000
TRUNCATE=0

[lguinn2]

Good catch on the SHOULD_LINEMERGE - I should have thought of that. When you have SHOULD_LINEMERGE=false, Splunk breaks the data stream into single-line events and ignores the BREAK_ONLY_BEFORE setting. I must have been dreaming to leave that in!

[EricPartington]

thanks for the input.. had to make a few modifications

[abcd_config]
DATETIME_CONFIG = NONE
#SHOULD_LINEMERGE = false
LEARN_MODEL = false
BREAK_ONLY_BEFORE=!-\sBEGIN
MAX_EVENTS=20000
TRUNCATE=0

also the monitor was being deployed to a forwarder but the index was living on another server. Had to make sure the monitor was deployed to the forwarder and the props.conf was on the indexer where the data was being stored. Otherwise the props.conf settings were not being read (and skipped).

thanks for getting me in the right direction.