Why is the timestamp of event data not being recognized and events are not breaking?

[rewritex]

I initially tried auto, but was getting the same issue of the event data not line breaking correctly. I tried to modify the sourcetype info for the timestamp and still receiving the error. I've tried a combination of linemerge =true/false and timestamp prefix = ^
For some additional information. I am monitoring a directory for the logs. Each log file is named differently request.log.2017-03-18-16-00-10 or request.log.2017-03-17-11-55-11 or etc. I use a custom stanza in the inputs.conf on the forwarder.

Any advice? Maybe I missed something? Thank You.

inputs.conf on the forwarder

[monitor:///logs/sam/pr/server0*/default/web/]
_TCP_ROUTING = group1
disabled = false
index = sw-sam
sourcetype = sam
crcSalt = <SOURCE>
ignoreOlderThan = 7d

[monitor:///logs/isam/ct/server0*/default/web/]
_TCP_ROUTING = group1
disabled = false
index = test-sw-sam
sourcetype = sam
crcSalt = <SOURCE>
ignoreOlderThan = 7d

index=_Internal Message

index=_internal isam message="Breaking event because limit of 256 has been exceeded - data_source=\"/logs/sam/pr/server01/default/web/request.log.2017-03-18-16-00-10\", data_host=\"hostes\", data_sourcetype=\"sam\""

Event Data

    2017-03-19_11:09:33 -0700 ^ 192.168.1.230 ^ unauthenticated ^ GET /images/logo-top.png HTTP/1.1 ^ 200 ^ 000000607 ^ 17265 
`^ / ^ Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
    2017-03-19_11:09:33 -0700 ^ 192.168.1.230 ^ unauthenticated ^ GET /images/login-button.png HTTP/1.1 ^ 200 ^ 000000305 ^ 1595 ^ / ^ Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
    2017-03-19_11:09:33 -0700 ^ 192.168.1.230 ^ unauthenticated ^ GET /images/toggle-icon.png HTTP/1.1 ^ 200 ^ 000000957 ^ 47909 ^ / ^ Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
    2017-03-19_11:09:33 -0700 ^ 192.168.1.230^ unauthenticated ^ GET /images/twitter_logo_blue_25x21.png HTTP/1.1 ^ 200 ^ 000000249 ^ 568 ^ / ^ Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Geck

Sourcetype

Timestamp format: %Y-%m-%d_%H:%M:%S %z
Lookahead: 26

[rewritex · Accepted solution]

Thank you for all of the comments. I ended up doing the below and it seems to be working fine now. I also put the props.conf and transforms.conf onto my cluster by pushing the cluster_bundle.

[somesoni2]

Use following for your sourcetype definition/props.conf entry on your Indexer/Heavy forwarder

[sam]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d+-\d+-\d+_\d+)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d_%H:%M:%S %z
MAX_TIMESTAMP_LOOKAHEAD = 25

[cpetterborg]

I used the same configuration that I see that you have used (which I did by hand to make sure I arrived at your same config) and it would not work. But then I used a cut and paste of the date into the Add Data app, and then modified it to see if It would work any better (in case there were some characters that I didn't type right), and I got this to work:

[ __auto__learned__ ]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=20\d\d-\d\d-\d\d_\d\d
TIME_FORMAT=%Y-%m-%d_%H:%M:%S %z
MAX_TIMESTAMP_LOOKAHEAD=32

It looks quite the same, but it did validate with the Add Data app. If all the characters make it through the text mill in this forum, it should work for you, too.

[rewritex]

Thank you for your advice. I totally forgot about the add data_input tool to develop/view sourcetype ...

[skoelpin]

Can you post the contents of your props.conf file?

This should be located on your indexer $SPLUNK_HOME\etc\system\local OR $SPLUNK_HOME\etc\apps\<APP-NAME>\local