Sourcetype won't split on monitored file after changing transforms.conf and props.conf.

[EdgarAllenProse]

I am testing splitting sourcetypes for a one time indexed file on my test box. All time formats are parsed correctly when the log ingests. The file splits just fine into exactly as many events as expected.

But there are 3 sourcetypes I need to split it into: Send, receive and scan as the message section of the logs vary heavily. The regex has been tested and works fine. No errors from btool.

inputs.conf

 [monitor://C:\\Users\\<user>\\Desktop\\security\\Splunk\\DEVELOPMENT\\Test_Ingest\\test_raw_spam.txt]
 disabled = false
 sourcetype = test_barracuda
 index = test

props.conf - to note, I tried the TRANSFORMS- line in the test_barracuda stanza, but still no results.

[test_barracuda]
CHARSET=AUTO
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
category=Custom
disabled=false
pulldown_type=true

[source:\\C:\\Users\\<user>\\Desktop\\security\\Splunk\\DEVELOPMENT\\Test_Ingest\\test_raw_spam.txt]
TRANSFORMS-changesourcetype = send_set_sourcetype, recv_set_sourcetype, scan_set_sourcetype

Transfoms.conf

[send_set_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\sSEND\s)
FORMAT = sourcetype::test_send

[recv_set_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\sRECV\s)
FORMAT = sourcetype::test_recv

[scan_set_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\sSCAN\s)
FORMAT = sourcetype::test_send

When I do a search after resetting, I am not seeing any results in the new sourcetypes, only in test_barricuda. Any thoughts?

I forced re-indexing of all file monitors because of the fact that this was a one time monitor, still no results in new sourcetypes.

command used:

splunk clean eventdata _thefishbucket

[somesoni2]

Give this a try

input.conf (same)

props.conf

[test_barracuda]
CHARSET=AUTO
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
category=Custom
disabled=false
pulldown_type=true
TRANSFORMS-overridest = set_sourcetype

transforms.conf

[set_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = \d+\s+(SEND|SCAN|RECV)\s
FORMAT = sourcetype::test_$1

Assuming you've a standalone Splunk instance, so restart Splunk after you make the change.

[EdgarAllenProse]

Is it okay that I did these in splunk_home/etc/app/search/local/?

[EdgarAllenProse]

I plugged those in and cleaned the fishbucket, then restarted still didn't work.

query

index=* NOT (sourcetype=Win* OR sourcetype=Perf*) | stats count by sourcetype

results in

sourcetype      count   
Test               3825
shiftlog              42
test_barracuda  22950 

[somesoni2]

Strange. Can you try to log in to Splunk web of your Test box and go to Settings-> Sourcetype and check if you see the updated sourcetype with new settings?

[EdgarAllenProse]

It did not update in the settings

[somesoni2]

Can we have a sample event?

[EdgarAllenProse]

Here is a sample containing each type of event: anonymized, but not in a way that would conflict with regex or the confs.

Feb 13 12:14:57 192.168.x.x outbound/smtp: 127.0.0.1 1487013294-09b08c0e0d22d7b0001-vy5CMk 0 0 SEND - 1 0112918C8063 250 2.6.0 <2050829162.21743.1487013293752.JavaMail@dc1prjasszap434.whc> [InternalId=91736206477550, Hostname=host.prod.outlook.com] 11518 bytes in 0.192, 58.365 KB/sec Queued mail for delivery #to#name-com.mail.protection.outlook.com[8.8.8.8]:25

Feb 13 12:14:56 192.168.x.x  scan: mail2-3.place.com[8.8.8.8] 1487013294-09b08c0e0d22d7b0001-vy5CMk 1487013294 1487013296 SCAN - prvs=7217d0fa1f=services_noreply@place.com name@otherplace.com - 7 88 corporate SZ:3263 SUBJ:Message: Attempt to retrieve your User ID

Feb 13 12:14:15 192.168.x.x  inbound/pass1: name.place1.com[8.8.8.8] 1487013254-09b08c0e0e22d7a0001-VY5SBA 1487013254 1487013255 RECV information=place2.com@thing.com name@thingy1.com 2 3 blacklist.org[8.8.8.8]