topic Re: Sourcetype won't split on monitored file after changing transforms.conf and props.conf. in Getting Data In

Sourcetype won't split on monitored file after changing transforms.conf and props.conf.

EdgarAllenProse — Mon, 13 Feb 2017 20:58:16 GMT

I am testing splitting sourcetypes for a one time indexed file on my test box. All time formats are parsed correctly when the log ingests. The file splits just fine into exactly as many events as expected.

But there are 3 sourcetypes I need to split it into: Send, receive and scan as the message section of the logs vary heavily. The regex has been tested and works fine. No errors from btool.

inputs.conf

 [monitor://C:\\Users\\<user>\\Desktop\\security\\Splunk\\DEVELOPMENT\\Test_Ingest\\test_raw_spam.txt]
 disabled = false
 sourcetype = test_barracuda
 index = test

props.conf - to note, I tried the TRANSFORMS- line in the test_barracuda stanza, but still no results.

[test_barracuda]
CHARSET=AUTO
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
category=Custom
disabled=false
pulldown_type=true

[source:\\C:\\Users\\<user>\\Desktop\\security\\Splunk\\DEVELOPMENT\\Test_Ingest\\test_raw_spam.txt]
TRANSFORMS-changesourcetype = send_set_sourcetype, recv_set_sourcetype, scan_set_sourcetype

Transfoms.conf

[send_set_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\sSEND\s)
FORMAT = sourcetype::test_send

[recv_set_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\sRECV\s)
FORMAT = sourcetype::test_recv

[scan_set_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\sSCAN\s)
FORMAT = sourcetype::test_send

When I do a search after resetting, I am not seeing any results in the new sourcetypes, only in test_barricuda. Any thoughts?

I forced re-indexing of all file monitors because of the fact that this was a one time monitor, still no results in new sourcetypes.

command used:

splunk clean eventdata _thefishbucket

Re: Sourcetype won't split on monitored file after changing transforms.conf and props.conf.

somesoni2 — Mon, 13 Feb 2017 21:38:26 GMT

Can we have a sample event?

Re: Sourcetype won't split on monitored file after changing transforms.conf and props.conf.

EdgarAllenProse — Mon, 13 Feb 2017 21:58:24 GMT

Here is a sample containing each type of event: anonymized, but not in a way that would conflict with regex or the confs.

Feb 13 12:14:57 192.168.x.x outbound/smtp: 127.0.0.1 1487013294-09b08c0e0d22d7b0001-vy5CMk 0 0 SEND - 1 0112918C8063 250 2.6.0 <2050829162.21743.1487013293752.JavaMail@dc1prjasszap434.whc> [InternalId=91736206477550, Hostname=host.prod.outlook.com] 11518 bytes in 0.192, 58.365 KB/sec Queued mail for delivery #to#name-com.mail.protection.outlook.com[8.8.8.8]:25

Feb 13 12:14:56 192.168.x.x  scan: mail2-3.place.com[8.8.8.8] 1487013294-09b08c0e0d22d7b0001-vy5CMk 1487013294 1487013296 SCAN - prvs=7217d0fa1f=services_noreply@place.com name@otherplace.com - 7 88 corporate SZ:3263 SUBJ:Message: Attempt to retrieve your User ID

Feb 13 12:14:15 192.168.x.x  inbound/pass1: name.place1.com[8.8.8.8] 1487013254-09b08c0e0e22d7a0001-VY5SBA 1487013254 1487013255 RECV information=place2.com@thing.com name@thingy1.com 2 3 blacklist.org[8.8.8.8]

Re: Sourcetype won't split on monitored file after changing transforms.conf and props.conf.

somesoni2 — Mon, 13 Feb 2017 22:06:11 GMT

Give this a try

input.conf (same)

props.conf

[test_barracuda]
CHARSET=AUTO
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
category=Custom
disabled=false
pulldown_type=true
TRANSFORMS-overridest = set_sourcetype

transforms.conf

[set_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = \d+\s+(SEND|SCAN|RECV)\s
FORMAT = sourcetype::test_$1

Assuming you've a standalone Splunk instance, so restart Splunk after you make the change.

Re: Sourcetype won't split on monitored file after changing transforms.conf and props.conf.

EdgarAllenProse — Mon, 13 Feb 2017 22:15:58 GMT

I plugged those in and cleaned the fishbucket, then restarted still didn't work.

query

index=* NOT (sourcetype=Win* OR sourcetype=Perf*) | stats count by sourcetype

results in

sourcetype      count   
Test               3825
shiftlog              42
test_barracuda  22950

Re: Sourcetype won't split on monitored file after changing transforms.conf and props.conf.

EdgarAllenProse — Mon, 13 Feb 2017 22:31:42 GMT

Is it okay that I did these in splunk_home/etc/app/search/local/?

Re: Sourcetype won't split on monitored file after changing transforms.conf and props.conf.

somesoni2 — Mon, 13 Feb 2017 22:33:19 GMT

Strange. Can you try to log in to Splunk web of your Test box and go to Settings-> Sourcetype and check if you see the updated sourcetype with new settings?

Re: Sourcetype won't split on monitored file after changing transforms.conf and props.conf.

EdgarAllenProse — Mon, 13 Feb 2017 22:44:58 GMT

It did not update in the settings