Updating metadata sourcetype with data from events

[brent_weaver]

Hello all... I have the following file:

conn.log: 1486576311.492453 Cid7Nq2yj6VZ3FdO8b  10.28.7.27  39525   10.12.7.17  8080    tcp -   -   -   -   OTH T   T   0   C   0   0   0   0   (empty)

I need to carve the first element (conn.log) delimited by . to be my source type. This value can vary but it will always have a . as a delimiter. Here is my props.conf:

[bro]
REPORT-format = BroAutoSrc, TrashComments
SHOULD_LINEMERGE = false
TRUNCATE = 0
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %s.%6N

And transforms.conf:

splunk[/opt/splunk/etc/apps/bro/default] # cat transforms.conf 
[BroAutoSrc]
DELIMS = "."
FIELDS = "orig_source"
REGEX = (\.[a-zA-Z0-9]+\.)?([a-zA-Z0-9]+)
FORMAT = sourcetype::bro_$1
DEST_KEY = MetaData:Sourcetype
WRITE_META = true

[TrashComments]
REGEX = ^\s*#
DEST_KEY = queue
FORMAT = nullQueue

This is not working. What am I doing wrong? Does the source typing have to take place on indexing or can it be done at search time? Any help is much appreciated, thanks.

[starcher]

Sourcetype like time, line breaking, source etc are index time fields not search time. Bro is a high volume data source. I would not recommend index time rewriting of sourcetype. Just set the sourcetype correctly at inputs.

[brent_weaver]

That is GREAT feedback and I did wonder about that! I was trying to basically recreate the Bro for IDS Splunk app since the sensors can generate allot of different kinds of files with different fields. The issue we have with this model is that we have the bro logs aggregating into our syslog-ng server so we lose the headings that are generated in the files, which is what the app uses to name fields/keys. I cannot imagine I am the first splunker to encounter this issue.

Any thoughts are more than welcome!