Getting Data In

Updating metadata sourcetype with data from events


Hello all... I have the following file:

conn.log: 1486576311.492453 Cid7Nq2yj6VZ3FdO8b  39525  8080    tcp -   -   -   -   OTH T   T   0   C   0   0   0   0   (empty)

I need to carve the first element (conn.log) delimited by . to be my source type. This value can vary but it will always have a . as a delimiter. Here is my props.conf:

REPORT-format = BroAutoSrc, TrashComments

And transforms.conf:

splunk[/opt/splunk/etc/apps/bro/default] # cat transforms.conf 
DELIMS = "."
FIELDS = "orig_source"
REGEX = (\.[a-zA-Z0-9]+\.)?([a-zA-Z0-9]+)
FORMAT = sourcetype::bro_$1
DEST_KEY = MetaData:Sourcetype

REGEX = ^\s*#
DEST_KEY = queue
FORMAT = nullQueue

This is not working. What am I doing wrong? Does the source typing have to take place on indexing or can it be done at search time? Any help is much appreciated, thanks.

0 Karma


Sourcetype like time, line breaking, source etc are index time fields not search time. Bro is a high volume data source. I would not recommend index time rewriting of sourcetype. Just set the sourcetype correctly at inputs.

0 Karma


That is GREAT feedback and I did wonder about that! I was trying to basically recreate the Bro for IDS Splunk app since the sensors can generate allot of different kinds of files with different fields. The issue we have with this model is that we have the bro logs aggregating into our syslog-ng server so we lose the headings that are generated in the files, which is what the app uses to name fields/keys. I cannot imagine I am the first splunker to encounter this issue.

Any thoughts are more than welcome!

0 Karma