Getting Data In
Highlighted

Updating metadata sourcetype with data from events

Builder

Hello all... I have the following file:

conn.log: 1486576311.492453 Cid7Nq2yj6VZ3FdO8b  10.28.7.27  39525   10.12.7.17  8080    tcp -   -   -   -   OTH T   T   0   C   0   0   0   0   (empty)

I need to carve the first element (conn.log) delimited by . to be my source type. This value can vary but it will always have a . as a delimiter. Here is my props.conf:

[bro]
REPORT-format = BroAutoSrc, TrashComments
SHOULD_LINEMERGE = false
TRUNCATE = 0
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %s.%6N

And transforms.conf:

splunk[/opt/splunk/etc/apps/bro/default] # cat transforms.conf 
[BroAutoSrc]
DELIMS = "."
FIELDS = "orig_source"
REGEX = (\.[a-zA-Z0-9]+\.)?([a-zA-Z0-9]+)
FORMAT = sourcetype::bro_$1
DEST_KEY = MetaData:Sourcetype
WRITE_META = true

[TrashComments]
REGEX = ^\s*#
DEST_KEY = queue
FORMAT = nullQueue

This is not working. What am I doing wrong? Does the source typing have to take place on indexing or can it be done at search time? Any help is much appreciated, thanks.

0 Karma
Highlighted

Re: Updating metadata sourcetype with data from events

SplunkTrust
SplunkTrust

Sourcetype like time, line breaking, source etc are index time fields not search time. Bro is a high volume data source. I would not recommend index time rewriting of sourcetype. Just set the sourcetype correctly at inputs.

0 Karma
Highlighted

Re: Updating metadata sourcetype with data from events

Builder

That is GREAT feedback and I did wonder about that! I was trying to basically recreate the Bro for IDS Splunk app since the sensors can generate allot of different kinds of files with different fields. The issue we have with this model is that we have the bro logs aggregating into our syslog-ng server so we lose the headings that are generated in the files, which is what the app uses to name fields/keys. I cannot imagine I am the first splunker to encounter this issue.

Any thoughts are more than welcome!

0 Karma