Hello all... I have the following file:
conn.log: 1486576311.492453 Cid7Nq2yj6VZ3FdO8b 10.28.7.27 39525 10.12.7.17 8080 tcp - - - - OTH T T 0 C 0 0 0 0 (empty)
I need to carve the first element (conn.log) delimited by . to be my source type. This value can vary but it will always have a . as a delimiter. Here is my props.conf:
[bro] REPORT-format = BroAutoSrc, TrashComments SHOULD_LINEMERGE = false TRUNCATE = 0 MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s.%6N
splunk[/opt/splunk/etc/apps/bro/default] # cat transforms.conf [BroAutoSrc] DELIMS = "." FIELDS = "orig_source" REGEX = (\.[a-zA-Z0-9]+\.)?([a-zA-Z0-9]+) FORMAT = sourcetype::bro_$1 DEST_KEY = MetaData:Sourcetype WRITE_META = true [TrashComments] REGEX = ^\s*# DEST_KEY = queue FORMAT = nullQueue
This is not working. What am I doing wrong? Does the source typing have to take place on indexing or can it be done at search time? Any help is much appreciated, thanks.
Sourcetype like time, line breaking, source etc are index time fields not search time. Bro is a high volume data source. I would not recommend index time rewriting of sourcetype. Just set the sourcetype correctly at inputs.
That is GREAT feedback and I did wonder about that! I was trying to basically recreate the Bro for IDS Splunk app since the sensors can generate allot of different kinds of files with different fields. The issue we have with this model is that we have the bro logs aggregating into our syslog-ng server so we lose the headings that are generated in the files, which is what the app uses to name fields/keys. I cannot imagine I am the first splunker to encounter this issue.
Any thoughts are more than welcome!