I have some BlueCoat proxy log files being indexed by Splunk. The indexer and Search Head both have the BlueCoat add-on which extracts a bunch of different fields. When the events are indexed, the Host is set to the name of the Universal Forwarder. I would like to change the Host value in these events to the value of an extracted field named x_bluecoat_proxy_primary_address. After reading the doc on props and transforms, here's what I've come up with on my indexer:
props.conf
[bluecoat:proxysg:access:file]
TRANSFORMS-proxyhost = x_bluecoat_proxy_primary_address
transforms.conf:
[x_bluecoat_proxy_primary_address]
SOURCE_KEY = field:x_bluecoat_proxy_primary_address
REGEX = (.*)
DEST_KEY = MetaData:Host
This is not working, obviously. Can someone clue me into what I'm doing wrong here? Thanks.
Change this:
SOURCE_KEY = field:x_bluecoat_proxy_primary_address
To this:
SOURCE_KEY = x_bluecoat_proxy_primary_address
Then deploy to the indexers, restart all Splunk instances there, and check only newly arriving data (old events will stay broken).
I got this error starting Splunk:
Undocumented key used in transforms.conf; stanza='x_bluecoat_proxy_primary_address' setting='SOURCE_KEY' key='x_bluecoat_proxy_primary_address'
I have never seen that so I cannot help you there.
Your SOURCE_KEY
must be wrong; surely you do not have a field named field:x_bluecoat_proxy_primary_address
. What it the field that you need to use and, more importantly, how is this field created?
Thank you for your response. I really don't understand Splunk well enough to create fields, and am having a very difficult time understanding these config files. All the fields I see were created by the BlueCoat TA that I have installed. The field that has value I want use for Host is named x_bluecoat_proxy_primary_address. But I'm not sure how to describe to you how it is created - I wish I could.
I believe you'd need to provide REGEX which can extract the value of x_bluecoat_proxy_primary_address from your raw events. Could you post some sample raw data?
Thanks for your response. Here is a raw event:
2016-07-07 19:39:53 31 172.20.176.110 200 TCP_NC_MISS 202 3848 GET http sync.tidaltv.com 80 /genericusersync.ashx ?dpid=1205/ user521 - - sync.tidaltv.com - "Mozilla/5.0 (Windows NT 6.1; rv:47.0) Gecko/20100101 Firefox/47.0" - OBSERVED "Web Ads/Analytics" - 192.168.15.21 SG-HTTP-Service
The BlueCoat TA is extracting 192.168.15.21 into a field named x_bluecoat_proxy_primary_address. This is the value I would like to use for Host. And when I read the doc for transforms, I saw this about SOURCE_KEY:
If starts with "field:" or "fields:" the meaning is changed.
Instead of looking up a KEY, it instead looks up an already indexed field.
For example, if a CSV field name "price" was indexed then
"SOURCE_KEY = field:price" causes the REGEX to match against the contents
of that field. It's also possible to list multiple fields here with
That's why I tried what I did. But I am really, really confused and frustrated by these config files, so I really have no idea if what I want to do is even possible.
Well, it does say that it works for already indexed field. The x_bluecoat_proxy_primary_address could very well be a search time extracted field, thus it didn't work. There should be a props.conf entry which is doing this search time field extraction. I would find that out and use the regular expression from that props.conf and use it in your Host Override transform.conf.