topic Re: Field transformation based on source in Getting Data In

Field transformation based on source

stefan_radovano — Thu, 12 Jun 2014 09:30:00 GMT

Hi All,

We log data from devices belonging to different customers, they are written to our syslog server in files named /data/log/CUSTOMER/site/router1.log, for example. I wanted to have a search-time field called customer with the value CUSTOMER, taken from the source filename.

I did this via the web GUI, under Settings/Fields/Field Transformation, this is what was written in $SPLUNK_HOME/etc/apps/search/local/transforms.conf:

[get_customer]
FORMAT = customer::$1
REGEX = \/data\/log\/(.*)\/site.*
SOURCE_KEY = MetaData:Source

Unfortunately nothing happens, I get no field named customer when I search. From what I can tell, the regex is correct. I also tried just "source" as SOURCE_KEY but nothing changed. Is anything wrong with my transform ?

I am also not sure how this transform is applied, is it run against log messages arriving via all indexes ?

As additional info, we are running splunk on a separate server (so basically the indexer) and we use a light forwarder on our syslog server. The transform above is done on the indexer.

Thanks,
Stefan

Re: Field transformation based on source

martin_mueller — Fri, 13 Jun 2014 21:34:38 GMT

Did you specify a REPORT-foo = get_customer entry in props.conf for that sourcetype?

Re: Field transformation based on source

emechler_splunk — Fri, 13 Jun 2014 22:10:01 GMT

In props.conf:

[your_sourcetype]
EXTRACT = \/data\/log\/(?<customer>\w+)\/site.* in source

No need to touch transforms.conf for this.

Re: Field transformation based on source

stefan_radovano — Sat, 14 Jun 2014 12:37:24 GMT

Yes, I do have that too.

Re: Field transformation based on source

stefan_radovano — Sat, 14 Jun 2014 13:15:28 GMT

I actually tried to do this via GUI (Fields/Field Extraction) but when I chose "source" for "Apply To", it also wanted me to specify which source. I obviously didn't want to restrict it to one particular source, didn't know what to put in there. Was I doing something wrong ?

In any case, this worked. I just had to restart splunk. Is there no way around restarting ?

Re: Field transformation based on source

emechler_splunk — Sat, 14 Jun 2014 16:41:59 GMT

You can do this in SPL itself:

| extract reload=t

Re: Field transformation based on source

martin_mueller — Sat, 14 Jun 2014 18:50:59 GMT

You could also just add in source after the end of the field extraction regex when editing it through the UI.