topic Re: Adding a field and changing source from Source in Getting Data In

Adding a field and changing source from Source

theouhuios — Wed, 30 Apr 2014 14:41:38 GMT

hello

I am trying to extract a field and change the value of source for apache logs. The source comes as

/tmp/usr/local/sse/apache-ers/servers/aep.qa.abcd.org/logs/https_access.log.3242r4252"

First, I am trying to extract aep.qa.abcd.org as the field filename and change the source to

  /tmp/usr/local/sse/apache-ers/servers/aep.qa.abcd.org/logs/https_access.log

props.conf

[apache]
TRANSFORMS-replace_values_from_source = replace_filename_from source , replace_source_from_source

transforms.conf

[replace_filename_from source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){6}/(?P<filename>[a-z.]+)\/
FORMAT = filename::$1


[replace_source_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::^(\S+\.log)\.
FORMAT = source::$1
DEST_KEY = MetaData:Source

It doesn't seem to work. I tested the regexes and they work fine with rex command. Any ideas?

Re: Adding a field and changing source from Source

theouhuios — Wed, 30 Apr 2014 18:42:49 GMT

I applied them on indexers and also applied the transforms on Search head. Still doesn't work

Re: Adding a field and changing source from Source

theouhuios — Thu, 01 May 2014 12:55:51 GMT

Removed transforms from index time and sent them only to searchtime for filename extraction. Even this doesn't work. Do I need to mention any DEST_KEY for it to work?

Re: Adding a field and changing source from Source

mcmaster — Mon, 28 Sep 2020 16:31:05 GMT

I'm not sure you need the source:: in the regex. We're not doing QUITE the same thing (not changing source, just sourcetype and index) but the "customer" field is an index-time field like what you're doing with filename. This is applied on our Splunk heavy forwarder that reads the files. If you're using a heavy forwarder to read the data, these should go there. If you're using a universal forwarder, or have the inputs.conf configured on the indexer, these should go on the indexer.

props.conf

[source::/var/log/netsyslog/...]
TRANSFORMS-sourcetype = set_sourcetype_from_source_path
TRANSFORMS-customer = set_customer_from_source_path
TRANSFORMS-index = set_index_from_source_path

transforms.conf

[set_sourcetype_from_source_path]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype
REGEX = /var/log/netsyslog/[^/]+/([^/]+)/
FORMAT = sourcetype::$1

[set_customer_from_source_path]
SOURCE_KEY = MetaData:Source
WRITE_META = true
REGEX = /var/log/netsyslog/([^/]+)/
FORMAT = customer::$1

[set_index_from_source_path]
SOURCE_KEY = MetaData:Source
DEST_KEY = _MetaData:Index
REGEX = /var/log/netsyslog/([^/]+)/([^/]+)/
FORMAT = $2_$1

I'm not sure that you need the "source::" in the regex. In your first transform, you have a space in the stanza name, as well as escaping only one of the / in the regex (you don't need to escape any of them). In the second transform, you have "^source::^" which I don't think is what you want. You can give these a try, but I haven't tested them:

props.conf

[apache]
TRANSFORMS-replace_values_from_source = replace_filename_from_source,replace_source_from_source

transforms.conf
[replace_filename_from source]
SOURCE_KEY = MetaData:Source
WRITE_META = true
REGEX = (?:/[^/]+){6}/(?P[a-z.]+)/
FORMAT = filename::$1

[replace_source_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^(\S+\.log)\.[^.]+$
FORMAT = source::$1
DEST_KEY = MetaData:Source

Re: Adding a field and changing source from Source

theouhuios — Fri, 02 May 2014 13:02:11 GMT

Still doesn't work. I am not sure why but I think its not even applying them,which is kinda strange. I have done the similar extraction of yours for host,source and index from source for our rsyslog data sometime back and it worked on the HF.

Re: Adding a field and changing source from Source

theouhuios — Fri, 02 May 2014 17:35:50 GMT

Tried it one by one too. Doesn't work.Any ideas?

Re: Adding a field and changing source from Source

mcmaster — Fri, 02 May 2014 17:46:39 GMT

The logs are definitely being identified as sourcetype apache, right?

Re: Adding a field and changing source from Source

theouhuios — Fri, 02 May 2014 18:10:03 GMT

[monitor:///tmp/usr/local/sse/apache-ers/servers/aep.qa.abcd.org/logs/access.log]
blacklist = .(txt|gz)$
sourcetype=apache

Checked it like 20 times.

Re: Adding a field and changing source from Source

theouhuios — Thu, 08 May 2014 15:47:57 GMT

Made a mistake of sending the file from HF. Parsing was being done on HF and these configs were not there. Moved them to UF and works like a charm.

Only change

[replace_source_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^(\S+\.log)\.[^.]+$
FORMAT = source::$1 --> Change this to FORMAT = $1
DEST_KEY = MetaData:Source