Getting Data In
Highlighted

Adding a field and changing source from Source

Motivator

hello

I am trying to extract a field and change the value of source for apache logs. The source comes as

/tmp/usr/local/sse/apache-ers/servers/aep.qa.abcd.org/logs/https_access.log.3242r4252"

First, I am trying to extract aep.qa.abcd.org as the field filename and change the source to

  /tmp/usr/local/sse/apache-ers/servers/aep.qa.abcd.org/logs/https_access.log

props.conf

[apache]
TRANSFORMS-replace_values_from_source = replace_filename_from source , replace_source_from_source

transforms.conf

[replace_filename_from source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){6}/(?P<filename>[a-z.]+)\/
FORMAT = filename::$1


[replace_source_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::^(\S+\.log)\.
FORMAT = source::$1
DEST_KEY = MetaData:Source

It doesn't seem to work. I tested the regexes and they work fine with rex command. Any ideas?

Tags (1)
0 Karma
Highlighted

Re: Adding a field and changing source from Source

Motivator

I applied them on indexers and also applied the transforms on Search head. Still doesn't work

0 Karma
Highlighted

Re: Adding a field and changing source from Source

Motivator

Removed transforms from index time and sent them only to searchtime for filename extraction. Even this doesn't work. Do I need to mention any DEST_KEY for it to work?

0 Karma
Highlighted

Re: Adding a field and changing source from Source

Path Finder

I'm not sure you need the source:: in the regex. We're not doing QUITE the same thing (not changing source, just sourcetype and index) but the "customer" field is an index-time field like what you're doing with filename. This is applied on our Splunk heavy forwarder that reads the files. If you're using a heavy forwarder to read the data, these should go there. If you're using a universal forwarder, or have the inputs.conf configured on the indexer, these should go on the indexer.

props.conf

[source::/var/log/netsyslog/...]
TRANSFORMS-sourcetype = set_sourcetype_from_source_path
TRANSFORMS-customer = set_customer_from_source_path
TRANSFORMS-index = set_index_from_source_path

transforms.conf

[set_sourcetype_from_source_path]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype
REGEX = /var/log/netsyslog/[^/]+/([^/]+)/
FORMAT = sourcetype::$1

[set_customer_from_source_path]
SOURCE_KEY = MetaData:Source
WRITE_META = true
REGEX = /var/log/netsyslog/([^/]+)/
FORMAT = customer::$1

[set_index_from_source_path]
SOURCE_KEY = MetaData:Source
DEST_KEY = _MetaData:Index
REGEX = /var/log/netsyslog/([^/]+)/([^/]+)/
FORMAT = $2_$1

I'm not sure that you need the "source::" in the regex. In your first transform, you have a space in the stanza name, as well as escaping only one of the / in the regex (you don't need to escape any of them). In the second transform, you have "^source::^" which I don't think is what you want. You can give these a try, but I haven't tested them:

props.conf

[apache]
TRANSFORMS-replace_values_from_source = replace_filename_from_source,replace_source_from_source

transforms.conf
[replacefilenamefrom source]
SOURCEKEY = MetaData:Source
WRITE
META = true
REGEX = (?:/[^/]+){6}/(?P[a-z.]+)/
FORMAT = filename::$1

[replace_source_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^(\S+\.log)\.[^.]+$
FORMAT = source::$1
DEST_KEY = MetaData:Source
0 Karma
Highlighted

Re: Adding a field and changing source from Source

Motivator

Still doesn't work. I am not sure why but I think its not even applying them,which is kinda strange. I have done the similar extraction of yours for host,source and index from source for our rsyslog data sometime back and it worked on the HF.

0 Karma
Highlighted

Re: Adding a field and changing source from Source

Motivator

Tried it one by one too. Doesn't work.Any ideas?

0 Karma
Highlighted

Re: Adding a field and changing source from Source

Path Finder

The logs are definitely being identified as sourcetype apache, right?

0 Karma
Highlighted

Re: Adding a field and changing source from Source

Motivator

[monitor:///tmp/usr/local/sse/apache-ers/servers/aep.qa.abcd.org/logs/access.log]
blacklist = .(txt|gz)$
sourcetype=apache

Checked it like 20 times.

0 Karma
Highlighted

Re: Adding a field and changing source from Source

Motivator

Made a mistake of sending the file from HF. Parsing was being done on HF and these configs were not there. Moved them to UF and works like a charm.

Only change

[replace_source_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^(\S+\.log)\.[^.]+$
FORMAT = source::$1 --> Change this to FORMAT = $1
DEST_KEY = MetaData:Source

View solution in original post

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.