Host override with event data

[boknows]

Hello,

I have logs coming in with the host showing as the UF.  I want to replace the host value with some event data.

Here is a sample of the data.

 ACME-001 HOST-003: status="407/0" srcip="1.0.0.2" user="VeroRivas" dhost="http://test_web.net/contents/content1.jpg?aa=bb&cc=dd" urlp="401" proto="HTTP/https" mtd="CONNECT" urlc="Movie" rep="2" mt="text/html" mlwr="-" app="-" bytes="001/0/0/3180" ua="Mozilla/5.0 (webOS/1.3; U; en-US) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/1.0 Safari/525.27.1 Desktop/1.0" lat="0/0/0/3" rule="rule1 ok" url="http://test_web.com/page3/c.jpg?ee=ff&gg=hh"
 ACME-001 ops-sys-002: status="407/0" srcip="1.0.0.11" user="roisiningle" dhost="http://test_web.net/contents/content1.jpg?aa=bb&cc=dd" urlp="401" proto="HTTP/https" mtd="CONNECT" urlc="Food" rep="-2" mt="text/html" mlwr="-" app="-" bytes="206/0/0/0040" ua="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1" lat="0/0/0/1" rule="rule1 ok" url="http://test_web.com/page5/e.jpg?ee=ff&gg=hh"
 ACME-001 BUSDEV-005: status="200/0" srcip="1.0.0.13" user="roonixr" dhost="http://test_web.net/users/user2.jpg?ee=ff&gg=hh" urlp="10" proto="HTTP/http" mtd="GET" urlc="Advertisement" rep="-3" mt="application/javascript" mlwr="-" app="-" bytes="142/020/032/023" ua="Mozilla/5.0 (X11; U; SunOS sun4m; en-US; rv:1.4b) Gecko/20030517 Mozilla Firebird/0.6" lat="0/05/30/53" rule="rule8 good" url="http://test_web.net/users/user2.jpg?ee=ff&gg=hh"

ACME-001 is what I want to be used for the for the value of host. I am in a index cluster environment with 1 SH, CM, 2 IDX and 1 UF. I have pushed these props and transforms to the indexers with no success. The UF is still showing as the host value. 


Props

[mcafee:wg:kv]
TRANSFORMS-changehost = changehost
SHOULD_LINEMERGE = false
DATETIME_CONFIG = current
#TIME_PREFIX =
#TIME_FORMAT =
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
#MAX_TIMESTAMP_LOOKAHEAD =
TRUNCATE = 999999
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+)





Transforms

[changehost]
DEST_KEY = MetaData:Host
REGEX = ^(?P<host>\S+)
FORMAT = host::$1

Any help would be much appreciated

[PickleRick]

Adding to the valid points already raised by @gcusello , "changehost" is a name which is not very unlikely to repeat in other  apps so I'd check with btool whether something doesn't overwrite it by any chance.

splunk btool transforms list changehost --debug

That's one thing.

Another one is - I'm never sure when you need to use WRITE_META and where you don't so I just to be on the safe side use it on all index-time extractions.

[gcusello]

Hi @boknows ,

the transforms.conf isn't correct: you aren't performing a field extraction, so please try:

[changehost]
DEST_KEY = MetaData:Host
REGEX = ^([^\s]+)
FORMAT = host::$1

Then, where did you locate them?

they must be located in the first full Splunk instance they pass through, in  other words in the first heavy Forwarder or, if not present any HF, in the Indexers.

Ciao.

Giuseppe

[boknows]

I am pushing the configs from the cluster master to two indexers. No HF. The change in transforms still did not work.  I am using the  Splunk_TA_mcafee-wg . Is it possible that  a configuration is taking precedence over my changes? I have tried making a local folder in the app and adding the props and transforms there. No luck. 

[gcusello]

Hi @boknows ,

it's correct to put the configurations in the local folder of your TA.

What's the flow of your data? where do you receive data?

these seem to be data received by syslog and ususlly they are received in an Heavy Forwarder, could you describe the flow of your data through the Splunk machines?

In other words, I suppose that there's a syslog receiver, is it a Universal Forwarder or an Heavy Forwarder (a Splunk instance)?

if it is an UF, between it and the Indexers, is there some other Splunk machine? if yes, it is an UF or an HF?

At least if you're sure that there isn't any HF, put the add-on on the Indexers, otherwise on the first HF.

Ciao.

Giuseppe

[boknows]

Hello,

Syslog is being sent to a UF and then to the Indexers. No HF to do parsing. Is what I am trying to accomplish possible using search time field extractions?

[gcusello]

Hi @boknows ,

host is a metadata cofigured at index time, so it should be set once a time.

You could also define a calculated field that overrides the host field but I don't like.

So I hint to put the transformation on the Indexers, and eventually also on the UF even if isn't required.

Check, using the regex command in Splunk Search, if there's something different in your events because the regex doesn't run, e.g. a space at the beginning of the event.

Ciao.

Giuseppe

[boknows]

I have 3 sources that I need to do this for and was able to have 2 come through putting the props in the TA that normalizes the data. The only difference in the 3 data sources is that the data source that I cant get to work is there is a space in the logs before its breaks. The regex that I have used for both other data sources is the same one that I I using just with a space prior to it. Not working though.

[gcusello · Accepted solution]

Hi @boknows 

please try this:

[host_override]
DEST_KEY = MetaData:Host
REGEX = ^\s*([^\s]+)
FORMAT = host::$1

to manage the data sources with the space at the beginning of the events.

and, as suggested by @PickleRick , change the name of the transformation.

Ciao.

Giuseppe

[boknows]

Changing the name made it work. I had the same class names in the transforms that had different regex. I appreciate the assistance.