Problem overriding host on an indexer

[dpadams]

I'm having some trouble overriding the default host assignment and am hoping for some help. I've tested out a regex with rex that looks to work correctly but it's not seeming to work from props.conf + transforms.conf.

For background, here's our setup:

• We've got a lot of custom logs.

• Each log has its original data stored in a SQL database.

• A single program on a single machine reads the rows off the SQL database and writes out the individual logs locally. Call this machine LogMachine.

• LogMachine has a copy of Splunk acting as a forwarder that sends all of the data into a remote Splunk indexer.

So, all of the logs are written locally to disk on LogMachine and then forwarded on. For policy reasons outside of my control, this part of the setup is fixed. Namely, that we've got all of the logs on one machine and forward them from there. The problem is that every event has a host of LogMachine rather than the original host machine. I've looked at the docs and several threads here and it looks like I'm meant to define a transform and declare it in props.conf.

I'm only testing out one log right now as I'm trying to get the mechanics sorted out before I work on doing the same thing for our other custom logs. Here's a snippet from a custom 'action' log:

[23/Jun/2012:01:50:06 +0000] add appuser FANGLET-AU000-0000000000 0 1336643d3082237d75191d4362fbd941 - 1.0 - 345101-VM3 SRST
[23/Jun/2012:01:51:38 +0000] add appuser FANGLET-US000-0000000000 0 9fb0638027e115dc36a313700ada3f54 - 1.1.4 - 345101-VM3 SRST
[23/Jun/2012:01:51:53 +0000] add appuser FANGLET-AU-EGGPLNT 0 d1128ee5236b17a41825832b890a8091 cdma_spyder 1.0 10 345101-VM3 SRST
[23/Jun/2012:01:52:47 +0000] add appuser FANGLET-AU000-0000000000 0 5d3ded5a3efbc9102c85e319d08c461d - 1.0 - 345101-VM3 SRST
[23/Jun/2012:06:48:04 +0000] add appuser FRINDO-UK-EGGPLNT 0 c9e9d9c86fe1592e3427592c4c4bc6a buzz 1.0 8 345999-VM4 SRST
[23/Jun/2012:06:48:20 +0000] add appuser FANGLET-AU000-0000000000 0 d0e3cc86221875df6485f28e6246bcf8 - 1.0 - 345999-VM4 SRST
[23/Jun/2012:06:48:56 +0000] add appuser FRINDO-US000-0000000000 0 459d7c547c40efa025feb0ea9fd93998 - 1.1.4 - 345999-VM4 SRST
[23/Jun/2012:06:48:57 +0000] add appuser FRINDO-US000-0000000000 0 8321965193395108fe7d85878f8c9a43 - 1.1.4 - SRST-Remote-2 SRST

Here's the transforms.conf text stanza that applies:

# transforms.conf
[action_host_override]
REGEX = (?i)^(?:[^ ]* ){10}(?P<host>[^ ]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

Here are props.conf stanzas that apply:

# props.conf
[source::.../action_log.txt]
sourcetype = action

[source::.../action_log.txt]
TRANSFORMS-action-host=action_host_override

I don't know if that needs to be in two stanzas or one - I've tried both with no change either way. Namely, the sourcetype is applied and the transform is ignored.

For good measure, here's the inputs.conf stanza for this sourcetype:

[monitor://C:\Program Files\SRTS\logs\action_log.txt]
disabled = false
index = srst
followTail = 0
sourcetype = action

I've tested that regex out with rex and it extracts the values that I expect. Namely 345101-VM3, 345999-VM4, SRST-Remote-2 and so on. Here's a rex sample set in the UI to the past 15 minutes:

sourcetype="action" | rex field=_raw "(?i)^(?:[^ ]* ){10}(?P<host>[^ ]+)"

I'm guessing that I need to change the regex for transforms.conf (?)

I originally had a SplunkUniversalForwarder running but gather from an earlier thread that I need at least a full forwarder. I installed Splunk 4.2.5 and licensed it as a forwarder. Here's the machine setup:

• Win 32
• Splunk 4.2.5 acting as a forwarder to an indexer running Splunk 4.2.2.

I do not control the indexing machine and the person managing it is always short on time. So, I'd prefer to run the transformation before forwarding.

I'd be extremely grateful for help resolving this problem!

Thanks,

[dpadams]

I've got a few lines more than a comment will accept so I'm answering my own question.

Thanks for the quick answer but it's still not working.

• Yes, this is a 4.2.5 heavy forwarder. I installed Splunk, set the license to Forwarder and then moved over and/or tweaked config files.

• Understood that the fix only applies to newly indexed data.

• The slashes in the props.conf have always worked in the past. Are you saying they need to be like so?

  [source::...\action_log.txt]
  
  sourcetype = action

• I changed the regex to this:

  REGEX = (?i)^(?:[^ ]* ){10}([^ ]+)

• I again tried combining the sourcetype and TRANSFORMS statements into one stanza and splitting them into two without any visible change. The two versions:

  [source::.../action_log.txt]
  sourcetype = action

  [source::.../action_log.txt]
  TRANSFORMS-action-host=action_host_override

versus

[source::.../action_log.txt]
sourcetype = action
TRANSFORMS-action-host=action_host_override

I do stop and start Splunk each time I modify the config files.

Can you think what else I might be missing here?

(I didn't manage to get the formatting all right above but the text should be accurate.)

[dpadams]

More odd details:

splunk list-forward-server

returns "Active fowrads: None" but "Configured but inactive forwards" lists the foward that I'm using. Events are forwarding without any obvious problem.

It's a bit frustrating to try and trouble-shoot this as it's not obvious what exactly Splunk is doing on this machine.

Good: It's applying the custom sourcetype and forwarding events.

Bad: It's ignoring the host overrirde and the CLI results for the system's state don't seem to reflect reality (?)

I'd be grateful for any suggestions.

[dpadams]

After rechecking all of the docs and config files carefully I've found nothing obviously wrong. After seeing other people with similar problems I ran

splunk display app

This lists the SplunkForwarder as unconfigured, disabled, and invisible. I've run

splunk enable app SplunkForwarder

The SplunkForwarder is now listd as unconfigured, enabled, and invisible. I've got config files in place and the Web management GUI lists the forwarding rule and says that it's enabled.

Could my problem be that somehow the heavy forwarder features aren't active? That would explain why my rule doesn't run.

[Drainy]

Remove the ?P from the regex as this is not an EXTRACT- plus the host field is defined within your format. Also you have changed direction of slashes for your props against your inputs.

Once it kicks into life, bear in mind that it will only take affect on newly indexed data. Just to be clear, you are using the Splunk 4.2.5 as a heavy forwarder correct? (Thats what is sounds like from your descriptions)