props & transforms not taking effect

[ltrand]

I am in desperate need to figure out what I'm doing wrong with this props config. Currently I am bringing in logs via syslog-ng that then get written to disk on a heavy forwarder. I push several configs to the HF's, in one of them I've specified the following:

App: global_hfs_props/local/

props.conf:

[source::/syslog/AIRWATCH/*/*.log]
LINE_BREAKER = (|)
TRANSFORMS-airwatch= airwatch_parsing

transforms.conf:

[airwatch_parsing]
# Attempt to grab all fields in one regex
REGEX = ((Event Type:)\s)(?<Event_Type>([^ ]*))(\s(Event:)\s)(?<Event>([^ ]*))(\s(User:)\s)(?<User>([^ ]*))(\s(Event Source:)\s)(?<Event_Source>([^ ]*))(\s(Event Module:)\s)(?<Event_Module>([^ ]*))(\s(Event Category:)\s)(?<Event_Category([^ ]*))(\s(Event Data:)\s)(?\s)(?<Event_Data>([^ ]*))

When on the HF I run the following command:

/opt/splunk $ bin/splunk test sourcetype /syslog/AIRWATCH/IPADDR/IPADDR.log
Using logging configuration at /opt/splunk/etc/log-cmdline.cfg.
PROPERTIES OF /syslog/AIRWATCH/IPADDR/IPADDR.log
PropertiesMap: {ANNOTATE_PUNCT -> 'True' BREAK_ONLY_BEFORE -> '' BREAK_ONLY_BEFORE_DATE -> 'True' CHARSET -> 'UTF-8' DATETIME_CONFIG -> 'CURRENT' HEADER_MODE -> '' LEARN_SOURCETYPE -> 'true' LINE_BREAKER_LOOKBEHIND -> '100' MAX_DAYS_AGO -> '2000' MAX_DAYS_HENCE -> '2' MAX_DIFF_SECS_AGO -> '3600' MAX_DIFF_SECS_HENCE -> '604800' MAX_EVENTS -> '256' MAX_TIMESTAMP_LOOKAHEAD -> '32625' MUST_BREAK_AFTER -> '' MUST_NOT_BREAK_AFTER -> '' MUST_NOT_BREAK_BEFORE -> '' SEGMENTATION -> 'indexing' SEGMENTATION-all -> 'full' SEGMENTATION-inner -> 'inner' SEGMENTATION-outer -> 'outer' SEGMENTATION-raw -> 'none' SEGMENTATION-standard -> 'standard' SHOULD_LINEMERGE -> 'False' TRANSFORMS -> '' TRANSFORMS-force-sourcetype_for_cisco_devices -> 'force_sourcetype_for_cisco_pix, force_sourcetype_for_cisco_asa, force_sourcetype_for_cisco_wap, force_sourcetype_for_cisco_fwsm, force_sourcetype_for_cisco_acs, force_sourcetype_for_cisco_ios, force_sourcetype_for_cisco_catchall' TRUNCATE -> '10000' detect_trailing_nulls -> 'false' is_valid -> 'True' maxDist -> '500' sourcetype -> 'IPADDR'}

I know test has been deprecated, but I don't know a better command to check to see if props or transforms are being applied to a specific file. Am I perhaps doing this wrong? I see that it's getting transforms from a props that doesn't specify a source to apply the transforms to, but I don't see when I specify the source that it gets applied.

I'm hoping that this gets me the fields I need & I want to have the "|" used as a end of message character since the logs sometimes run together.

-Thanks!

[lguinn2 · Accepted solution]

Okay, you are sending data to a heavy forwarder (HF) and you expect it to parse your data. So far, so good. BUT

You shouldn't be doing field extractions at index time. Use search time field extractions. Search time field extraction specifications belong on the indexer (or the search head, if you have one).

Next, you can do the field extractions as you have it by just using props.conf:

[source::/syslog/AIRWATCH/*/*.log]
EXTRACT-everything = ((Event Type:)\s)(?<Event_Type>([^ ]*))(\s(Event:)\s)(?<Event>([^ ]*))(\s(User:)\s)(?<User>([^ ]*))(\s(Event Source:)\s)(?<Event_Source>([^ ]*))(\s(Event Module:)\s)(?<Event_Module>([^ ]*))(\s(Event Category:)\s)(?<Event_Category([^ ]*))(\s(Event Data:)\s)(?\s)(?<Event_Data>([^ ]*))

But I think there is an easier way to do it! On the indexer, put the following in props.conf

[source::/syslog/AIRWATCH/*/*.log]
REPORT-aw=extract_airwatch

and in transforms.conf

[extract_airwatch]
DELIMS=":"
FORMAT = $1::$2

Now you don't have to write such a complicated regular expression. Finally, the LINE_BREAKER needs to stay on the HF.

Finally, when you change props.conf and transforms.conf on a HF, you need to restart it for the changes to take effect. Search time field extractions do not require a restart. However, it is a good idea to explicitly request a reload of the conf files by doing this

http://yourservername:8000/debug/refresh

assuming that your SplunkWeb UI is running on port 8000...

[ltrand]

Thanks for the help! Moving it to the search head absolutely soved my problem. The $1::$2 breaks because of the multiword key values, so I get weird entries. So I stuck with the ugly regex I wrote.

The only issue I am still having though is that the line breaker is still not applying, but I am trying to work through it.

[nychawk]

ltrand;

I ran across this posting while searching for AirWatch TA's and/or apps.

Can you share what you've done?

Thank you in advance.

[j0shrice]

Also looking for AirWatch TA with CIM mapping