Getting Data In

props & transforms not taking effect

ltrand
Contributor

I am in desperate need to figure out what I'm doing wrong with this props config. Currently I am bringing in logs via syslog-ng that then get written to disk on a heavy forwarder. I push several configs to the HF's, in one of them I've specified the following:

App: global_hfs_props/local/

props.conf:

[source::/syslog/AIRWATCH/*/*.log]
LINE_BREAKER = (|)
TRANSFORMS-airwatch= airwatch_parsing

transforms.conf:

[airwatch_parsing]
# Attempt to grab all fields in one regex
REGEX = ((Event Type:)\s)(?<Event_Type>([^ ]*))(\s(Event:)\s)(?<Event>([^ ]*))(\s(User:)\s)(?<User>([^ ]*))(\s(Event Source:)\s)(?<Event_Source>([^ ]*))(\s(Event Module:)\s)(?<Event_Module>([^ ]*))(\s(Event Category:)\s)(?<Event_Category([^ ]*))(\s(Event Data:)\s)(?\s)(?<Event_Data>([^ ]*))

When on the HF I run the following command:

/opt/splunk $ bin/splunk test sourcetype /syslog/AIRWATCH/IPADDR/IPADDR.log
Using logging configuration at /opt/splunk/etc/log-cmdline.cfg.
PROPERTIES OF /syslog/AIRWATCH/IPADDR/IPADDR.log
PropertiesMap: {ANNOTATE_PUNCT -> 'True' BREAK_ONLY_BEFORE -> '' BREAK_ONLY_BEFORE_DATE -> 'True' CHARSET -> 'UTF-8' DATETIME_CONFIG -> 'CURRENT' HEADER_MODE -> '' LEARN_SOURCETYPE -> 'true' LINE_BREAKER_LOOKBEHIND -> '100' MAX_DAYS_AGO -> '2000' MAX_DAYS_HENCE -> '2' MAX_DIFF_SECS_AGO -> '3600' MAX_DIFF_SECS_HENCE -> '604800' MAX_EVENTS -> '256' MAX_TIMESTAMP_LOOKAHEAD -> '32625' MUST_BREAK_AFTER -> '' MUST_NOT_BREAK_AFTER -> '' MUST_NOT_BREAK_BEFORE -> '' SEGMENTATION -> 'indexing' SEGMENTATION-all -> 'full' SEGMENTATION-inner -> 'inner' SEGMENTATION-outer -> 'outer' SEGMENTATION-raw -> 'none' SEGMENTATION-standard -> 'standard' SHOULD_LINEMERGE -> 'False' TRANSFORMS -> '' TRANSFORMS-force-sourcetype_for_cisco_devices -> 'force_sourcetype_for_cisco_pix, force_sourcetype_for_cisco_asa, force_sourcetype_for_cisco_wap, force_sourcetype_for_cisco_fwsm, force_sourcetype_for_cisco_acs, force_sourcetype_for_cisco_ios, force_sourcetype_for_cisco_catchall' TRUNCATE -> '10000' detect_trailing_nulls -> 'false' is_valid -> 'True' maxDist -> '500' sourcetype -> 'IPADDR'}

I know test has been deprecated, but I don't know a better command to check to see if props or transforms are being applied to a specific file. Am I perhaps doing this wrong? I see that it's getting transforms from a props that doesn't specify a source to apply the transforms to, but I don't see when I specify the source that it gets applied.

I'm hoping that this gets me the fields I need & I want to have the "|" used as a end of message character since the logs sometimes run together.

-Thanks!

0 Karma
1 Solution

lguinn2
Legend

Okay, you are sending data to a heavy forwarder (HF) and you expect it to parse your data. So far, so good. BUT

You shouldn't be doing field extractions at index time. Use search time field extractions. Search time field extraction specifications belong on the indexer (or the search head, if you have one).

Next, you can do the field extractions as you have it by just using props.conf:

[source::/syslog/AIRWATCH/*/*.log]
EXTRACT-everything = ((Event Type:)\s)(?<Event_Type>([^ ]*))(\s(Event:)\s)(?<Event>([^ ]*))(\s(User:)\s)(?<User>([^ ]*))(\s(Event Source:)\s)(?<Event_Source>([^ ]*))(\s(Event Module:)\s)(?<Event_Module>([^ ]*))(\s(Event Category:)\s)(?<Event_Category([^ ]*))(\s(Event Data:)\s)(?\s)(?<Event_Data>([^ ]*))

But I think there is an easier way to do it! On the indexer, put the following in props.conf

[source::/syslog/AIRWATCH/*/*.log]
REPORT-aw=extract_airwatch

and in transforms.conf

[extract_airwatch]
DELIMS=":"
FORMAT = $1::$2

Now you don't have to write such a complicated regular expression. Finally, the LINE_BREAKER needs to stay on the HF.

Finally, when you change props.conf and transforms.conf on a HF, you need to restart it for the changes to take effect. Search time field extractions do not require a restart. However, it is a good idea to explicitly request a reload of the conf files by doing this

http://yourservername:8000/debug/refresh

assuming that your SplunkWeb UI is running on port 8000...

View solution in original post

lguinn2
Legend

Okay, you are sending data to a heavy forwarder (HF) and you expect it to parse your data. So far, so good. BUT

You shouldn't be doing field extractions at index time. Use search time field extractions. Search time field extraction specifications belong on the indexer (or the search head, if you have one).

Next, you can do the field extractions as you have it by just using props.conf:

[source::/syslog/AIRWATCH/*/*.log]
EXTRACT-everything = ((Event Type:)\s)(?<Event_Type>([^ ]*))(\s(Event:)\s)(?<Event>([^ ]*))(\s(User:)\s)(?<User>([^ ]*))(\s(Event Source:)\s)(?<Event_Source>([^ ]*))(\s(Event Module:)\s)(?<Event_Module>([^ ]*))(\s(Event Category:)\s)(?<Event_Category([^ ]*))(\s(Event Data:)\s)(?\s)(?<Event_Data>([^ ]*))

But I think there is an easier way to do it! On the indexer, put the following in props.conf

[source::/syslog/AIRWATCH/*/*.log]
REPORT-aw=extract_airwatch

and in transforms.conf

[extract_airwatch]
DELIMS=":"
FORMAT = $1::$2

Now you don't have to write such a complicated regular expression. Finally, the LINE_BREAKER needs to stay on the HF.

Finally, when you change props.conf and transforms.conf on a HF, you need to restart it for the changes to take effect. Search time field extractions do not require a restart. However, it is a good idea to explicitly request a reload of the conf files by doing this

http://yourservername:8000/debug/refresh

assuming that your SplunkWeb UI is running on port 8000...

ltrand
Contributor

Thanks for the help! Moving it to the search head absolutely soved my problem. The $1::$2 breaks because of the multiword key values, so I get weird entries. So I stuck with the ugly regex I wrote.

The only issue I am still having though is that the line breaker is still not applying, but I am trying to work through it.

0 Karma

nychawk
Communicator

ltrand;

I ran across this posting while searching for AirWatch TA's and/or apps.

Can you share what you've done?

Thank you in advance.

0 Karma

j0shrice
Path Finder

Also looking for AirWatch TA with CIM mapping

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...