Log Parsing is not working

sivaranjiniG

i have a standalone splunk machine there i am monitoring a airwatch sample log

Nov 13 20:48:19 AirWatch AirWatch Syslog Details are as follows Event Type: ConsoleEvent: ApplicationPublishedUser: xxx.yyyEvent Source: ServerEvent Module: AppsEvent Category: ApplicationsEvent Data: LoginSessionID=;Application=xxx;ApplicationType=Internal;ApplicationVersion=0.95.251112.0;ApplicationUUID=xxx-35e8-40a3-b7ec-e3e28261002d;ApplicationBundle=com.xxx.mersal Event Timestamp: Nov 13 20:48:18
Nov 13 20:48:19 AirWatch AirWatch Syslog Details are as follows Event Type: DeviceEvent: InstallApplicationRequestedUser: xxx.yyyEvent Source: ServerEvent Module: DashboardEvent Category: CommandEvent Data: ApplicationType=Internal;Application=xxx;ApplicationVersion=CU2407.95.251112;ApplicationUUID=xxx-35e8-40a3-b7ec-e3e28261002d Event Timestamp: Nov 13 20:48:18
Nov 13 20:48:19 AirWatch AirWatch Syslog Details are as follows Event Type: DeviceEvent: InstallApplicationRequestedUser: xxx.yyyEvent Source: ServerEvent Module: DashboardEvent Category: CommandEvent Data: ApplicationType=Internal;Application=xxx;ApplicationVersion=CU2407.95.251112;ApplicationUUID=xxx-35e8-40a3-b7ec-e3e28261002d Event Timestamp: Nov 13 20:48:18
Nov 13 20:48:19 AirWatch AirWatch Syslog Details are as follows Event Type: DeviceEvent: InstallApplicationRequestedUser: xxx.yyyEvent Source: ServerEvent Module: DashboardEvent Category: CommandEvent Data: ApplicationType=Internal;Application=xxx;ApplicationVersion=CU2407.95.251112;ApplicationUUID=xxx-35e8-40a3-b7ec-e3e28261002d Event Timestamp: Nov 13 20:48:18
Nov 13 20:48:19 AirWatch AirWatch Syslog Details are as follows Event Type: DeviceEvent: InstallApplicationRequestedUser: xxx.yyyEvent Source≈: ServerEvent Module: DashboardEvent Category: CommandEvent Data: ApplicationType=Internal;Application=xxx;ApplicationVersion=CU2407.95.251112;ApplicationUUID=xxx-35e8-40a3-b7ec-e3e28261002d Event Timestamp: Nov 13 20:48:18
Nov 13 20:48:19 AirWatch AirWatch Syslog Details are as follows Event Type: DeviceEvent: InstallApplicationRequestedUser: xxx.yyyEvent Source: ServerEvent Module: DashboardEvent Category: CommandEvent Data: ApplicationType=Internal;Application=xxx;ApplicationVersion=CU2407.95.251112;ApplicationUUID=xxx-35e8-40a3-b7ec-e3e28261002d Event Timestamp: Nov 13 20:48:18
Nov 13 20:48:19 AirWatch AirWatch Syslog Details are as follows Event Type: DeviceEvent: InstallApplicationRequestedUser: xxx.yyyEvent Source: ServerEvent Module: DashboardEvent Category: CommandEvent Data: ApplicationType=Internal;Application=xxx;ApplicationVersion=CU2407.95.251112;ApplicationUUID=xxx-35e8-40a3-b7ec-e3e28261002d Event Timestamp: Nov 13 20:48:18

Inputs.conf

[monitor:///Applications/Splunk/etc/apps/Data_APP/Data/dummyfile.log]
disabled = false
host = hostname
index = omnissa_idx
sourcetype = omnissa:airwatch:syslog

props.conf

[omnissa:airwatch:syslog] 
SHOULD_LINEMERGE = false 
TIME_PREFIX = ^ 
MAX_TIMESTAMP_LOOKAHEAD = 100 
REPORT-main = airwatch_main_fields, airwatch_kv_fields

Transforms.conf

[airwatch_main_fields] 
REGEX = Event Type:\s*(?<event_type>[^:]+):\s*(?<event_action>.*?)(?=User:)User:\s*(?<user>.*?)(?=Event Source:)Event Source:\s*(?<event_source>.*?)(?=Event Module:)Event Module:\s*(?<event_module>.*?)(?=Event Category:)Event Category:\s*(?<event_category>.*?)(?=Event Data:)Event Data:\s*(?<event_data>.*?)(?=Event Timestamp:)Event Timestamp:\s*(?<event_timestamp>.*) 
FORMAT = 

[airwatch_kv_fields] 
REGEX = (\w+)=([^;]+) 
FORMAT = $1::$2

the same regex is working when i apply it using rex field=_raw <regex> but its not working when i put it in the transforms

So when i use this configuration, i dont see any fields getting extracted..

But i am not sure how these below fields are extracted automatically

even with below props those 4 fields are extracted

[omnissa:duplicate]

SHOULD_LINEMERGE = false

SK99

@sivaranjiniG , have you defined props and transforms under the search app context? Or if you created a new app, did you check the permission for app where you have defined props and transforms? can they be used in app context search or globally? (This can be validated in Splunk by going to Manage Apps (looking for sharing attribute) and searching for the custom app (if that's the case))

I was curious to know, have your tried removing "FORMAT = " (by default it should be considered an empty string)

gcusello

Hi @sivaranjiniG ,

Splunk automatically extracts all the pairs field=value, for this reason it extracts

ApplicationType=Internal;
Application=xxx;
ApplicationVersion=CU2407.95.251112;
ApplicationUUID=xxx-35e8-40a3-b7ec-e3e28261002d

What is your issue: have you the other fields that you configured in props.conf and transforms.conf?

Ciao.

Giuseppe

sivaranjiniG

The issues is other fields are not getting extracted with the regex.

Like i mentioned in my question, when i use the regex expression in search query with rex command it works i see the fields getting created but when i use the same expression in transforms its not getting extracted

gcusello

Hi @sivaranjiniG ,

probably the sequence of information isn't the same in all the events, so putting all the extractions in the same regex, you didn't match the most events.

Try to create extractions focused on the fields you really need and create those extraction using a regex for each one, so e.g.:

| rex "Event Type: (?<Event_Type>\w+)\s*Event:"

Ciao.

Giuseppe