Why are some default fields not being extracted for data coming in via TCP syslog with my current props and transforms.conf?

[dannestor]

I have data incoming via TCP syslog. I have created the following transforms to process them:

• etc/system/local/props.conf:

  [source::tcp:1514]
  TRANSFORMS-windows = set_sourcetype_snare, set_source_wineventlog

• etc/system/local/transforms.conf:

  [set_source_wineventlog]
  REGEX = AgentDevice=WindowsLog.AgentLogFile=(.?)\s
  FORMAT = source::WinEventLog:$1
  DEST_KEY = MetaData:Source

  [set_sourcetype_snare]
  REGEX = AgentDevice=WindowsLog
  FORMAT = sourcetype::windows_snare_syslog
  DEST_KEY = MetaData:Sourcetype
  These work as expected, and the source and sourcetype are set accordingly. However, I expected that setting these two fields would also trigger some other Splunk built-in transforms. For example:

  [splunk@l1807s local]$ ~/bin/splunk btool props list windows_snare_syslog
  [windows_snare_syslog]
  ...
  TRANSFORMS = syslog-host
  and

  [splunk@l1807s local]$ cat ~/etc/apps/Splunk_TA_windows/default/props.conf
  ...

  Apply the following properties to all Windows events

  [source::(MonitorWare|NTSyslog|Snare|WinEventLog|WMI:WinEventLog)...]
  ...
  FIELDALIAS-event_id_for_windows = RecordNumber as event_id
  ...
  As far as I can tell, the default processing is not happening. I see the source and sourcetype fields as set by my transforms, however for example I don't find the field event_id, and host is incorrectly set. What am I doing wrong here, and how can I achieve the intended behaviour?

[woodcock]

Although you can change the host and sourcetype fields, all configurations inside of props.conf may reference the original/pre-overridden values if the overriding happens afterwards.

[dannestor]

Are you sure about this? This documentation page states the contrary: http://docs.splunk.com/Documentation/Splunk/6.3.1511/Admin/Wheretofindtheconfigurationfiles

[woodcock]

Yes, I am sure: I am battle-tested on this. Where in that link do you see anything to the contrary? Please post the text in a comment/followup.

[dannestor]

Another thought: in my example, isn't event-id a field extracted at search time? Shouldn't it pick up changes to source and sourcetype, which are made during indexing?

[dannestor]

Under the heading "precedence in a global context" (which is the context where indexing happens), the system local directories have the highest priority. I just realized however that this applies only to the situation where the same attribute is defined in multiple files, and says nothing about the order of evaluation of different attributes, which is what you were referring to. Do you have a documentation link where the latter is documented?

[Runals]

Is the data coming in and the props/transforms on the same Splunk instance or is there some separation?

[dannestor]

No, there isn't any separation. It's a single-machine test installation.

[richgalloway]

I recommend you take the transforms and other actions you desire from other sources/sourcetypes and copy them to your own props.conf and transforms.conf files. That will ensure they work and protect you from future changes made to the other apps.