Splunk_ta_windows: Why are Index, source, and sourceType missing from my search?

[yr]

Hi

I have installed splunk_ta_windows using deployment server using UF on windows clients and everything is fine.  I created index and pointed in inputs.conf and all looks good. 

i also search data fine but some sources and sourcetypes are missing when i input the query.

[gcusello]

Hi @yr,

which ones are missing?

Are you sure to tha those logs?

Are they missing always or only  sometimes?

Ciao.

Giuseppe

[yr]

Hi Gcusello

Let me give you little more detail.

a. we use custom index 

b. we deployed splunk_ta_windows using deloyment server

c. we have modify inputs.conf on deployment server

d. inputs.conf has index=<our index name> in each stanza

e. we used default inputs.conf and changed the index to our 

now we see windows log data if we use in search and specify index name but if we ud thru sourcetype than data does not search, also we see only few sourcetypes.

your help is appreciated

[gcusello]

Hi @yr,

as I already asked:

• which sourcetypes are missing?

• Are they missing always or only  sometimes?

• did you checked that in the missing sourcetypes you have disabled=0? because by default all the inputs are disabled.

Ciao.

Giuseppe

[yr]

I only get sourcetype wineventlog but when i add to security or application or system than does not search  any.

i have disabled=0 in all inputs.conf stanza

thank you for your help

[gcusello]

Hi @yr,

as @PickleRick sais, you have only one sourcetype: WinEventLog (or XmlWinEventLog if you're ingesting them as XML), it was chenged: before you have wineventlog:Security.

You can distinguish logs based on source.

Ciao.

Giuseppe

[yr]

Hello Friends,

here is my snipped of inputs.conf tog et you an idea or may be mistaked on my end ??

again thank you for your help

------------------

This is my snip of inputs.conf

# cat inputs.conf
[perfmon://CPU]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
mode = single
object = Processor
_meta = os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:CPU
disabled = 0
index=uat

[perfmon://Memory]
counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes
interval = 30
mode = single
object = Memory
_meta = os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:Memory
disabled = 0
index=uat

[WinEventLog://Application]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 10
renderXml=true
sourcetype = WinEventLog:Application
index=uat

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 10
renderXml=true

blacklist1 = EventCode="(4662|566)" Message="Object Type:(?!\s*groupPolicyContainer)"
blacklist2 = EventCode="(4656|4670|4663|4703|4658|4688)" Message="Account Name:(\W+\w+$)"
blacklist3 = EventCode="4624" Message="An account was successfully logged on"
blacklist4 = EventCode="(4688|4689)" Message="%SplunkUniversalForwarder%"
blacklist5 = EventCode="6278" Message="Network Policy Server granted full access to a user because the host met the defined health policy."

#whitelist = 1101, 1104, 4616, 4657, 4697
sourcetype = WinEventLog:Security
index=uat

[WinEventLog://System]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 10
renderXml=true
sourcetype = WinEventLog:System
index=uat

[WinEventLog://Setup]
checkpointInterval = 10
current_only = 0
disabled = 0
start_from = oldest
renderXml=true
sourcetype = WinEventLog:Setup
index=uat

[monitor://$SPLUNK_HOME\var\log\splunk\*.log*]
sourcetype = uf
dissabled = 0
index = _internal

[gcusello]

Hi @yr,

as I said, I don't know why, since some time Splunk changed approach using the same sourcetype for all WinEventLogs distinguishing them by source.

I saw that you forced sourcetype in each inputs stanza, in this way you should be sure to have the sourcetype you want, in this way you shouldn't miss any log.

I disagree with the last input stanza: Splunk logs are ingested in another input stanza and this is a duplication, in addition you forced sourcetype, in this way you're losing some internal monitoring features (e.g. Monitoring Console).

Ciao.

Giuseppe

[PickleRick]

Events from EventLog are ingested with WinEventLog (or XmlWinEventLog if you're ingesting them as XML)  sourcetype. There should be no other sourcetypes. The events are distinguishable by source (not sourcetype).

[yr]

Hi

agreed but why source and sourcetype os mixed up ? it does not goes what i have mentioned in inputs.conf.

how do i fix it ?  

DC01.xxx.xxx</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>CORP\ADmaint</Data><Data Name='SubjectUserName'>ADmaint</Data><Data Name='SubjectDomainName'>CORP</Data><Data Name='SubjectLogonId'>0x1b73fc</Data><Data Name='PrivilegeList'>SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege

• host = DC01
• source = WinEventLog:Security
• sourcetype = WinEventLog

this source and sourcetype are mixed and not according to inputs.conf

[PickleRick]

It needs a longer explanation. I believe long time ago the things were as you tried to set them up - the events were distinguishable by sourcetypes. But since there is no actual need to treat them as separate sourcetypes (sourcetype defines how the data is processed - ingested and parsed) because the data is in the same format regardless of which particular EventLog channel it came from and having separate sourcetypes for each EventLog  channel would mean that you'd need to define settings for each new channel you ingest (and you can pull any of the channels you see in your EventLog!).

So there was a shift in the approach to windows events (and it happened looooong time ago). And in order to accomodate all those forwarders installed long time ago and still working with old defaults (configured as you tried to set it up), there are transforms in TA_windows which "normalize" the sources and sourcetypes.

This is from default/transforms.conf:

## Setting generic sourcetype and unique source
[ta-windows-fix-classic-source]
DEST_KEY = MetaData:Source
REGEX = (?m)^LogName=(.+?)\s*$
FORMAT = source::WinEventLog:$1

[ta-windows-fix-xml-source]
DEST_KEY = MetaData:Source
REGEX = <Channel>(.+?)<\/Channel>.*
FORMAT = source::XmlWinEventLog:$1

[ta-windows-fix-sourcetype]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY = MetaData:Sourcetype
REGEX = sourcetype::([^:]*)
FORMAT = sourcetype::$1

Even if you explicitly configure your inputs to provide source and sourcetype "old style" the transforms will get invoked during indexing an will overwrite the metadata fields to the "new style".

So all windows EventLog-sourced events are of either WinEventLog sourcetype or XmlWinEvenLog one (depending on whether you ingest them as "classic" or XML).

[yr]

Hi PickleRick,

Thank you for good research and shared the knowledge. 

How can i fix this issue if you can please share more tips ?

thanks

[PickleRick]

But why would you want to fix that? Just search by source if you want evetns from one event log channel.

[yr]

Hi PickleRick,

Agreed. 

Than do i remove the sourcetype= statement from stanza in inputs.conf  ? ( becuase it is over written any way )

please share your thoughts.

also 

do i create seperate index for metrics mentioned in my inputs.conf of keep with eventtype index ?

here is snipped of inputs.conf

------------------------------- inputs.conf ----------

#

###### OS Logs ######
#
[WinEventLog://Application]
disabled = false
start_from = oldest
current_only = 0
checkpointInterval = 5
renderXml=true
index = winos

-----

------

-----

#
###### Host monitoring ######
#

[WinHostMon://Computer]
interval = 600
disabled = false
type = Computer
index = winos

[WinHostMon://Process]
interval = 600
disabled = false
type = Process
index = winos

-----

-----

#
###### Win Registry Monitoring
#

[WinRegMon://default]
disabled = false
hive = .*
proc = .*
type = rename|set|delete|create
index = winos

-------

------

#
# perfmonance Monitoring
#

###### Splunk 5.0+ Performance Counters ######
## CPU
[perfmon://CPU]
counters = % Processor Time; % User Time; % Privileged Time; Interrupts/sec; % DPC Time; % Interrupt Time; DPCs Queued/sec; DPC Rate; % Idle Time; % C1 Time; % C2 Time; % C3 Time; C1 Transitions/sec; C2 Transitions/sec; C3 Transitions/sec
instances = *
interval = 30
mode = single
object = Processor
_meta = os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:CPU
disabled = 0
index = ?????

Please share your expertise

thanks

[PickleRick]

And what sourcetype would you expect? And do you have inputs producing events with those sourcetypes?

[yr]

yes i see some sourcetypes when i do only search using index= and in event i see some sourcetypes.

[PickleRick]

Again - what sourcetypes did you expect?