topic Re: Index, source and sourceType for splunk_ta_windows in Getting Data In

Splunk_ta_windows: Why are Index, source, and sourceType missing from my search?

yr — Mon, 28 Aug 2023 18:30:54 GMT

I have installed splunk_ta_windows using deployment server using UF on windows clients and everything is fine. I created index and pointed in inputs.conf and all looks good.

i also search data fine but some sources and sourcetypes are missing when i input the query.

Re: Index, source and sourceType for splunk_ta_windows

gcusello — Sun, 27 Aug 2023 08:36:19 GMT

Hi @yr,

which ones are missing?

Are you sure to tha those logs?

Are they missing always or only sometimes?

Ciao.

Giuseppe

Re: Index, source and sourceType for splunk_ta_windows

yr — Sun, 27 Aug 2023 15:27:58 GMT

Hi Gcusello

Let me give you little more detail.

a. we use custom index

b. we deployed splunk_ta_windows using deloyment server

c. we have modify inputs.conf on deployment server

d. inputs.conf has index=<our index name> in each stanza

e. we used default inputs.conf and changed the index to our

now we see windows log data if we use in search and specify index name but if we ud thru sourcetype than data does not search, also we see only few sourcetypes.

your help is appreciated

Re: Index, source and sourceType for splunk_ta_windows

PickleRick — Sun, 27 Aug 2023 18:50:49 GMT

And what sourcetype would you expect? And do you have inputs producing events with those sourcetypes?

Re: Index, source and sourceType for splunk_ta_windows

yr — Sun, 27 Aug 2023 19:38:11 GMT

yes i see some sourcetypes when i do only search using index= and in event i see some sourcetypes.

Re: Index, source and sourceType for splunk_ta_windows

PickleRick — Sun, 27 Aug 2023 21:55:37 GMT

Again - what sourcetypes did you expect?

Re: Index, source and sourceType for splunk_ta_windows

gcusello — Mon, 28 Aug 2023 07:52:13 GMT

Hi @yr,

as I already asked:

which sourcetypes are missing?
Are they missing always or only sometimes?
did you checked that in the missing sourcetypes you have disabled=0? because by default all the inputs are disabled.

Ciao.

Giuseppe

Re: Index, source and sourceType for splunk_ta_windows

yr — Mon, 28 Aug 2023 14:35:29 GMT

I only get sourcetype wineventlog but when i add to security or application or system than does not search any.

i have disabled=0 in all inputs.conf stanza

thank you for your help

Re: Index, source and sourceType for splunk_ta_windows

PickleRick — Mon, 28 Aug 2023 15:38:35 GMT

Events from EventLog are ingested with WinEventLog (or XmlWinEventLog if you're ingesting them as XML) sourcetype. There should be no other sourcetypes. The events are distinguishable by source (not sourcetype).

Re: Index, source and sourceType for splunk_ta_windows

gcusello — Mon, 28 Aug 2023 15:42:44 GMT

Hi @yr,

as @PickleRick sais, you have only one sourcetype: WinEventLog (or XmlWinEventLog if you're ingesting them as XML), it was chenged: before you have wineventlog:Security.

You can distinguish logs based on source.

Ciao.

Giuseppe

Re: Index, source and sourceType for splunk_ta_windows

yr — Mon, 28 Aug 2023 16:02:16 GMT

Hello Friends,

here is my snipped of inputs.conf tog et you an idea or may be mistaked on my end ??

again thank you for your help

------------------

This is my snip of inputs.conf

# cat inputs.conf
[perfmon://CPU]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
mode = single
object = Processor
_meta = os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:CPU
disabled = 0
index=uat

[perfmon://Memory]
counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes
interval = 30
mode = single
object = Memory
_meta = os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:Memory
disabled = 0
index=uat

[WinEventLog://Application]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 10
renderXml=true
sourcetype = WinEventLog:Application
index=uat

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 10
renderXml=true

blacklist1 = EventCode="(4662|566)" Message="Object Type:(?!\s*groupPolicyContainer)"
blacklist2 = EventCode="(4656|4670|4663|4703|4658|4688)" Message="Account Name:(\W+\w+$)"
blacklist3 = EventCode="4624" Message="An account was successfully logged on"
blacklist4 = EventCode="(4688|4689)" Message="%SplunkUniversalForwarder%"
blacklist5 = EventCode="6278" Message="Network Policy Server granted full access to a user because the host met the defined health policy."

#whitelist = 1101, 1104, 4616, 4657, 4697
sourcetype = WinEventLog:Security
index=uat

[WinEventLog://System]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 10
renderXml=true
sourcetype = WinEventLog:System
index=uat

[WinEventLog://Setup]
checkpointInterval = 10
current_only = 0
disabled = 0
start_from = oldest
renderXml=true
sourcetype = WinEventLog:Setup
index=uat

[monitor://$SPLUNK_HOME\var\log\splunk\*.log*]
sourcetype = uf
dissabled = 0
index = _internal

Re: Index, source and sourceType for splunk_ta_windows

yr — Mon, 28 Aug 2023 22:50:46 GMT

agreed but why source and sourcetype os mixed up ? it does not goes what i have mentioned in inputs.conf.

how do i fix it ?

DC01.xxx.xxx</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>CORP\ADmaint</Data><Data Name='SubjectUserName'>ADmaint</Data><Data Name='SubjectDomainName'>CORP</Data><Data Name='SubjectLogonId'>0x1b73fc</Data><Data Name='PrivilegeList'>SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege

host = DC01
source = WinEventLog:Security
sourcetype = WinEventLog

this source and sourcetype are mixed and not according to inputs.conf

Re: Index, source and sourceType for splunk_ta_windows

gcusello — Tue, 29 Aug 2023 05:55:05 GMT

Hi @yr,

as I said, I don't know why, since some time Splunk changed approach using the same sourcetype for all WinEventLogs distinguishing them by source.

I saw that you forced sourcetype in each inputs stanza, in this way you should be sure to have the sourcetype you want, in this way you shouldn't miss any log.

I disagree with the last input stanza: Splunk logs are ingested in another input stanza and this is a duplication, in addition you forced sourcetype, in this way you're losing some internal monitoring features (e.g. Monitoring Console).

Ciao.

Giuseppe

Re: Index, source and sourceType for splunk_ta_windows

PickleRick — Tue, 29 Aug 2023 08:14:01 GMT

It needs a longer explanation. I believe long time ago the things were as you tried to set them up - the events were distinguishable by sourcetypes. But since there is no actual need to treat them as separate sourcetypes (sourcetype defines how the data is processed - ingested and parsed) because the data is in the same format regardless of which particular EventLog channel it came from and having separate sourcetypes for each EventLog channel would mean that you'd need to define settings for each new channel you ingest (and you can pull any of the channels you see in your EventLog!).

So there was a shift in the approach to windows events (and it happened looooong time ago). And in order to accomodate all those forwarders installed long time ago and still working with old defaults (configured as you tried to set it up), there are transforms in TA_windows which "normalize" the sources and sourcetypes.

This is from default/transforms.conf:

## Setting generic sourcetype and unique source
[ta-windows-fix-classic-source]
DEST_KEY = MetaData:Source
REGEX = (?m)^LogName=(.+?)\s*$
FORMAT = source::WinEventLog:$1

[ta-windows-fix-xml-source]
DEST_KEY = MetaData:Source
REGEX = <Channel>(.+?)<\/Channel>.*
FORMAT = source::XmlWinEventLog:$1

[ta-windows-fix-sourcetype]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY = MetaData:Sourcetype
REGEX = sourcetype::([^:]*)
FORMAT = sourcetype::$1

Even if you explicitly configure your inputs to provide source and sourcetype "old style" the transforms will get invoked during indexing an will overwrite the metadata fields to the "new style".

So all windows EventLog-sourced events are of either WinEventLog sourcetype or XmlWinEvenLog one (depending on whether you ingest them as "classic" or XML).

Re: Index, source and sourceType for splunk_ta_windows

yr — Tue, 29 Aug 2023 16:47:49 GMT

Hi PickleRick,

Thank you for good research and shared the knowledge.

How can i fix this issue if you can please share more tips ?

thanks

Re: Index, source and sourceType for splunk_ta_windows

PickleRick — Tue, 29 Aug 2023 17:42:53 GMT

But why would you want to fix that? Just search by source if you want evetns from one event log channel.

Re: Index, source and sourceType for splunk_ta_windows

yr — Tue, 29 Aug 2023 22:10:49 GMT

Hi PickleRick,

Agreed.

Than do i remove the sourcetype= statement from stanza in inputs.conf ? ( becuase it is over written any way )

please share your thoughts.

also

do i create seperate index for metrics mentioned in my inputs.conf of keep with eventtype index ?

here is snipped of inputs.conf

------------------------------- inputs.conf ----------

###### OS Logs ######
#
[WinEventLog://Application]
disabled = false
start_from = oldest
current_only = 0
checkpointInterval = 5
renderXml=true
index = winos

-----

------

-----

#
###### Host monitoring ######
#

[WinHostMon://Computer]
interval = 600
disabled = false
type = Computer
index = winos

[WinHostMon://Process]
interval = 600
disabled = false
type = Process
index = winos

-----

#
###### Win Registry Monitoring
#

[WinRegMon://default]
disabled = false
hive = .*
proc = .*
type = rename|set|delete|create
index = winos

-------

------

#
# perfmonance Monitoring
#

###### Splunk 5.0+ Performance Counters ######
## CPU
[perfmon://CPU]
counters = % Processor Time; % User Time; % Privileged Time; Interrupts/sec; % DPC Time; % Interrupt Time; DPCs Queued/sec; DPC Rate; % Idle Time; % C1 Time; % C2 Time; % C3 Time; C1 Transitions/sec; C2 Transitions/sec; C3 Transitions/sec
instances = *
interval = 30
mode = single
object = Processor
_meta = os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:CPU
disabled = 0
index = ?????

Please share your expertise

thanks