Re: Why is no data coming into the summary index b...

pauloyelakin · ‎02-11-2018

I just installed splunk into the lab environment. I installed a deployment server and a universal forwarder to a win10 machine (risenlab1). I couldn't get any data so I turned off all firewall. then all of a sudden 272 event came out.

The only problem was no more data was coming in. Then I stumbled upon this command:

index=_internal | stats count by host

this search showed that data is coming in to the internal index... I'm not quite sure what an internal index is and what this means... and I also noticed that data is being collected from another host machine (risen300) to the internal index. I did not install a universal forwarder to that host so how is data being forwarded from it??

below are my conf files

**

indexes.conf

**

    [msad]
    homePath = $SPLUNK_DB/msad/db
    coldPath = $SPLUNK_DB/msad/colddb
    thawedPath = $SPLUNK_DB/msad/thaweddb
    maxDataSize = 10000
    maxHotBuckets = 10

[perfmon]
homePath = $SPLUNK_DB/perfmon/db
coldPath = $SPLUNK_DB/perfmon/colddb
thawedPath = $SPLUNK_DB/perfmon/thaweddb
maxDataSize = 10000
maxHotBuckets = 10

[wineventlog]
homePath = $SPLUNK_DB/wineventlog/db
coldPath = $SPLUNK_DB/wineventlog/colddb
thawedPath = $SPLUNK_DB/wineventlog/thaweddb
maxDataSize = 10000
maxHotBuckets = 10

**

outputs.conf

**

[tcpout]
defaultGroup=default-autolb-group

[tcpout:default-autolb-group]
server=192.168.0.37:9997

[tcpout-server://192.168.0.37:9997]

Input.conf

# Copyright (C) 2009-2016 Splunk Inc. All Rights Reserved.
# DO NOT EDIT THIS FILE!
# Please make all changes to files in $SPLUNK_HOME/etc/apps/Splunk_TA_windows/local.
# To make changes, copy the section/stanza you want to change from $SPLUNK_HOME/etc/apps/Splunk_TA_windows/default
# into ../local and edit there.
#

[default]
evt_dc_name =
evt_dns_name =


###### OS Logs ######
[WinEventLog://Application]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
index = wineventlog
renderXml=false

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\s*groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\s*groupPolicyContainer)"
index = wineventlog
renderXml=false

[WinEventLog://System]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
index = wineventlog
renderXml=false


####### OS Logs (Splunk 5.x only) ######
# If you are running Splunk 5.x remove the above OS log stanzas and uncomment these three.
#[WinEventLog:Application]
#disabled = 0
#start_from = oldest
#current_only = 0
#checkpointInterval = 5
#index = wineventlog
#
#[WinEventLog:Security]
#disabled = 0
#start_from = oldest
#current_only = 0
#evt_resolve_ad_obj = 1
#checkpointInterval = 5
#index = wineventlog
#
#[WinEventLog:System]
#disabled = 0
#start_from = oldest
#current_only = 0
#checkpointInterval = 5
#index = wineventlog


###### DHCP ######
[monitor://$WINDIR\System32\DHCP]
disabled = 0
whitelist = DhcpSrvLog*
crcSalt = 
sourcetype = DhcpSrvLog
index = windows


###### Windows Update Log ######
[monitor://$WINDIR\WindowsUpdate.log]
disabled = 0
sourcetype = WindowsUpdateLog
index = windows


###### Scripted Input (See also wmi.conf)
[script://.\bin\win_listening_ports.bat]
disabled = 0
## Run once per hour
interval = 3600
sourcetype = Script:ListeningPorts
index = windows

[script://.\bin\win_installed_apps.bat]
disabled = 0
## Run once per day
interval = 86400
sourcetype = Script:InstalledApps
index = windows

###### Host monitoring ######
[WinHostMon://Computer]
interval = 600
disabled = 0
type = Computer
index = windows

[WinHostMon://Process]
interval = 600
disabled = 0
type = Process
index = windows

[WinHostMon://Processor]
interval = 600
disabled = 0
type = Processor
index = windows

[WinHostMon://Application]
interval = 600
disabled = 0
type = Application
index = windows

[WinHostMon://NetworkAdapter]
interval = 600
disabled = 0
type = NetworkAdapter
index = windows

[WinHostMon://Service]
interval = 600
disabled = 0
type = Service
index = windows

[WinHostMon://OperatingSystem]
interval = 600
disabled = 0
type = OperatingSystem
index = windows

[WinHostMon://Disk]
interval = 600
disabled = 0
type = Disk
index = windows

[WinHostMon://Driver]
interval = 600
disabled = 0
type = Driver
index = windows

[WinHostMon://Roles]
interval = 600
disabled = 0
type = Roles
index = windows

###### Print monitoring ######
[WinPrintMon://printer]
type = printer
interval = 600
baseline = 1
disabled = 0
index = windows

[WinPrintMon://driver]
type = driver
interval = 600
baseline = 1
disabled = 0
index = windows

[WinPrintMon://port]
type = port
interval = 600
baseline = 1
disabled = 0
index = windows

###### Network monitoring ######
[WinNetMon://inbound]
direction = inbound
disabled = 0
index = windows

[WinNetMon://outbound]
direction = outbound
disabled = 0
index = windows

###### Splunk 5.0+ Performance Counters ######
## CPU
[perfmon://CPU]
counters = % Processor Time; % User Time; % Privileged Time; Interrupts/sec; % DPC Time; % Interrupt Time; DPCs Queued/sec; DPC Rate; % Idle Time; % C1 Time; % C2 Time; % C3 Time; C1 Transitions/sec; C2 Transitions/sec; C3 Transitions/sec
disabled = 0
instances = *
interval = 10
object = Processor
useEnglishOnly=true
index = perfmon

## Logical Disk
[perfmon://LogicalDisk]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
disabled = 0
instances = *
interval = 10
object = LogicalDisk
useEnglishOnly=true
index = perfmon

## Physical Disk
[perfmon://PhysicalDisk]
counters = Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
disabled = 0
instances = *
interval = 10
object = PhysicalDisk
useEnglishOnly=true
index = perfmon

## Memory
[perfmon://Memory]
counters = Page Faults/sec; Available Bytes; Committed Bytes; Commit Limit; Write Copies/sec; Transition Faults/sec; Cache Faults/sec; Demand Zero Faults/sec; Pages/sec; Pages Input/sec; Page Reads/sec; Pages Output/sec; Pool Paged Bytes; Pool Nonpaged Bytes; Page Writes/sec; Pool Paged Allocs; Pool Nonpaged Allocs; Free System Page Table Entries; Cache Bytes; Cache Bytes Peak; Pool Paged Resident Bytes; System Code Total Bytes; System Code Resident Bytes; System Driver Total Bytes; System Driver Resident Bytes; System Cache Resident Bytes; % Committed Bytes In Use; Available KBytes; Available MBytes; Transition Pages RePurposed/sec; Free & Zero Page List Bytes; Modified Page List Bytes; Standby Cache Reserve Bytes; Standby Cache Normal Priority Bytes; Standby Cache Core Bytes; Long-Term Average Standby Cache Lifetime (s)
disabled = 0
interval = 10
object = Memory
useEnglishOnly=true
index = perfmon

## Network
[perfmon://Network]
counters = Bytes Total/sec; Packets/sec; Packets Received/sec; Packets Sent/sec; Current Bandwidth; Bytes Received/sec; Packets Received Unicast/sec; Packets Received Non-Unicast/sec; Packets Received Discarded; Packets Received Errors; Packets Received Unknown; Bytes Sent/sec; Packets Sent Unicast/sec; Packets Sent Non-Unicast/sec; Packets Outbound Discarded; Packets Outbound Errors; Output Queue Length; Offloaded Connections; TCP Active RSC Connections; TCP RSC Coalesced Packets/sec; TCP RSC Exceptions/sec; TCP RSC Average Packet Size  
disabled = 0
instances = *
interval = 10
object = Network Interface
useEnglishOnly=true
index = perfmon

## Process
[perfmon://Process]
counters = % Processor Time; % User Time; % Privileged Time; Virtual Bytes Peak; Virtual Bytes; Page Faults/sec; Working Set Peak; Working Set; Page File Bytes Peak; Page File Bytes; Private Bytes; Thread Count; Priority Base; Elapsed Time; ID Process; Creating Process ID; Pool Paged Bytes; Pool Nonpaged Bytes; Handle Count; IO Read Operations/sec; IO Write Operations/sec; IO Data Operations/sec; IO Other Operations/sec; IO Read Bytes/sec; IO Write Bytes/sec; IO Data Bytes/sec; IO Other Bytes/sec; Working Set - Private
disabled = 0
instances = *
interval = 10
object = Process
useEnglishOnly=true
index = perfmon

## System
[perfmon://System]
counters = File Read Operations/sec; File Write Operations/sec; File Control Operations/sec; File Read Bytes/sec; File Write Bytes/sec; File Control Bytes/sec; Context Switches/sec; System Calls/sec; File Data Operations/sec; System Up Time; Processor Queue Length; Processes; Threads; Alignment Fixups/sec; Exception Dispatches/sec; Floating Emulations/sec; % Registry Quota In Use
disabled = 0
instances = *
interval = 10
object = System
useEnglishOnly=true
index = perfmon

[admon://default]
disabled = 0
monitorSubtree = 1

[WinRegMon://default]
disabled = 0
hive = .*
proc = .*
type = rename|set|delete|create
index = windows

[WinRegMon://hkcu_run]
disabled = 0
hive = \\REGISTRY\\USER\\.*\\Software\\Microsoft\\Windows\\CurrentVersion\\Run\\.*
proc = .*
type = set|create|delete|rename
index = windows

[WinRegMon://hklm_run]
disabled = 0
hive = \\REGISTRY\\MACHINE\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run\\.*
proc = .*
type = set|create|delete|rename
index = windows

pauloyelakin · ‎02-12-2018

Hi,
thanks for responding.

My indexes.conf file is in RISEN200 (192.168.0.37) which is my indexer/search head. this is the same application where my deployment server is in. URL: C:\Program Files\Splunk\etc\apps\splunk_app_windows_infrastructure\local

I did this as per instruction on how to install splunk app for windows infrastructure... (http://docs.splunk.com/Documentation/MSApp/1.4.3/MSInfra/InstallaSplunkIndexer)

as for risen 300, it is not in the deployment server, there are no splunk services running on it and risenlab1 only has one dns name...

ansif · ‎02-13-2018

Really strange. Can you stop service on another server and check if risen300 is still sending data?

pauloyelakin · ‎02-16-2018

I stopped services on risenlab1 and data was still being collected by the internal indexer for Risen300 AND risenlab1...

one other thing that's strange is my index summary increases by 270 indexed item... I believe this happens everytime I boot risenlab1.

I'm just not sure what the problem is at this point... The only thing I can think of is if my index.conf file wrong? honestly I'm not sure what all those lines mean... I just copied the example into the local folder of my app as per the instruction in the link below...

http://docs.splunk.com/Documentation/MSApp/1.4.3/MSInfra/InstallaSplunkIndexer

ansif · ‎02-18-2018

I don't think the example given in the doc is wrong. Even the configuration is wrong once you stopped the Splunk daemon it should not send data to Indexer.

pauloyelakin · ‎02-19-2018

update: so it would appear that data has been indexed the whole time. i queried [index=wineventlog host=risenlab1 ] and got a plethora of data in the result.

But why is my data summary not reflecting all these data???

ansif · ‎02-11-2018

Hi @pauloyelakin :The internal logs from Splunk software are useful for troubleshooting or metric analysis.

http://docs.splunk.com/Documentation/Splunk/7.0.2/Troubleshooting/WhatSplunklogsaboutitself

May I know where the indexes.conf file is placed (Server and the dir)? If it is not placed in indexer then all parameters get dropped at indexer.

Regarding risen300 check if deployment server -> setting -> Forwarder Management has this server and Splunk service is running on this host. And check if the your single host (risenlab1) has only one dns name.

Why is no data coming into the summary index but I can see data in the internal index?

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

Splunk and Fraud

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...