Splunk Search

i feel dumb, but I see no data when I search in splunk.

lennyburns
Path Finder

I created 8 data inputs, each one is supposed to tail log files mathing a certain whitelist regex.
These inputs see the files (my preview worked and i see the # of files in the data inputs page.

The Application im using for the inputs is SEARCH.

When I go to the SEARCH app, I type a word I KNOW is in the logs, and I get nothing.
I type * and I get nothing.

I'm clearly missing something basic.

This wasn't this hard when i did this a few years ago.

sgarvin55
Splunk Employee
Splunk Employee

I learn something new everyday!
Thanks gkanapathy, I owe you a beer!

0 Karma

sgarvin55
Splunk Employee
Splunk Employee

Ok, after trying to figure out why the wildcard "" wouldn't work in your inputs.conf, it was determined that it can't be used in the Splunk data preview or the stanza in inputs.conf.
EX:[monitor:///Users/username/Desktop/tmp/`debug-
.txt`] Although the docs say differently. I tried and I got the same result.

With that said, I had to create these configurations to get it to work:

inputs.conf
[monitor:///Users/username/Desktop/tmp] <---location of debug.txt and log.txt logs
whitelist = (log-.*\.txt|debug-.*\.txt)

props.conf
[source::.../tmp/debug...txt]
sourcetype = debug
TRANSFORMS-index = choose_debug_index

[source::.../tmp/log...txt]
sourcetype = log
TRANSFORMS-index = choose_log_index

transforms.conf
[choose_debug_index]
SOURCE_KEY = _raw
REGEX = .*
DEST_KEY = _MetaData:Index
FORMAT = debugtest

[choose_log_index]
SOURCE_KEY = _raw
REGEX = .*
DEST_KEY = _MetaData:Index
FORMAT = logtest

NOTE: The names of the indexes and transforms stanzas above can be changed to what suits your needs.
Hope that helps.

0 Karma

lennyburns
Path Finder

this format gets me close, but i dont understand how to adapt all this to my input data.

I dont understand alot of what you put in here or mean for me to replace, some is obvious, other parts are not.

I'm using a Windows SPLUNK instance, and so do I need to escape all these caharacters?

some stuff in here isnot anything i have ever seen.

{chhose log/debug index}, etc...
can you explain it more? perhaps use a sample of my data from above for your example?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

there is a way to show code in an Answers posting. The "code" button in the input editor (looks like "101-010" will make anything you type in literal. There should also be a help link somewhere to the formatting codes so you can type them directly (it is in Markdown), but some fail has removed it.

0 Karma

hexx
Splunk Employee
Splunk Employee

Usually, when I want to check that some data that I set to be indexed is present, I am as loose as possible with my search terms.

Typically, I will search over all time for :

index=* OR index=_* <TERM>

...where is a string (unique, if possible) naturally delimited in the source by delimiters such as white spaces, colons, slashes, etc.

Oh and there's no reason to feel dumb.

lennyburns
Path Finder

I get no results. 😞

0 Karma

lguinn2
Legend

You can search with just the index specified. The search will return all events in the index (constrained by your time range selection).

0 Karma

mikelanghorst
Motivator

What about with just the index specified, no other strings to filter.

0 Karma

mikelanghorst
Motivator

Here's the likely reason:
index = vmware_view_brokers

The summary app and by default your role will only search index=main by default. Simply add:
index=vmware_view_brokers

to your search and you should see the data just fine

You can change the default role that is searched under Manager> User Roles > your role.

lennyburns
Path Finder

system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000
system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
search thawedPath = $SPLUNK_DB\vmware_view_brokers\thaweddb
system throttleCheckPeriod = 15

0 Karma

lennyburns
Path Finder

system indexThreads = auto
system maxBloomBackfillBucketAge = 30d
system maxConcurrentOptimizes = 3
system maxDataSize = auto
system maxHotBuckets = 3
system maxHotIdleSecs = 0
system maxHotSpanSecs = 7776000
system maxMemMB = 5
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxRunningProcessGroupsLowPriority = 1
system maxTotalDataSizeMB = 500000
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0

0 Karma

lennyburns
Path Finder

C:\Program Files\Splunk\bin>btool indexes list --debug vmware_view_brokers
search [vmware_view_brokers]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
search coldPath = $SPLUNK_DB\vmware_view_brokers\colddb
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system enableOnlineBucketRepair = true
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 188697600
search homePath = $SPLUNK_DB\vmware_view_brokers\db

0 Karma

lennyburns
Path Finder

so i just run this command?

0 Karma

mikelanghorst
Motivator

Hmm, this might take some more investigating. What's the output of:
./splunk bin splunk cmd btool indexes list --debug vmware_view_brokers

Alternatively, if you have access to the IRC channel you could pop in there, then we could look deeper and then update this Question with the outcome.

0 Karma

lennyburns
Path Finder

I think this might be the problem..

In the indexes section, the vmware_view_brokers index is only at 1MB and EVENT COUNT is 0, and the rest is N/A.

argh.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

also make sure the indexes are actually created.

0 Karma

lennyburns
Path Finder

so, i did that...
index="vmware_view_brokers" WARN

and i dont find any occurance of WARN

0 Karma

lennyburns
Path Finder

how can i make the default index the brokers index? i plan to use splunk for nothign else but watching brokers.

0 Karma

lennyburns
Path Finder

Here is a sample of a few...

[monitor://\XXX-vdi-csa01\c$\Documents and Settings\All Users\Application Data\VMware\VDM\logs]
disabled = false
followTail = 1
host = XXX-VDI-CSA01_DEBUG-LOG
sourcetype = vmware_view_connection_broker_debug_log
index = vmware_view_brokers
blacklist = log-.txt
whitelist = debug-
.txt

[monitor://\XXX-vdi-csb01\c$\Documents and Settings\All Users\Application Data\VMware\VDM\logs]
disabled = false
followTail = 1
host = XXX-VDI-CSB01_LOG
sourcetype = vmware_view_connection_server_log
index = vmware_view_brokers
blacklist = debug-.txt
whitelist = log-
.txt

0 Karma

mikelanghorst
Motivator

Could you post the relevant stanza from inputs.conf? Are you sending the data to an index you're then not searching for?

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...