topic Re: Not showing data for a particular sourcetype in Splunk Search

Not showing data for a particular sourcetype

pratapa — Thu, 18 Jun 2020 11:03:27 GMT

Events are not getting generated after the date 15th June, 2019 for the following query.

index=webmethods_prd sourcetype="webmethods:wmerror"

However, events are getting generated for the dates before 15th June,2019.

User needs the events to be generated for the dates after 15th June, 2019 as well.

What could be the problem?

Re: Not showing data for a particular sourcetype

gcusello — Thu, 18 Jun 2020 12:15:30 GMT

Hi @pratapa ,

to debug ingesting you have to check all the data supply chain:

data are generated by the source system?
- you can check this analyzing files on source system;
the source system reach to send logs to the Indexer?
- you can check this with a simple search index=_internal host=source_system_hostname
are index and sourcetype correct?
- you can check this analyzing inputs.conf on the source system.

Ciao.

Giuseppe

Re: Not showing data for a particular sourcetype

richgalloway — Thu, 18 Jun 2020 12:27:36 GMT

The problem could be any of a number of things.
Perhaps the application generating the data stopped. Or the network connection between it and Splunk broke. Perhaps the user is not searching for his data properly. Perhaps the data is there, but the timestamp is incorrect so the data can't be found. Do you have more information?

Re: Not showing data for a particular sourcetype

pratapa — Fri, 19 Jun 2020 11:27:04 GMT

data are generated by the source system?

you can check this analyzing files on source system;

I need to check from the user what data he is looking for.

the source system reach to send logs to the Indexer?

you can check this with a simple search index=_internal host=source_system_hostname

Yes source system can able to reach indexer to send logs.

This I verified by the following query.

index=_internal host=aeapwmora02

Events got generated of today's data.

are index and sourcetype correct?

you can check this analyzing inputs.conf on the source system.

I verified the inputs.conf on the source sytem but nothing defined in inputs.conf

[root@aeaxwmora02 local]# pwd
/opt/splunkforwarder/etc/system/local
[root@aeaxwmora02 local]# cat inputs.conf
[default]
host = aeaxwmora02
[root@aeaxwmora02 local]#

We found that index webmethods_prd is existing by the following query.

| eventcount summarize=false index=* index=_* | dedup index | fields index

and sourcetype is existing by the following query.

sourcetype="webmethods:wmerror" host=aeaxwmora02

Events are getting generated and we got latest event dated 6/15/2019.

No events generated after the data 6/15/2019.

Under interesting fields, value of index is showing as webmethods_prd.

What should be the next steps.

Do I need to get the information from the user what source files he is looking for along with the path.

and configure in inputs.conf

Re: Not showing data for a particular sourcetype

pratapa — Mon, 22 Jun 2020 12:35:13 GMT

Can you please help me on next steps.

Re: Not showing data for a particular sourcetype

gcusello — Tue, 23 Jun 2020 05:52:05 GMT

Hi @pratapa ,

if logs are generated by the target system, try to force sourcetype in inputs.conf.

Then check in the past received logs, if the timestamp is correctly read by Splunk or if there some difference (e.g. solar time, date format, etc...).

Ciao.

Giuseppe

RedNot showing data for a particular sourcetype

pratapa — Tue, 23 Jun 2020 08:49:48 GMT

User says that it’s a database, not a file, and hence doesn’t follow the usual Splunk forwarder file indexing process.

Source is like this

dbmon-tail://wmb2bprd/webmethods:WMERROR

How should I go.

Re: RedNot showing data for a particular sourcetype

gcusello — Tue, 23 Jun 2020 09:14:31 GMT

Hi @pratapa ,

insert in your inputs.conf the sourcetype to assign to this log.

Then running a search in Splunk on the old logs, check if the Timestamp is correctly assigned to the events.

Ciao.

Giuseppe

Re: Not showing data for a particular sourcetype

pratapa — Thu, 25 Jun 2020 09:45:59 GMT

Hi,

I incorporated the following in inputs.conf.

[monitor://dbmon-tail://wmb2bprd/webmethods:WMERROR]
index = webmethods_prd
sourcetype = webmethods:wmerror
disabled = false

[root@aeaxwmora02 local]# pwd
/opt/splunkforwarder/etc/system/local
[root@aeaxwmora02 local]# cat inputs.conf
[default]
host = aeaxwmora02

[monitor://dbmon-tail://wmb2bprd/webmethods:WMERROR]
index = webmethods_prd
sourcetype = webmethods:wmerror
disabled = false

Restarted Splunk, but still showing data till 6/15/2019. Data after 6/15/2019 is not showing.

Re: Not showing data for a particular sourcetype

gcusello — Thu, 25 Jun 2020 14:22:14 GMT

Hi @pratapa ,

I don't see errors.

could you share an example of data (before 15) and the search you're running?

Ciao.

Giuseppe

Re: Not showing data for a particular sourcetype

pratapa — Fri, 03 Jul 2020 10:32:41 GMT

Following is the search query.

index=webmethods_prd sourcetype=webmethods:wmerror

Time range picker : All time

Sample data of 15th June, 2019.

15/06/2019 03:30:46.948	aeaxwmora02	dbmon-tail://wmb2bprd/webmethods:WMERROR	webmethods:wmerror

Re: Not showing data for a particular sourcetype

gcusello — Fri, 03 Jul 2020 15:10:44 GMT

Hi @pratapa ,

I need some sample of raw data to try to understand which is the problem.

Ciao.

Giuseppe

Re: Not showing data for a particular sourcetype

isoutamo — Sun, 05 Jul 2020 19:37:06 GMT

dbmon-tail is probably part of DB Connect v.1.x? Maybe someone has updated OS or something else on that server. If so, it’s probably easiest to figure out if you could do a fresh instal of DB Connect 3.3.x and configure it to connect DB of Webmethods..

r. Ismo

Re: Not showing data for a particular sourcetype

pratapa — Tue, 07 Jul 2020 06:05:48 GMT

Type Field Value Actions

Selected		host	aeapwmora02
	source	dbmon-tail://wmb2bprd/webmethods:WMERROR
	sourcetype	webmethods:wmerror
Event		AUDITTIMESTAMP	1560497446.948
	CONTEXTID	d34b7125-313a-4d6f-9638-ad324032802f
	ERRORMESSAGE	document is null
	ERRSTACKTRACE	Error Type:exception, Error Message:document is null, Service Name:
	INSERTTIMESTAMP	1560497446.949
	MSGID	8090bffc-5837-93c2-cb9c-833f427fb709
	PARENTCONTEXTID	61739280-d3af-44fd-a7c6-0afd3cf4d450
	ROOTCONTEXTID	702f9ad0-1c19-4181-b543-56f3ee69e010
	SERVERID	aeapwmappb2b39.ce.corp:5555
	SERVICENAME	pub.event.exception:logToFile
	eventtype	err0r (error)
nix-all-logs
nix_errors (error)

Re: Not showing data for a particular sourcetype

pratapa — Tue, 07 Jul 2020 06:08:13 GMT











	eventtype	err0r (error)
nix-all-logs
nix_errors (error)
	oracle_sid	wmb2bprd
	tag	error
	unix_category	all_hosts
	unix_group	default
	wmApplication	database
	wmApplicationType	b2b
	wmEnvironment	prd
Time		_time	2019-06-15T03:30:46.948+10:00
Default		index	webmethods_prd
	linecount	12
	punct	--_::.=----=----=----=.=.=..:=..:=----=__=_:,__:__
	splunk_server	aeapsplnd01

Re: Not showing data for a particular sourcetype

pratapa — Tue, 07 Jul 2020 10:42:51 GMT

The issue was "Java Bridge server was not running". We engaged splunk support and they fixed the issue.

We configured to connect DB of webmethods and restarted splunk. That resolved the issue.

Re: Not showing data for a particular sourcetype

pratapa — Tue, 07 Jul 2020 10:43:39 GMT

This issue is resolved.