Splunk Search

Not showing data for a particular sourcetype

pratapa
Explorer

Events are not getting generated after the date 15th June, 2019 for the following query.

index=webmethods_prd sourcetype="webmethods:wmerror"

 

However, events are getting generated for the dates before 15th June,2019. 

User needs the events to be generated for the dates  after 15th June, 2019 as well.

What could be the problem?

Labels (1)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

dbmon-tail is probably part of DB Connect v.1.x? Maybe someone has updated OS or something else on that server. If so, it’s probably easiest to figure out if you could do a fresh instal of DB Connect 3.3.x and configure it to connect DB of Webmethods..

r. Ismo

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust
The problem could be any of a number of things.
Perhaps the application generating the data stopped. Or the network connection between it and Splunk broke. Perhaps the user is not searching for his data properly. Perhaps the data is there, but the timestamp is incorrect so the data can't be found. Do you have more information?
---
If this reply helps you, Karma would be appreciated.
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

dbmon-tail is probably part of DB Connect v.1.x? Maybe someone has updated OS or something else on that server. If so, it’s probably easiest to figure out if you could do a fresh instal of DB Connect 3.3.x and configure it to connect DB of Webmethods..

r. Ismo

0 Karma

pratapa
Explorer

The issue was "Java Bridge server was not running". We engaged splunk support and they fixed the issue.

We configured to connect DB of webmethods and restarted splunk. That resolved the issue.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @pratapa ,

to debug ingesting you have to check all the data supply chain:

  • data are generated by the source system?
    • you can check this analyzing files on source system;
  • the source system reach to send logs to the Indexer?
    • you can check this with a simple search index=_internal host=source_system_hostname​
  • are index and sourcetype correct?
    • you can check this analyzing inputs.conf on the source system.

Ciao.

Giuseppe

0 Karma

pratapa
Explorer

data are generated by the source system?

  • you can check this analyzing files on source system;

I need to check from the user what data he is looking for.

 

the source system reach to send logs to the Indexer?

  • you can check this with a simple search index=_internal host=source_system_hostname​

Yes source system can able to reach indexer to send logs.

This I verified by the following query.

index=_internal host=aeapwmora02

Events got generated of today's data.

 

are index and sourcetype correct?

  • you can check this analyzing inputs.conf on the source system.

 

I verified the inputs.conf on the source sytem but nothing defined in inputs.conf

 

[root@aeaxwmora02 local]# pwd
/opt/splunkforwarder/etc/system/local
[root@aeaxwmora02 local]# cat inputs.conf
[default]
host = aeaxwmora02
[root@aeaxwmora02 local]#

We found that index  webmethods_prd is existing by the following query.

| eventcount summarize=false index=* index=_* | dedup index | fields index

 

and sourcetype is existing by the following query.

sourcetype="webmethods:wmerror" host=aeaxwmora02

Events are getting generated and we got latest event dated 6/15/2019. 

No events generated after the data 6/15/2019. 

Under interesting fields, value of  index is showing as webmethods_prd.

 

What should be the next steps.

 

Do I need to get the information from the user what source files he is looking for along with the path.

and configure in inputs.conf

 

 

0 Karma

pratapa
Explorer

Can you please help me on next steps.

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @pratapa ,

if logs are generated by the target system, try to force sourcetype in inputs.conf.

Then check in the past received logs, if the timestamp is correctly read by Splunk or if there some difference (e.g. solar time, date format, etc...).

Ciao.

Giuseppe

0 Karma

pratapa
Explorer

User says that it’s a database, not a file, and hence doesn’t follow the usual Splunk forwarder file indexing process.

Source is like this 

dbmon-tail://wmb2bprd/webmethods:WMERROR

 

How should I go.

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @pratapa ,

insert in your inputs.conf the sourcetype to assign to this log.

Then running a search in Splunk on the old logs, check if the Timestamp is correctly assigned to the events.

Ciao.

Giuseppe

0 Karma

pratapa
Explorer

Hi,

 

I incorporated the following in inputs.conf.

[monitor://dbmon-tail://wmb2bprd/webmethods:WMERROR]
index = webmethods_prd
sourcetype = webmethods:wmerror
disabled = false

[root@aeaxwmora02 local]# pwd
/opt/splunkforwarder/etc/system/local
[root@aeaxwmora02 local]# cat inputs.conf
[default]
host = aeaxwmora02

[monitor://dbmon-tail://wmb2bprd/webmethods:WMERROR]
index = webmethods_prd
sourcetype = webmethods:wmerror
disabled = false

Restarted Splunk, but still showing data till 6/15/2019. Data  after 6/15/2019 is not showing.

 

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @pratapa ,

I don't see errors.

could you share an example of data (before 15) and the search you're running?

Ciao.

Giuseppe

0 Karma

pratapa
Explorer

Following is the search query.

 

index=webmethods_prd sourcetype=webmethods:wmerror

 

Time range picker : All time

Sample data of 15th June, 2019.

15/06/2019
03:30:46.948
aeaxwmora02dbmon-tail://wmb2bprd/webmethods:WMERRORwebmethods:wmerror
 

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @pratapa ,

I need some sample of raw data to try to understand which is the problem.

Ciao.

Giuseppe

0 Karma

pratapa
Explorer
Tags (1)
0 Karma

pratapa
Explorer

This issue is resolved.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...