splunk does not show all data for type completed for jenkins_statistics

[khandelwaly]

we are using splunk forwarder to forward the jenkins data to splunk. Noticed that splunk does not display all the data.

here is the example:

index=jenkins_statistics (host=abc.com/*) event_tag=job_event job_name="*abc/develop*"
| stats count by job_name, type returns completed = 74 and started = 118 

Ideally whatever is started should also be completed. so can you help me figuring out what could be the problem?

[khandelwaly]

Sorry for not providing enough information earlier.

We are running 5 jobs daily in our system but we are seeing some jenkins job data are not getting reported back on splunk. Out of 5, splunk shows only 3 jobs if we have the query like 

index=jenkins_statistics (host=abc.com/*) event_tag=job_event type=completed job_name="*abc/develop*"
| stats count by job_name, type

If we remove the type from the above query, we get more data which tells us that some jobs are marking as started but splunk not getting the completed event for the same job, hence data discrepancies are there. 

So just wanted to check do we have guaranteed delivery for Splunk event from Jenkins to Splunk? As per my understanding, events are sent to an Splunk HTTP Endpoint Collector endpoint and they are sent fire and forget

Either some of the events are getting dropped at that level or there is a bug somewhere in the https://plugins.jenkins.io/splunk-devops plugin that is causing events to get missed

[PickleRick]

Don't stats. Just look for raw events. If you have them, the problem is probably in parsing. If you don't, search why you didn't get them ingested properly.

[PickleRick]

It's impossible to answer such question without knowing your data and your environment. You can start debugging by checking which jobs were started and verifying if you can find a corresponding job completed event for them. If so check if the data is in different format or if your extractions properly match the fields. If not check your ingestion pipeline to see why there are missing events.