Events Source Validation

[hectorvp]

Hello Splunkers,

I've been in some weird requirement/situation, which is,

we need to validate if events  of particular source and sourcetype are getting  forwarded by UF or not.

For Ex:  Our application XYZ is logging events in windows event logging system we need to make sure that events are getting forwarded by UF to the indexers for particular source that is "XYZ". & application registers the events in windows event logging system very rarely say once in 7 days.

Can we validate this using only UFs internal logs?

Yes we have this big restriction of validating above situation only with UFs internal logs, we simple cannot query for the source and check its earliest coz we dont have access to indexers containing actual logs.

I checked with metrics.log but it will tell only about top 10 sources of events, in that case if we have 35 sources on single server, I guess we won't be able to do it.

Can anyone help me with this?

[gcusello]

Hi @hectorvp,

let me understand, you need to check that :

• at least every seven days,
• from one or more hosts (e.g. host1),
• you receive logs with source="XYZ",
• logs stored in and index (e.g. index1)

is this correct?

If this is your need you have to run a search like this.

index=index1 host=host1 source="XYZ" earliest=-7d@d latest=@d

and schedule it every night e.g. at 1.00:

if you haven't results you have to fire the alert.

If the hosts to check are many, you could put them in a lookup.

Ciao.

Giuseppe

[hectorvp]

Hi @gcusello ,

Our main aim is to forward events from our servers to customers indexers.

Since we don't have access to the customer's indexers, we are not able to validate that are we sending all appn logs using above approach.

Is  all source sending events to customer indexers can be identified with _internal index??  I saw metrics.log only shows top 10 results for source type.

We are storing _internal events of UFs in our single indexer for health checkup of UFs. Can we validate within internal events?

Or what if we add an HF in between, which will see the source log for the first time in a day and store it in our indexer and then stops sending logs to our indexer and will forward only to the customer indexers, so our indexer will have only 1 log per source per server and won't replicate whole logs and make unnecessarily 2x of cost for license.

Further on search head will use above approach the ony mentioned by you to raise an alert?? Is this possible on HF?

[gcusello · Accepted solution]

Hi @hectorvp,

the best approach is to have a little Splunk license to install on the HF or connect the HF to the License Master and locally index a copy of store some events to run a search.

If this isn't possible you could configure your HF to send one syslog event every day and in this way have and heartbeat from your sources.

Ciao.

Giuseppe

[hectorvp]

Hi @gcusello ,

First of all thanks for the suggested approaches.

Apologies, but it would be a great help if you can explain the 2nd approach in detail, I mean how can I configure HF to send single event of a particular source per host to our indexer in a day + all the events to customer indexers and then  we can have validation to logs we are sending.

Question is raised on community with the link,

https://community.splunk.com/t5/Monitoring-Splunk/Forward-single-event-from-HF-to-Indexer/td-p/52778...