How to drop a subnet at index?

[debjit_k]

Hi @gcusell,

I have 2 double

1. How can I drop a source IP 10.0.0.0/24 subnet at indexer, I am aware of dropping a host at the indexer level but not this.

2. I'm getting duplicate data I.e. Duplicate data is being indexed. My query is how can know from which host the data is duplicate so that I can offboard those devices.. 

Kindly guide me for the above 2 solution.

Thanks

Debjit

[gcusello]

Hi @debjit_k,

next time, please make two different questions, so you'll have more choices to have an answer to both of them.

Anyway, about the first question:

• could you better describe your requirement?
• what do you mean with dropping a source IP at Indexer?
• do you mean filter events?
• are you speaking at search time or at index time?

About the second question:

• do you want to identify hosts that send duplicated data or what else?
• have you identified the index or the sourcetype with duplicated data?

If yes, you can reduce the scope of your search adding the condition to the main search, otherwise you can run a search like this:

index=*
| stats dc(_raw) raw_count BY host
| where raw_count>1

in this way, you have the list of hosts with duplicated events.

Obviously this search gives results if the events are completely duplicated, if there's something different (also only timestamp) it doesn't give results.

Ciao.

Giuseppe

[debjit_k]

Hi @gcusell,

Your point has been noted.

1. For question number 1 

Yes im talking about filtering event at index time.

My idea is to define 

Source type in transform.com

And in props.conf 

Need to write the regex for the subnet and queue should be null. Kindly correct me if im wrong

For question 2 

Yes I want to identify the host having duplicate data and yes I have identity the index.

For question 2 I'm guessing you give the required query which will help to identify the host from where we are getting duplicate data.

Thanks 

Debjit

[gcusello]

Hi @debjit_k,

about question 1:

you have to assign a fixed sourcetype to your input, then on Indexers or (if present) on Heavy Forwarders, you have to override this sourcetype following the instructios at https://docs.splunk.com/Documentation/Splunk/9.0.2/Data/Advancedsourcetypeoverrides#:~:text=You%20ca....

About question 2, you can put the index in the main search so you can find the host with duplicated data using my search.

Ciao,

Giuseppe

[debjit_k]

Hi @gcusell,

Thank you for the support 

For question 1 can you please help me with the regex of subnet ex 10.10.1.0/12

For question 2 is there any where from where we can get to know that data in host 1 is similar to data in host 2 

Thanks 

Debjit

[gcusello]

Hi @debjit_k,

about question 1, please share a sample of your data.

About question 2, you don't have a fully duplicated data! in this case you have to identify the part of the event that's duplicated, extract it and use this part for the above check.

Ciao.

Giuseppe