topic Re: Dropping a subnet at index in Getting Data In

How to drop a subnet at index?

debjit_k — Mon, 19 Dec 2022 15:34:42 GMT

I have 2 double

1. How can I drop a source IP 10.0.0.0/24 subnet at indexer, I am aware of dropping a host at the indexer level but not this.

2. I'm getting duplicate data I.e. Duplicate data is being indexed. My query is how can know from which host the data is duplicate so that I can offboard those devices..

Kindly guide me for the above 2 solution.

Thanks

Debjit

Re: Dropping a subnet at index

gcusello — Sun, 18 Dec 2022 09:23:43 GMT

Hi @debjit_k,

next time, please make two different questions, so you'll have more choices to have an answer to both of them.

Anyway, about the first question:

could you better describe your requirement?
what do you mean with dropping a source IP at Indexer?
do you mean filter events?
are you speaking at search time or at index time?

About the second question:

do you want to identify hosts that send duplicated data or what else?
have you identified the index or the sourcetype with duplicated data?

If yes, you can reduce the scope of your search adding the condition to the main search, otherwise you can run a search like this:

index=* | stats dc(_raw) raw_count BY host | where raw_count>1

in this way, you have the list of hosts with duplicated events.

Obviously this search gives results if the events are completely duplicated, if there's something different (also only timestamp) it doesn't give results.

Ciao.

Giuseppe

Re: Dropping a subnet at index

debjit_k — Sun, 18 Dec 2022 13:55:40 GMT

Hi @gcusell,

Your point has been noted.

1. For question number 1

Yes im talking about filtering event at index time.

My idea is to define

Source type in transform.com

And in props.conf

Need to write the regex for the subnet and queue should be null. Kindly correct me if im wrong

For question 2

Yes I want to identify the host having duplicate data and yes I have identity the index.

For question 2 I'm guessing you give the required query which will help to identify the host from where we are getting duplicate data.

Thanks

Debjit

Re: Dropping a subnet at index

gcusello — Sun, 18 Dec 2022 16:36:49 GMT

Hi @debjit_k,

about question 1:

you have to assign a fixed sourcetype to your input, then on Indexers or (if present) on Heavy Forwarders, you have to override this sourcetype following the instructios at https://docs.splunk.com/Documentation/Splunk/9.0.2/Data/Advancedsourcetypeoverrides#:~:text=You%20can%20override%20source%20types,directly%20on%20the%20instance%20itself.

About question 2, you can put the index in the main search so you can find the host with duplicated data using my search.

Ciao,

Giuseppe

Re: Dropping a subnet at index

debjit_k — Mon, 19 Dec 2022 01:37:20 GMT

Hi @gcusell,

Thank you for the support

For question 1 can you please help me with the regex of subnet ex 10.10.1.0/12

For question 2 is there any where from where we can get to know that data in host 1 is similar to data in host 2

Thanks

Debjit

Re: Dropping a subnet at index

gcusello — Mon, 19 Dec 2022 07:36:25 GMT

Hi @debjit_k,

about question 1, please share a sample of your data.

About question 2, you don't have a fully duplicated data! in this case you have to identify the part of the event that's duplicated, extract it and use this part for the above check.

Ciao.

Giuseppe