Hi @gcusell,
I have 2 double
1. How can I drop a source IP 10.0.0.0/24 subnet at indexer, I am aware of dropping a host at the indexer level but not this.
2. I'm getting duplicate data I.e. Duplicate data is being indexed. My query is how can know from which host the data is duplicate so that I can offboard those devices..
Kindly guide me for the above 2 solution.
Thanks
Debjit
Hi @debjit_k,
next time, please make two different questions, so you'll have more choices to have an answer to both of them.
Anyway, about the first question:
About the second question:
If yes, you can reduce the scope of your search adding the condition to the main search, otherwise you can run a search like this:
index=*
| stats dc(_raw) raw_count BY host
| where raw_count>1
in this way, you have the list of hosts with duplicated events.
Obviously this search gives results if the events are completely duplicated, if there's something different (also only timestamp) it doesn't give results.
Ciao.
Giuseppe
Hi @gcusell,
Your point has been noted.
1. For question number 1
Yes im talking about filtering event at index time.
My idea is to define
Source type in transform.com
And in props.conf
Need to write the regex for the subnet and queue should be null. Kindly correct me if im wrong
For question 2
Yes I want to identify the host having duplicate data and yes I have identity the index.
For question 2 I'm guessing you give the required query which will help to identify the host from where we are getting duplicate data.
Thanks
Debjit
Hi @debjit_k,
about question 1:
you have to assign a fixed sourcetype to your input, then on Indexers or (if present) on Heavy Forwarders, you have to override this sourcetype following the instructios at https://docs.splunk.com/Documentation/Splunk/9.0.2/Data/Advancedsourcetypeoverrides#:~:text=You%20ca....
About question 2, you can put the index in the main search so you can find the host with duplicated data using my search.
Ciao,
Giuseppe
Hi @gcusell,
Thank you for the support
For question 1 can you please help me with the regex of subnet ex 10.10.1.0/12
For question 2 is there any where from where we can get to know that data in host 1 is similar to data in host 2
Thanks
Debjit
Hi @debjit_k,
about question 1, please share a sample of your data.
About question 2, you don't have a fully duplicated data! in this case you have to identify the part of the event that's duplicated, extract it and use this part for the above check.
Ciao.
Giuseppe