Getting Data In

How to drop a subnet at index?

debjit_k
Path Finder

Hi @gcusell,

I have 2 double

 

1. How can I drop a source IP 10.0.0.0/24 subnet at indexer, I am aware of dropping a host at the indexer level but not this.

2. I'm getting duplicate data I.e. Duplicate data is being indexed. My query is how can know from which host the data is duplicate so that I can offboard those devices.. 

 

Kindly guide me for the above 2 solution.

 

Thanks

Debjit

Labels (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k,

next time, please make two different questions, so you'll have more choices to have an answer to both of them.

Anyway, about the first question:

  • could you better describe your requirement?
  • what do you mean with dropping a source IP at Indexer?
  • do you mean filter events?
  • are you speaking at search time or at index time?

About the second question:

  • do you want to identify hosts that send duplicated data or what else?
  • have you identified the index or the sourcetype with duplicated data?

If yes, you can reduce the scope of your search adding the condition to the main search, otherwise you can run a search like this:

index=*
| stats dc(_raw) raw_count BY host
| where raw_count>1

in this way, you have the list of hosts with duplicated events.

Obviously this search gives results if the events are completely duplicated, if there's something different (also only timestamp) it doesn't give results.

Ciao.

Giuseppe

debjit_k
Path Finder

Hi @gcusell,

Your point has been noted.

1. For question number 1 

Yes im talking about filtering event at index time.

My idea is to define 

Source type in transform.com

And in props.conf 

Need to write the regex for the subnet and queue should be null. Kindly correct me if im wrong

 

 

For question 2 

Yes I want to identify the host having duplicate data and yes I have identity the index.

For question 2 I'm guessing you give the required query which will help to identify the host from where we are getting duplicate data.

 

Thanks 

Debjit

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k,

about question 1:

you have to assign a fixed sourcetype to your input, then on Indexers or (if present) on Heavy Forwarders, you have to override this sourcetype following the instructios at https://docs.splunk.com/Documentation/Splunk/9.0.2/Data/Advancedsourcetypeoverrides#:~:text=You%20ca....

About question 2, you can put the index in the main search so you can find the host with duplicated data using my search.

Ciao,

Giuseppe

debjit_k
Path Finder

Hi @gcusell,

 

Thank you for the support 

 

For question 1 can you please help me with the regex of subnet ex 10.10.1.0/12

 

For question 2 is there any where from where we can get to know that data in host 1 is similar to data in host 2 

 

Thanks 

Debjit

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k,

about question 1, please share a sample of your data.

About question 2, you don't have a fully duplicated data! in this case you have to identify the part of the event that's duplicated, extract it and use this part for the above check.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...