Splunk Index Best Practices

[loginsoft]

Hi-

We are indexing JSON data into Splunk. We push the data once every 24 hours. The Rest API will not give "Delta:", on every run but returns all data. So we don't have a way to recognize what is updated and what is same. Time stamp is also same. The JSON is huge.

In this case what is the best way to Index? Delete old data update all? or is there a way to Archive existing data and update new Json data? etc?

Thanks

Suren

[niketn]

@loginsoft Build the input through Splunk Add On Builder app which allows you to keep a track of already indexed check point. Besides you also can extend the underlying python with your customization to ensure that already indexed data is ignored. Read the documentation for setting the checkpoint value (also covers how to set it for JSON data): https://docs.splunk.com/Documentation/AddonBuilder/latest/UserGuide/ConfigureDataCollection

[isoutamo]

Hi

probably the best option is use scripted input which has some kind on “book keeping” / delta what has already ingested and based on that it sends only new / modified data to the indexers. 
r. Ismo