topic Re: splunk remove data after indexing in Deployment Architecture

splunk remove data after indexing

TISKAR — Tue, 20 Sep 2016 09:23:22 GMT

Hello

I have a big problem with the addition of data.
initially given 9 million are added. but after I find Splunk removes one million data.

Can you help please.

Re: splunk remove data after indexing

haley_swarnapat — Tue, 20 Sep 2016 10:07:46 GMT

Have you set your search time range to "All Time" ?

If it doesn't work, then go to Settings -> Indexes -> click on your index
1. Check your index How big is "Current Size" and "Max Size"?

2. How many "Event Count" are shown?
You might need to add more space if your index is running out of space

Re: splunk remove data after indexing

TISKAR — Tue, 20 Sep 2016 10:34:58 GMT

Thank you for you reaction,
range=ALL Time
The Max Size=500GB
The current Size=1MB (I find it also removes all events)
Event Count=0

Re: splunk remove data after indexing

woodcock — Tue, 20 Sep 2016 13:12:11 GMT

Splunk is a FIFO system so if your index is set at a size of 80G and 10M events is roughly 1G, then the first (earliest) 10G will be frozen (purged) to make room for the last (latest) 10G.

Re: splunk remove data after indexing

TISKAR — Tue, 29 Sep 2020 11:02:47 GMT

Thank you for your reaction,
I create author index and , and I created a folder only contains three files, the Splunk began indexing but at some point it removes all that has indexed(COUNT EVENT=0) , knowing that I have not found this problem in Splunk light.

index characteristic:
range=ALL Time
The Max Size=500GB

index.conf:
[indexTest]
coldPath = $SPLUNK_DB/ffjj/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/indexTest/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/indexTest/thaweddb
disabled = 0

input.conf:
[monitor:///data/splunk/test]
disabled = false
index = indexTest
sourcetype = LICENCIE

Re: splunk remove data after indexing

woodcock — Tue, 20 Sep 2016 13:52:05 GMT

You have tagged this splunk-enterprise but then you mention "splunk light" What are you using?

Re: splunk remove data after indexing

TISKAR — Tue, 20 Sep 2016 14:09:39 GMT

Now I use splubk Entrprise . (before I used splunk light I has not found The Problem)

Re: splunk remove data after indexing

woodcock — Tue, 20 Sep 2016 14:14:22 GMT

What do you see with this search:

 | tstats count valuse(sourcetype) where index=* OR index=_*

Re: splunk remove data after indexing

TISKAR — Tue, 20 Sep 2016 14:37:51 GMT

this command gives 0 event and null values

Re: splunk remove data after indexing

haley_swarnapat — Wed, 21 Sep 2016 04:21:19 GMT

It's a typo, he was mentioning this:
| tstats count values(sourcetype) where index=* OR index=_*

Just FYI, if your free space falls below 5GB, Splunk will stop indexing by default.
If this is the case, you might need to delete some temp files from your OS.
Or you can adjust this limit by going to : Settings -> System Settings -> General Settings -> Pause indexing if free disk space (in MB) falls below *

Re: splunk remove data after indexing

TISKAR — Wed, 21 Sep 2016 13:26:58 GMT

after searching , I think the problem comes from three files , I install Splunk entreprise in other computer problems remains with me these three files , I replaced these files by three other files of the same format and size but different data, me the data are added without problem.

Thank you all

Re: splunk remove data after indexing

TISKAR — Tue, 29 Sep 2020 11:06:23 GMT

Hello every one,

Thank you for your participation,
I found the problem , but I did not understand why the problem is that I have more time , I added from the fields of these dates in TIMESTAMP_FIELDS after I left That a single TIMESTAMP_FIELDS field , I'll find out why . and what is the criteria on TIMESTAMP_FIELDS ??