topic Re: Why search results adds up when source is modified? in Splunk Search

Why search results adds up when source is modified?

boney_s — Fri, 05 Dec 2014 09:36:45 GMT

Hello friends,

I have indexed my own .log file in to Splunk and there are about 10 events in that log files. I wonder why the number of events returned gets doubled if I make a change on the .log file manually (edit it on notepad and save). For example if i change the value of one field from A to B (field seperated by 😃 and let say there are x results, splunk will show x + x events for the new search.
somebody please explain me how splunk works in this scenario?. Thanks in advance.

Re: Why search results adds up when source is modified?

MuS — Fri, 05 Dec 2014 09:46:36 GMT

did you add a

crcSalt = <SOURCE>

in inputs.conf for this log?

Re: Why search results adds up when source is modified?

boney_s — Fri, 05 Dec 2014 10:38:37 GMT

No. I tried including that command to input.conf but still not working.

Re: Why search results adds up when source is modified?

MuS — Fri, 05 Dec 2014 10:41:30 GMT

don't do this, because this can cause exactly this - re-indexing files. Only include this if really needed.

Re: Why search results adds up when source is modified?

somesoni2 — Fri, 05 Dec 2014 20:39:40 GMT

When Splunk is monitoring a file or files in a folder, it creates a crc (Cyclic Redundancy Check) handler for each file, so as it will not re-index a file with same data (even though the file gets renamed). The attribute which handles this crc handler is called initCrcLength and is present in inputs.conf.

Default value for this property is 256 bytes (first 256 bytes in the file). So, if you change anything in the file which is within the limit set by initCrcLength attribute, Splunk will treat it as new file and will re-index all the entries (not just the updated entries).

Re: Why search results adds up when source is modified?

boney_s — Mon, 08 Dec 2014 05:50:25 GMT

Thank you guys. So i need to reduce the initCrcLength. BTW what is the minimum value for that parameter?

Re: Why search results adds up when source is modified?

somesoni2 — Mon, 08 Dec 2014 17:08:34 GMT

Per documentations, it should be in the range of 256-1048576 so the minimum value is the default value. Is it possible for you to change the way you update the file, means if you just want to update one entry, create a file with just that entry or something?

Re: Why search results adds up when source is modified?

boney_s — Tue, 09 Dec 2014 03:59:40 GMT

Thanks my friend.

Re: Why search results adds up when source is modified?

ngatchasandra — Mon, 28 Sep 2020 18:26:20 GMT

To answer this question, I have done the steps like follow:

First time, i created a index and i indexed a csv file with that index. When I start search by putting “index=test1” in search bar, the result give me 3 events:

9/15/14
10:52:20.411 AM
8750321,"*ALL",JPD910,"11484/6788",UNKNOWN,"Mon Sep 15 10:52:20.411000","IPCMISC.C299
State information for process 10036, User=8750321, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP06, Version=LIS0001, Thread ID=9280, Thread Name=WRK:8750321_09D55650_P5542319."
host = student16-PC source = APPCB02.csv sourcetype = csv

9/13/14
3:28:27.380 PM

6635103,"*ALL",JPD910,"2664/14920",UNKNOWN,"Sat Sep 13 15:28:27.380002","IPCMISC.C299
State information for process 7764, User=6635103, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP07, Version=LIS0001, Thread ID=12632, Thread Name=WRK:Starting jdeCallObject."
host = student16-PC source = APPCB02.csv sourcetype = csv

9/11/14
12:59:54.203 PM
6107085,"*ALL",JPD910,"12760/14360",UNKNOWN,"Thu Sep 11 12:59:54.203001","IPCMISC.C299
State information for process 10784, User=6107085, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP04, Version=LIS0001, Thread ID=13536, Thread Name=WRK:Starting jdeCallObject."
host = student16-PC source = APPCB02.csv sourcetype = csv
Second time, in by editing this csv file, I make a change of first value of first event like this:
8750321 its become 8750999.
When I re-indexed the same file with another index which I created and name it test2, the same search give me results like follow with 6 events:

9/15/14
10:52:20.411 AM
" State information for process 10036, User=8750321, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP06, Version=LIS0001, Thread ID=9280, Thread Name=WRK:8750321_09D55650_P5542319."""
host = student16-PC source = APPCB02.csv sourcetype = csv

9/15/14
10:52:20.411 AM
"8750999,""*ALL"",JPD910,""11484/6788"",UNKNOWN,""Mon Sep 15 10:52:20.411000"",""IPCMISC.C299"
host = student16-PC source = APPCB02.csv sourcetype = csv

9/13/14
3:28:27.380 PM

" State information for process 7764, User=6635103, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP07, Version=LIS0001, Thread ID=12632, Thread Name=WRK:Starting jdeCallObject."""
host = student16-PC source = APPCB02.csv sourcetype = csv

9/13/14
3:28:27.380 PM

"6635103,""*ALL"",JPD910,""2664/14920"",UNKNOWN,""Sat Sep 13 15:28:27.380002"",""IPCMISC.C299"
host = student16-PC source = APPCB02.csv sourcetype = csv

9/11/14
12:59:54.203 PM
" State information for process 10784, User=6107085, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP04, Version=LIS0001, Thread ID=13536, Thread Name=WRK:Starting jdeCallObject."""
host = student16-PC source = APPCB02.csv sourcetype = csv

9/11/14
12:59:54.203 PM
"6107085,""*ALL"",JPD910,""12760/14360"",UNKNOWN,""Thu Sep 11 12:59:54.203001"",""IPCMISC.C299"
host = student16-PC source = APPCB02.csv sourcetype = csv
In last, I make the same process with a index name test (index=test sourcetype=tes), but now, I has configured manually the sourcetype before saving. This give me the results like follow:

User,Role,Environment,PID,"Thread_Name","Date_Thread","File_Thread"
host = student16-PC source = APPCB02.csv sourcetype = tes

9/15/14
10:52:20.411 AM
8750321,"*ALL",JPD910,"11484/6788",UNKNOWN,"Mon Sep 15 10:52:20.411000","IPCMISC.C299
State information for process 10036, User=8750321, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP06, Version=LIS0001, Thread ID=9280, Thread Name=WRK:8750321_09D55650_P5542319."
host = student16-PC source = APPCB02.csv sourcetype = tes

9/13/14
3:28:27.380 PM

6635103,"*ALL",JPD910,"2664/14920",UNKNOWN,"Sat Sep 13 15:28:27.380002","IPCMISC.C299
State information for process 7764, User=6635103, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP07, Version=LIS0001, Thread ID=12632, Thread Name=WRK:Starting jdeCallObject."
host = student16-PC source = APPCB02.csv sourcetype = tes

9/11/14
12:59:54.203 PM
6107085,"*ALL",JPD910,"12760/14360",UNKNOWN,"Thu Sep 11 12:59:54.203001","IPCMISC.C299
State information for process 10784, User=6107085, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP04, Version=LIS0001, Thread ID=13536, Thread Name=WRK:Starting jdeCallObject."
host = student16-PC source = APPCB02.csv sourcetype = tes

This is the same result with first step only that, first event represent the head line of csv file.
In conclusion, I can say that, the search results adds up when source is modified because the sourcetype is not configured manually.

Note: If you don't specify a sourcetype when you searching, the events is adds(duplication) anytime you re-launch a search (index=test) like this: example, when i launch a same search of step3 without it sourcetype after one day, i obtain now 10 events like follow:

12/10/14 
1:47:56.000 PM  
User,Role,Environment,PID,"Thread_Name","Date_Thread","File_Thread"
host = student16-PC source = APPCB02.csv sourcetype = tes


9/15/14 
10:52:20.411 AM 
"   State information for process 10036, User=8750321, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP06, Version=LIS0001, Thread ID=9280, Thread Name=WRK:8750321_09D55650_P5542319."""
host = student16-PC source = C:\Users\student16\Desktop\APPCB022.csv sourcetype = csv


9/15/14 
10:52:20.411 AM 
"8758888,""*ALL"",JPD910,""11484/6788"",UNKNOWN,""Mon Sep 15 10:52:20.411000"",""IPCMISC.C299"
host = student16-PC source = C:\Users\student16\Desktop\APPCB022.csv sourcetype = csv


9/15/14 
10:52:20.411 AM 
8750321,"*ALL",JPD910,"11484/6788",UNKNOWN,"Mon Sep 15 10:52:20.411000","IPCMISC.C299
    State information for process 10036, User=8750321, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP06, Version=LIS0001, Thread ID=9280, Thread Name=WRK:8750321_09D55650_P5542319."
host = student16-PC source = APPCB02.csv sourcetype = tes


9/13/14 
3:28:27.380 PM  
"   State information for process 7764, User=6635103, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP07, Version=LIS0001, Thread ID=12632, Thread Name=WRK:Starting jdeCallObject."""
host = student16-PC source = C:\Users\student16\Desktop\APPCB022.csv sourcetype = csv


9/13/14 
3:28:27.380 PM  
"6635103,""*ALL"",JPD910,""2664/14920"",UNKNOWN,""Sat Sep 13 15:28:27.380002"",""IPCMISC.C299"
host = student16-PC source = C:\Users\student16\Desktop\APPCB022.csv sourcetype = csv


9/13/14 
3:28:27.380 PM  
6635103,"*ALL",JPD910,"2664/14920",UNKNOWN,"Sat Sep 13 15:28:27.380002","IPCMISC.C299
    State information for process 7764, User=6635103, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP07, Version=LIS0001, Thread ID=12632, Thread Name=WRK:Starting jdeCallObject."
host = student16-PC source = APPCB02.csv sourcetype = tes


9/11/14 
12:59:54.203 PM 
"   State information for process 10784, User=6107085, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP04, Version=LIS0001, Thread ID=13536, Thread Name=WRK:Starting jdeCallObject."""
host = student16-PC source = C:\Users\student16\Desktop\APPCB022.csv sourcetype = csv


9/11/14 
12:59:54.203 PM 
"6107085,""*ALL"",JPD910,""12760/14360"",UNKNOWN,""Thu Sep 11 12:59:54.203001"",""IPCMISC.C299"
host = student16-PC source = C:\Users\student16\Desktop\APPCB022.csv sourcetype = csv


9/11/14 
12:59:54.203 PM 
6107085,"*ALL",JPD910,"12760/14360",UNKNOWN,"Thu Sep 11 12:59:54.203001","IPCMISC.C299
    State information for process 10784, User=6107085, Role=*ALL, Environment=JPD910, Profile=NONE, Application=P5542319, Client Machine=WKCLSE1WEBCP04, Version=LIS0001, Thread ID=13536, Thread Name=WRK:Starting jdeCallObject."
host = student16-PC source = APPCB02.csv sourcetype = tes

Re: Why search results adds up when source is modified?

boney_s — Mon, 15 Dec 2014 12:07:44 GMT

Hello Guys,

I have one more question related to the one above .I have indexed one DB table in splunk and its data changes frequently. Why splunk shows previously stored results even if the DB table is empty. Is there any way to show real time DB table values in Splunk?. I.e if table has 10 rows splunk should show "10 events" and if table has 0 rows splunk should show "0 events". Is that possible?

Thanks in advance

Re: Why search results adds up when source is modified?

MuS — Mon, 15 Dec 2014 12:35:48 GMT

read the docs about DB connect http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Installtheconnector and the dbquery command http://docs.splunk.com/Documentation/DBX/1.1.6/DeployDBX/Commands

Re: Why search results adds up when source is modified?

boney_s — Tue, 16 Dec 2014 04:58:50 GMT

Thanks man that really saved my time.
BTW what is the minimum time I can give for a "scheduled report". Right now I am using */1 * * * * (cron) for running it every 1 minute I guess. Is it possible to reduce this time to seconds range.