topic Re: All files are not indexing in Getting Data In

All files are not indexing

msona — Tue, 22 Feb 2011 15:37:47 GMT

Dear All,

I have basic 4 types of files. under

   C:\Program Files\Splunk\etc\apps\my logs\home_logs\KLZ\host1\

    abck_KLZ_CPU_110213.csv
    abck01_KLZ_Disk_110213.csv
    abck01_KLZ_Network_110213.csv
    abck01_KLZ_Swap_Rate_110213.csv
......

    C:\Program Files\Splunk\etc\apps\my logs\home_logs\KLZ\host2\

    defg_KLZ_CPU_110213.csv
    defg01_KLZ_Disk_110213.csv
    defg01_KLZ_Network_110213.csv
    defg01_KLZ_Swap_Rate_110213.csv
......
.....
.....

There are lots of host and its corrosponding files. I wanted 4 types of sourcetype ie. network, cpu, swap-rate and disk. and host name should be host1,host2,.... for this my conf files are as below: inputs.conf:

[monitor:C:\Program Files\Splunk\etc\apps\my logs\home_logs\..\..\*]
disabled = false
index = my_indx
host_segment = 8

props.conf:

CHARSET=SHIFT-JIS 

[source::...Disk...]
sourcetype = disk

[source::...CPU...]
sourcetype = cpu

[source::...Network...]
sourcetype = network

[source::...Swap_Rate...]
sourcetype = swap-rate

The result using above confs. Only 1 directory getting indexed. and for other directories only 1 file (same sourcetype) getting indexed. Other files are not indexed.

Hope you understand my problem. Your help will be appreciated.

Thanks in advance

Re: All files are not indexing

meno — Tue, 22 Feb 2011 23:50:16 GMT

Hi msona,

What do you think of this approach:

inputs.conf
-----------
[monitor:C:\Program Files\Splunk\etc\apps\my logs\home_logs\...]
disabled = false
index = my_indx


props.conf
----------
[source::C:\Program Files\Splunk\etc\apps\my logs\home_logs\...]
TRANSFORMS-checkpath = sourcetype-transform, host-transform


transforms.conf
---------------
[sourcetype-transform]
SOURCE_KEY = MetaData:Source

#         defg _ KLZ _  CPU _110213.csv
REGEX= .*/[^_]+_[^_]+_([^_\d]+).*\.csv

# just for the case the regex doesn't match
#DEFAULT_VALUE = sourcetype::default-sourcetype

DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$1



[host-transform]
SOURCE_KEY = MetaData:Source

#           defg _ KLZ _  CPU _110213.csv
REGEX= .*/([^_])+_[^_]+_[^_\d]+.*\.csv

# just for the case the regex doesn't match
#DEFAULT_VALUE = host::default-host

DEST_KEY = MetaData:Host
FORMAT = sourcetype::$1

obs: I did not test this configuration and maybe there are small errors in it. But I want to show that you can examine the path (MetaData:Source) of your input to create or modify basic fields (source, sourcetype, host) at index time.

Re: All files are not indexing

msona — Wed, 23 Feb 2011 14:43:29 GMT

Hi meno,

Thanks for the answer.
I checked but its not working :(. Its taking default host and source type.
I want the host as directory name and sourcetype as some part of file name.

Re: All files are not indexing

msona — Wed, 23 Feb 2011 14:52:28 GMT

for the host name I have added
host_segment = 8 in the input.conf

Re: All files are not indexing

msona — Fri, 25 Feb 2011 13:02:10 GMT

Can Anybody Help me Please ????

Re: All files are not indexing

Ron_Naken — Sat, 26 Feb 2011 16:32:14 GMT

Your posted config should work, except for the monitor. If I understand what you're trying to accomplish, I would expect it to look like this:

[monitor://C:\Program Files\Splunk\etc\apps\my logs\home_logs\]
disabled = false
recursive = true
host_segment = 8
index = my_indx
crcSalt = <SOURCE>

EDIT: added crcSalt to resolve 'too small' error.

This will index everything in the home_logs folder and all sub-folders. Recursive is the default, but declaring it will help ensure there's not an override somewhere else.

Re: All files are not indexing

msona — Mon, 28 Sep 2020 09:25:24 GMT

I am getting the error while indexing:

02-25-2011 19:41:58.030 ERROR TailingProcessor - Ignoring path due to: File will not be read, is too small to match seekptr checksum (file=C:\EDN\test01\kednwbs01_KLZ_Disk_110213.csv). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Re: All files are not indexing

Hajime — Mon, 28 Feb 2011 14:51:27 GMT

Hi msona,

I think the application name is wrong.
Application name can only contain the following characters: a-zA-Z0-9_-
Please check your splunkd.log (C:\Program Files\Splunk\var\log\splunk\splunkd.log)
If there are error messages such as bellow, please add the setting in configuration: "crcSalt = <SOURCE>" In inputs.conf

02-28-2011 15:00:00.000 ERROR TailingProcessor - Ignoring path due to: File will not be read, seekptr checksum did not match (file=XXXXX). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

*Note: For more information, please see: http://www.splunk.com/base/Documentation/latest/Admin/Monitorfilesanddirectories#Monitor_syntax_and_examples

Re: All files are not indexing

Ron_Naken — Mon, 28 Feb 2011 15:49:06 GMT

I'll edit the answer to show how to add the CRC salt. Use as the value.