Getting Data In

All files are not indexing

Explorer

Dear All,

I have basic 4 types of files. under

   C:\Program Files\Splunk\etc\apps\my logs\home_logs\KLZ\host1\

    abck_KLZ_CPU_110213.csv
    abck01_KLZ_Disk_110213.csv
    abck01_KLZ_Network_110213.csv
    abck01_KLZ_Swap_Rate_110213.csv
......

    C:\Program Files\Splunk\etc\apps\my logs\home_logs\KLZ\host2\

    defg_KLZ_CPU_110213.csv
    defg01_KLZ_Disk_110213.csv
    defg01_KLZ_Network_110213.csv
    defg01_KLZ_Swap_Rate_110213.csv
......
.....
.....

There are lots of host and its corrosponding files. I wanted 4 types of sourcetype ie. network, cpu, swap-rate and disk. and host name should be host1,host2,.... for this my conf files are as below: inputs.conf:

[monitor:C:\Program Files\Splunk\etc\apps\my logs\home_logs\..\..\*]
disabled = false
index = my_indx
host_segment = 8

props.conf:

CHARSET=SHIFT-JIS 

[source::...Disk...]
sourcetype = disk

[source::...CPU...]
sourcetype = cpu

[source::...Network...]
sourcetype = network

[source::...Swap_Rate...]
sourcetype = swap-rate

The result using above confs. Only 1 directory getting indexed. and for other directories only 1 file (same sourcetype) getting indexed. Other files are not indexed.

Hope you understand my problem. Your help will be appreciated.

Thanks in advance

Tags (1)

Path Finder

Hi msona,

  1. I think the application name is wrong.
    Application name can only contain the following characters: a-zA-Z0-9_-

  2. Please check your splunkd.log (C:\Program Files\Splunk\var\log\splunk\splunkd.log)
    If there are error messages such as bellow, please add the setting in configuration: "crcSalt = <SOURCE>" In inputs.conf

02-28-2011 15:00:00.000 ERROR TailingProcessor - Ignoring path due to: File will not be read, seekptr checksum did not match (file=XXXXX). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

*Note: For more information, please see: http://www.splunk.com/base/Documentation/latest/Admin/Monitorfilesanddirectories#Monitor_syntax_and_...

Splunk Employee
Splunk Employee

Your posted config should work, except for the monitor. If I understand what you're trying to accomplish, I would expect it to look like this:

[monitor://C:\Program Files\Splunk\etc\apps\my logs\home_logs\]
disabled = false
recursive = true
host_segment = 8
index = my_indx
crcSalt = <SOURCE>

EDIT: added crcSalt to resolve 'too small' error.

This will index everything in the home_logs folder and all sub-folders. Recursive is the default, but declaring it will help ensure there's not an override somewhere else.

Splunk Employee
Splunk Employee

I'll edit the answer to show how to add the CRC salt. Use as the value.

0 Karma

Explorer

I am getting the error while indexing:

02-25-2011 19:41:58.030 ERROR TailingProcessor - Ignoring path due to: File will not be read, is too small to match seekptr checksum (file=C:\EDN\test01\kednwbs01KLZDisk110213.csv). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submitissue for more info.

0 Karma

Path Finder

Hi msona,

What do you think of this approach:

inputs.conf
-----------
[monitor:C:\Program Files\Splunk\etc\apps\my logs\home_logs\...]
disabled = false
index = my_indx


props.conf
----------
[source::C:\Program Files\Splunk\etc\apps\my logs\home_logs\...]
TRANSFORMS-checkpath = sourcetype-transform, host-transform


transforms.conf
---------------
[sourcetype-transform]
SOURCE_KEY = MetaData:Source

#         defg _ KLZ _  CPU _110213.csv
REGEX= .*/[^_]+_[^_]+_([^_\d]+).*\.csv

# just for the case the regex doesn't match
#DEFAULT_VALUE = sourcetype::default-sourcetype

DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$1



[host-transform]
SOURCE_KEY = MetaData:Source

#           defg _ KLZ _  CPU _110213.csv
REGEX= .*/([^_])+_[^_]+_[^_\d]+.*\.csv

# just for the case the regex doesn't match
#DEFAULT_VALUE = host::default-host

DEST_KEY = MetaData:Host
FORMAT = sourcetype::$1

obs: I did not test this configuration and maybe there are small errors in it. But I want to show that you can examine the path (MetaData:Source) of your input to create or modify basic fields (source, sourcetype, host) at index time.

Explorer

Can Anybody Help me Please ????

0 Karma

Explorer

for the host name I have added
host_segment = 8 in the input.conf

0 Karma

Explorer

Hi meno,

Thanks for the answer.
I checked but its not working :(. Its taking default host and source type.
I want the host as directory name and sourcetype as some part of file name.

0 Karma