topic Re: CSV Japanese header extraction in Getting Data In

CSV Japanese header extraction

msona — Fri, 11 Feb 2011 22:44:42 GMT

Dear all,

I want to extract the Japanese CSV header from csv log file. I am configurations are as follows.

inputs.conf
---------------------------------------
[monitor:C:\Program Files\Splunk\etc\apps\tougou\tougou_logs\*.csv]
disabled = false
host =My-PC
index =myindex

props.conf
[source::...CPU...]
sourcetype = cpu

[source::...Disk...]
sourcetype = disk

[source::...Network...]
sourcetype = network

[source::...Swap_Rate...]
sourcetype = swap-rate


[disk]
CHECK_FOR_HEADER=TRUE
 #REPORT-disk=argus_extractions_disk

[network]
CHECK_FOR_HEADER=TRUE
#REPORT-network=argus_extractions_network

[cpu]
CHECK_FOR_HEADER=TRUE
#REPORT-cpu=argus_extractions_cpu

[swap-rate]
CHECK_FOR_HEADER=TRUE
#REPORT-swap-rate=argus_extractions_swap_rate


----------------------------------------------------
transforms.conf
----------------------------------------------------
[argus_extractions_disk]
DELIMS=","
FIELDS="タイムゾーン","記録時間","システム名","タイム・スタンプ","ディスク名","マウント・ポイント","ファイル・システム・タイプ","サイズ (MB)","使用 ディスク (MB)","使用 ディスク 率","フリー・ ディスク (MB)","フリー・ ディスク 率","合計 i ノード","使用済み i ノード数","i ノード使用率","空き i ノード","フリーの i ノード 率"

[argus_extractions_network]
DELIMS=","
FIELDS="タイムゾーン","記録時間","システム名","タイム・スタンプ","ネットワーク・インターフェース名","IPアドレス","インターフェース状況","最大転送単位","受信 数 (KB)","1 秒当たりの 受信 バイト","送信 数 (KB)","1 秒当たりの 送信 バイト","受信パケット","1 秒当たりの 受信 パケット","送信パケット","1 秒当たりの 送信 パケット","入力エラー","出力エラー","合計衝突","衝突  (分あたり)","衝突率","入力エラー (分あたり)","出力エラー (分あたり)","エラー (%)","ドロップした入力パケット","ドロップした出力パケット","入力 FIFO バッファー・オーバーラン","出力 FIFO バッファー・オーバーラン","パケット・フレーム・エラー","キャリア・ロス","入力エラー (%)","出力エラー (%)","デバイス・タイプ","MACアドレス"

[argus_extractions_cpu]
DELIMS=","
FIELDS="タイムゾーン","記録時間","システム名","タイム・スタンプ","CPU ID","ユーザー CPU (%)","ユーザー・ナイス CPU (%)","システム CPU (%)","アイドル CPU (%)","使用中の CPU (%)","I/O 待機率 (%)","システム CPU に対するユーザー (%)"

[argus_extractions_swap_rate]
DELIMS=","
FIELDS="タイムゾーン","記録時間","システム名","タイム・スタンプ","合計 スワップ・スペース  (MB)  (移動平均)","使用 スワップ・スペース  (MB)  (移動平均)","使用 スワップ・ スペース (バイト/時)","スワップ・スペースが いっぱいになる までの日数","スワップ・ スペース使用の ピーク  (MB)","スワップがフルになるまでの最小日数","空いている 実メモリーの 最低水準点  (KB)"

Your help appreaciate.

Regards newbie

Re: CSV Japanese header extraction

Hajime — Mon, 14 Feb 2011 12:25:00 GMT

Hi, Unfortunately you can not use Japanese field names.

Splunk only accepts field names that contain alpha-numeric characters or an underscore.

For more information, please see:
http://www.splunk.com/base/Documentation/latest/Admin/Configureindex-timefieldextraction#Define_additional_indexed_fields

Re: CSV Japanese header extraction

msona — Tue, 15 Feb 2011 13:50:03 GMT

Hello,

Thanks for the answer but How can I remove this header ?
I have written English fields so that I can able to extract all the fields. But I am getting that header in fields. How I can remove that header from fields ???

Re: CSV Japanese header extraction

Hajime — Wed, 16 Feb 2011 09:38:06 GMT

If the header starts from "タイムゾーン,", you can remove that header in the following way.

props.conf: (e.g. sourcetype=cpu)

[cpu]
TRANSFORMS-null = setnull

transforms.conf:

[setnull]
REGEX = ^タイムゾーン,.*
DEST_KEY = queue
FORMAT = nullQueue

Please try, should work.

Re: CSV Japanese header extraction

msona — Mon, 28 Sep 2020 09:25:00 GMT

Dear Hajime San,
Thank you very much for quick answers. I have tried with above but only one source type removed. Here is my configuration.

props.conf
[disk]
REPORT-disk=argus_extractions_disk
TRANSFORMS-null= setnull

[network]
REPORT-network=argus_extractions_network
TRANSFORMS-null= setnull

[cpu]
REPORT-cpu=argus_extractions_cpu
TRANSFORMS-null= setnull

[swap-rate]
REPORT-swap-rate=argus_extractions_swap_rate
TRANSFORMS-null= setnull

transforms.conf:
[setnull]
REGEX = ^タイムゾーン,*
DEST_KEY = queue
FORMAT = nullQueue

Re: CSV Japanese header extraction

Hajime — Thu, 17 Feb 2011 17:09:34 GMT

Hello. As you point out, my examples are applied to one source type.
I think your settings are correct.

Re: CSV Japanese header extraction

msona — Fri, 18 Feb 2011 10:29:52 GMT

Hi, yes I understood, but my all sorurcetypes are starting from タイムゾーン, so REGEX is same for all source type. Should I write different name only ?

Re: CSV Japanese header extraction

Hajime — Fri, 18 Feb 2011 12:18:56 GMT

Yes. You should be set individually.
However, you can use [default] to set globally.
For example, In props.conf:
[default]
TRANSFORMS-null= setnull

Re: CSV Japanese header extraction

msona — Mon, 28 Sep 2020 09:25:28 GMT

Hi Hajime san,

I am facing the problem that all the data is not getting index and follwoing error I got in splunkd.log :

02-25-2011 19:41:58.030 ERROR TailingProcessor - Ignoring path due to: File will not be read, is too small to match seekptr checksum (file=C:\EDN\test01\kednwbs01_KLZ_Disk_110213.csv). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Re: CSV Japanese header extraction

Hajime — Mon, 28 Feb 2011 14:56:27 GMT

Please try to add the setting in configuration: "crcSalt = " In inputs.conf --- For more information, please see: http://www.splunk.com/base/Documentation/latest/Admin/Monitorfilesanddirectories#Monitor_syntax_and_examples

Re: CSV Japanese header extraction

Suda — Wed, 06 Apr 2011 15:40:35 GMT

Hello,

You might be able to extract fields with using Japanese field names in this case.

I don't recommend it.So, Hajime-san is right. But... you can (I confirm it with Splunk 4.1.7.)

And you keep in mind that Splunk cannot handle any field names containing space characters. Could you try to change " "(space) to "_"(underscore) in transforms.conf settings?

For example,

[argus_extractions_disk]
DELIMS=","
FIELDS="タイムゾーン","記録時間","システム名","タイム・スタンプ","ディスク名","マウント・ポイント","ファイル・システム・タイプ","サイズ_(MB)","使用_ディスク_(MB)","使用 ディスク_率","フリー・_ディスク (MB)","フリー・_ディスク_率","合計_i_ノード","使用済み_i_ノード数","i_ノード使用率","空き_i_ノード","フリーの_i_ノード_率"

I hope it will help your splunking!

Thanks.

Kenichi Suda