topic Re: Index data from csv with 1 column in Getting Data In

Is is possible to index a small CSV file with only 1 column (both with monitoring and manually)?

rayar — Thu, 03 Mar 2022 17:33:53 GMT

I am trying to index a small CSV file with only 1 column (both with monitoring and manually )

is it impossible ?

was able to index only after I added additional column

for monitoring I have defined the below

inputs.conf

[monitor:///opt/mailboxes_not_created_empid/*.csv]
disabled = 0
sourcetype = csv_current_time
index = mailboxes_not_created_empid
crcSalt = <SOURCE>
initCrcLength = 512

the csv (comma separated ) file is

Employee_Number

141941

180536

189377

Re: Index data from csv with 1 column

gcusello — Thu, 03 Mar 2022 11:01:59 GMT

Hi @rayar,

yes, it's possible, what's the problem you encountered?

Having only one clumn you don't need INDEXED_EXTRACTION.

Ciao.

Giuseppe

Re: Index data from csv with 1 column

rayar — Thu, 03 Mar 2022 11:06:48 GMT

no data is indexed into index

sorry , what do you mean you don't need INDEXED_EXTRACTION ?

Re: Index data from csv with 1 column

gcusello — Thu, 03 Mar 2022 11:39:52 GMT

Hi @rayar,

if you don't have data in your index, I need to know, at first, the file to ingest is in the same Splunk server on in another one with a Universal Forwarder?

if it's in the UF, you have at first to check the connection and you can do this in two ways:

check if you already have in the _internal index the UF Splunk internal logs:

index=_internal host=your_host

if you have events, connection is ok otherwise, you have to check the connection using telnet

telnet IP_Server_Splunk 9997

if you don't have the connection, you have do some things:

to check firewalls routes between UF and Indexer,
to check local firewalls,
to check the outputs.conf configuration,
to enable Log Reaceiving on Indexer.

If instead you have internal Splunk logs, this means that connection is esteblished and you have to check the file reading, you can do this with a CLI command on the server where the file is located:

ls -la /opt/mailboxes_not_created_empid/*.csv

checking also the reading grants.

Tell me your checks.

Ciao.

Giuseppe

Re: Index data from csv with 1 column

gcusello — Thu, 03 Mar 2022 11:41:31 GMT

Hio @rayar,

sorry I forgot INDEXED_EXTRACTION: it's a way to assign fields to csv or json files, but, having you only one column it could be not relevant.

Ciao.

Giuseppe

Re: Index data from csv with 1 column

rayar — Thu, 03 Mar 2022 12:15:40 GMT

its an active UF

I see the logs in _internal

03-03-2022 13:03:59.915 +0200 DEBUG TailingProcessor - Deferred notification for path='/opt/mailboxes_not_created_empid/Employee_Numbers_15012022_183613.csv'.

03-03-2022 13:03:56.916 +0200 DEBUG TailingProcessor - File state notification for path='/opt/mailboxes_not_created_empid/Employee_Numbers_15012022_183613.csv' (first time).

Re: Index data from csv with 1 column

gcusello — Thu, 03 Mar 2022 12:50:41 GMT

Hi @rayar,

Please some other details:

did you located the inputs.conf in the UF?
Where did you located inputs.conf, in which folder?
Did you used a Deployment Server or did you manually modified inputs.conf?
Did you restarted Spluink on UF after inputs.conf modification?

Ciao.

Giuseppe

Re: Index data from csv with 1 column

rayar — Thu, 03 Mar 2022 12:55:11 GMT

did you located the inputs.conf in the UF? - yes
Where did you located inputs.conf, in which folder? - /opt/splunkforwarder/etc/apps/search/local/inputs.conf"
Did you used a Deployment Server or did you manually modified inputs.conf? - manual
Did you restarted Spluink on UF after inputs.conf modification? - yes

Also once I do "Add Data" manually from the SH with csv sourcetype I can't index , only if I add another column

I think the issue is in indexing 1 column csv file

Re: Index data from csv with 1 column

gcusello — Thu, 03 Mar 2022 13:04:09 GMT

Hi @rayar,

I suppose that you checked the localization of the csv files using the ls command I hinted in the first answer.

Did you tried to copy a csv file in another one with a different filename?

Obviously index mailboxes_not_created_empid is created and available.

What does it happen tryng to ingest by GUI on the SH?

Ciao.

Giuseppe

Re: Index data from csv with 1 column

rayar — Thu, 03 Mar 2022 15:10:33 GMT

I tried to index the file manually from SH using CSV sourcetype and it got indexed only after I added a new column since its looking for ,

Re: Index data from csv with 1 column

gcusello — Thu, 03 Mar 2022 15:34:19 GMT

Hi @rayar,

I triend to load a csv file with only one column.

It runs, using this props.conf on Indexer.

[test_csv] DATETIME_CONFIG=CURRENT SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true CHARSET=AUTO INDEXED_EXTRACTIONS=csv KV_MODE=none category=Structured description=Comma-separated value format. Set header and other settings in "Delimited Settings" disabled=false pulldown_type=true

Ciao.

Giuseppe

Re: Index data from csv with 1 column

rayar — Thu, 10 Mar 2022 11:44:18 GMT

I used the below

[csv_1_column_test]
BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG = CURRENT
KV_MODE = none
LINE_BREAKER = [(\n\r]+)
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
CHARSET = AUTO
HEADER_FIELD_LINE_NUMBER = 1
INDEXED_EXTRACTIONS = csv
NO_BINARY_CHECK = true

Re: Index data from csv with 1 column

gcusello — Thu, 10 Mar 2022 11:46:34 GMT

JI @rayar,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉