Splunk doesn't index csv at all

fabrizioalleva — Fri, 17 Jun 2022 14:25:43 GMT

Hi all,

I'm working on a deploy with Universal Forwader, Heavy Forwarder and Indexer Cluster and Search Cluster.

The problem is this:

I'm indexing data from different csv since long time. For the first time yesterday I realized that not all the raw of my csv files are indexed at all.

For example:

In a csv I count 24k rows and when I perform a stats count on the index I see only 16/17k rows.

Each file rotates every minutes.

In the log there's anything that leads to an error.

In the UNIVERSAL FORWARDE I've this in inputs.conf

[batch:///var/opt/OV/shared/perfSpi/datafiles/metric/final/F5_ResurcesGroup*]
disabled = 0
index = f5_metrics
sourcetype = f5_metrics
initCrcLength = 100000
move_policy = sinkhole

In the HEAVY FORWARDER:

props.conf

[f5_metrics]
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER=1
HEADER_FIELD_DELIMITER =,
FIELD_DELIMITER=,
HEADER_FIELD_LINE_NUMBER = 0
SEDCMD-dropheader = s/^"Node.+//g
SEDCMD-select_fields = s/([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)/\1,\2,\4,\5,\9,\17,\18/g
#SEDCMD-select_fields = s/([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)/\1,\4,\5,\9,\17,\18/g
TRANSFORMS-f5_fields_name_extract=f5_fields_name_extract

and in the transform.conf

[f5_fields_name_extract]
REGEX=([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)
FORMAT=NodeID::$1 TimeStamp::$2 period_length::$3 ltmVirtualServStatClientCurConns::$4 ltmVirtualServStatVsUsageRatio1m::$5 DisplayAttribute::$6 PollingInterval::$7
#FORMAT=NodeID::$1 period_length::$2 ltmVirtualServStatClientCurConns::$3 ltmVirtualServStatVsUsageRatio1m::$4 DisplayAttribute::$5 PollingInterval::$6
WRITE_META = true

Any suggestion ?

Thanks Fabrizio

Re: Splunk doesn't index csv at all

fabrizioalleva — Fri, 17 Jun 2022 17:44:21 GMT

Solved. by myself

I've to set TIME_PREFIX in props.conf to instruct that the timestamp is the second field and not the first.

It's nice to note that the problem there is now that I've some object that has the ID similiar to a UNIX TIMESTAMP.

Closed

topic Re: Splunk doesn't index csv at all in Splunk Enterprise

Splunk doesn't index csv at all

Re: Splunk doesn't index csv at all