Splunk Enterprise

Splunk doesn't index csv at all

fabrizioalleva
Path Finder

Hi all,

I'm working on a deploy with Universal Forwader, Heavy Forwarder and Indexer Cluster and Search Cluster.

The problem is this:

I'm indexing data from different csv since long time. For the first time yesterday I realized that not all the raw of my csv files are indexed at all. 

For example:

In a csv I count 24k rows and when I perform a stats count on the index I see only 16/17k rows.

Each file rotates every minutes. 

In the log there's anything that leads to an error.

In the UNIVERSAL FORWARDE I've this in inputs.conf

[batch:///var/opt/OV/shared/perfSpi/datafiles/metric/final/F5_ResurcesGroup*]
disabled = 0
index = f5_metrics
sourcetype = f5_metrics
initCrcLength = 100000
move_policy = sinkhole

In the HEAVY FORWARDER:

props.conf

[f5_metrics]
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER=1
HEADER_FIELD_DELIMITER =,
FIELD_DELIMITER=,
HEADER_FIELD_LINE_NUMBER = 0
SEDCMD-dropheader = s/^"Node.+//g
SEDCMD-select_fields = s/([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)/\1,\2,\4,\5,\9,\17,\18/g
#SEDCMD-select_fields = s/([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)/\1,\4,\5,\9,\17,\18/g
TRANSFORMS-f5_fields_name_extract=f5_fields_name_extract

and in the transform.conf 

[f5_fields_name_extract]
REGEX=([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)
FORMAT=NodeID::$1 TimeStamp::$2 period_length::$3 ltmVirtualServStatClientCurConns::$4 ltmVirtualServStatVsUsageRatio1m::$5 DisplayAttribute::$6 PollingInterval::$7
#FORMAT=NodeID::$1 period_length::$2 ltmVirtualServStatClientCurConns::$3 ltmVirtualServStatVsUsageRatio1m::$4 DisplayAttribute::$5 PollingInterval::$6
WRITE_META = true

Any suggestion ?

Thanks Fabrizio

 

 

 

Labels (2)
0 Karma
1 Solution

fabrizioalleva
Path Finder

Solved. by myself 

I've to set TIME_PREFIX in props.conf to instruct that the timestamp is the second field and not the first.

It's nice to note that the problem there is now that I've some object that has the ID similiar to a UNIX TIMESTAMP. 

 

Closed

View solution in original post

0 Karma

fabrizioalleva
Path Finder

Solved. by myself 

I've to set TIME_PREFIX in props.conf to instruct that the timestamp is the second field and not the first.

It's nice to note that the problem there is now that I've some object that has the ID similiar to a UNIX TIMESTAMP. 

 

Closed

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...