Splunk Enterprise

Splunk doesn't index csv at all

fabrizioalleva
Path Finder

Hi all,

I'm working on a deploy with Universal Forwader, Heavy Forwarder and Indexer Cluster and Search Cluster.

The problem is this:

I'm indexing data from different csv since long time. For the first time yesterday I realized that not all the raw of my csv files are indexed at all. 

For example:

In a csv I count 24k rows and when I perform a stats count on the index I see only 16/17k rows.

Each file rotates every minutes. 

In the log there's anything that leads to an error.

In the UNIVERSAL FORWARDE I've this in inputs.conf

[batch:///var/opt/OV/shared/perfSpi/datafiles/metric/final/F5_ResurcesGroup*]
disabled = 0
index = f5_metrics
sourcetype = f5_metrics
initCrcLength = 100000
move_policy = sinkhole

In the HEAVY FORWARDER:

props.conf

[f5_metrics]
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER=1
HEADER_FIELD_DELIMITER =,
FIELD_DELIMITER=,
HEADER_FIELD_LINE_NUMBER = 0
SEDCMD-dropheader = s/^"Node.+//g
SEDCMD-select_fields = s/([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)/\1,\2,\4,\5,\9,\17,\18/g
#SEDCMD-select_fields = s/([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)/\1,\4,\5,\9,\17,\18/g
TRANSFORMS-f5_fields_name_extract=f5_fields_name_extract

and in the transform.conf 

[f5_fields_name_extract]
REGEX=([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)
FORMAT=NodeID::$1 TimeStamp::$2 period_length::$3 ltmVirtualServStatClientCurConns::$4 ltmVirtualServStatVsUsageRatio1m::$5 DisplayAttribute::$6 PollingInterval::$7
#FORMAT=NodeID::$1 period_length::$2 ltmVirtualServStatClientCurConns::$3 ltmVirtualServStatVsUsageRatio1m::$4 DisplayAttribute::$5 PollingInterval::$6
WRITE_META = true

Any suggestion ?

Thanks Fabrizio

 

 

 

Labels (2)
0 Karma
1 Solution

fabrizioalleva
Path Finder

Solved. by myself 

I've to set TIME_PREFIX in props.conf to instruct that the timestamp is the second field and not the first.

It's nice to note that the problem there is now that I've some object that has the ID similiar to a UNIX TIMESTAMP. 

 

Closed

View solution in original post

0 Karma

fabrizioalleva
Path Finder

Solved. by myself 

I've to set TIME_PREFIX in props.conf to instruct that the timestamp is the second field and not the first.

It's nice to note that the problem there is now that I've some object that has the ID similiar to a UNIX TIMESTAMP. 

 

Closed

0 Karma
Get Updates on the Splunk Community!

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...

From Alert to Resolution: How Splunk Observability Helps SREs Navigate Critical ...

It's 3:17 AM, and your phone buzzes with an urgent alert. Wire transfer processing times have spiked, and ...