Getting Data In

How to edit my props.conf to make sure CSV file data gets indexed?

yanivdutt
Explorer

Hi,
I am using below props file for CSV but data is not getting indexed or sent into Splunk. Need help in updating props

[data_csv]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
HEADER_MODE = firstline
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
PREAMBLE_REGEX = ^\tDate
FIELD_NAMES = [SiteID,Domain,TotalRequests,SavedRequests,TotalBytes,SavedBytes,HitTime,BitsPerSecond,Human requests per second,Bot requests per second,Blocked requests per second,BlockedRequests]

Below is data from CSV

SiteID  Domain  TotalRequests   SavedRequests   TotalBytes  SavedBytes  HitTime BitsPerSecond   Human requests per second   Bot requests per second Blocked requests per second BlockedRequests
1001602 yahoo.com   0   0   0   0                       
1028875 yahoo2.com  0   0   0   0                       
1033027 yahoo1.com  3088    656 52976733    23001447    10/12/16 0:00   963 0   0   0   0
1033027 yahoo.com                   10/12/16 0:10   1350    0   0   0   0
0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi Yanivdutt!

I would strongly recommend you run one of your sample CSV through the Add Data wizard on one of your searchheads.

http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Howdoyouwanttoadddata

It will allow you to test these settings and allow you to preview how your data looks once they have been applied!

http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Setsourcetype

I think you may have a couple conflicting settings....

Is the sample you pasted exactly what the file looks like? Can you share one with the preamble? I can run it through the add data wiz and show you what I come up with.

- MattyMo
0 Karma

yanivdutt
Explorer

SiteID Domain TotalRequests SavedRequests TotalBytes SavedBytes HitTime BitsPerSecond Human requests per second Bot requests per second Blocked requests per second BlockedRequests
1001602 linelink.yahoo.com 0 0 0 0

1028875 npe-facade.api.yahoo.com 0 0 0 0

1033027 ma.my.yahoo.com 3088 656 52976733 23001447 10/12/16 0:00 963 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 0:10 1350 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 0:20 91 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 0:30 251 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 0:40 581 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 17:50 74 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 18:00 25109 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 18:10 21946 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 18:20 73558 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 18:30 7011 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 18:40 7781 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 18:50 27902 0 0 0 0
1036959 e2e.commerce.digital.yahoo.com 0 0 0 0

0 Karma

mattymo
Splunk Employee
Splunk Employee

Ah, just noticed...thats not a csv at all...appears semi space delimited, although I am not sure I'd even call it that....

You are going to have issues with this data in this format as there are spaces in the headers and there are fields that don't get populated and dont contain nulll values...

for example...your first 7 headers don't match all of your events...

SiteID Domain TotalRequests SavedRequests TotalBytes SavedBytes HitTime
1033027 ma.my.yahoo.com 10/12/16 0:10 1350 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 0:20 91 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 0:30 251 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 0:40 581 0 0 0 0
1033027 ma.my.yahoo.com 10/12/16 17:50 74 0 0 0 0

Is there anyway to clean up the format of this data??? What is the source of this data?

At this point index time csv fields are not going to work on this...you may be able to do some searchtime magic...but truly, this data needs to be formatted better

Even Excel doesn't like it lol

alt text

- MattyMo
0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...