topic Re: CSV input, only headers in Getting Data In

CSV input, only headers

zidoz — Thu, 02 Jul 2020 12:07:43 GMT

Hi all,

I've configured a universal forwarder on Windows server to monitor a folder with csv files.

These files are logs from our mail relay system, so they are being written regularly.

I can see the files in my Splunk Search head, but only the title of the columns, not the data itself

I've configured the sourcetype as CSV, added crcSalt=<SOURCE> to the inputs configuration on the Windows Server.

Does anyone have any idea why I'm only getting the headers?

Thanks all

Re: CSV input, only headers

gcusello — Thu, 02 Jul 2020 12:11:40 GMT

Hi @zidoz ,

could you share your props.conf?

Probably the problem is on the props.conf file, where do you have it?

Remember that ingesting csv files, props.conf must be on UF and Indexer (or Heavy Forwarder when present).

Ciao.

Giuseppe

Re: CSV input, only headers

zidoz — Thu, 02 Jul 2020 12:56:44 GMT

Hi @gcusello ,

I have a props.conf file on the UF under the path below:

C:\Program Files\SplunkUniversalForwarder\etc\system\default

And I have one on the indexer under:

/opt/splunk/etc/system/default

Which one would you like to see?

Re: CSV input, only headers

gcusello — Thu, 02 Jul 2020 13:21:56 GMT

Hi @zidoz ,

files in $SPLUNK_HOME/etc/system default cnnot be changed!

this means that you haven't a props.conf.

So download a copy of the csv file and following the guided web procedure [Settings -- Add data] find the correct configuration for you input.

Then copy this pros.conf file on Indexer and on Universal Forwarder, not on default folder: create your own app.

Then restart Splunk on both the systems.

For more infos see at https://docs.splunk.com/Documentation/Splunk/8.0.4/Data/Extractfieldsfromfileswithstructureddata

Ciao.

Giuseppe

Re: CSV input, only headers

zidoz — Sun, 05 Jul 2020 10:09:19 GMT

Hi @gcusello ,

I think I copied the right prop.conf file to both the indexer and UF, but I'm still getting only the headers.

The data I copied is attached below:

# Version 7.3.4

[splunkd]
EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) - (?P<event_message>.+)

[scheduler]
EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) - (?P<event_message>.+)

[splunk_web_service]
EXTRACT-useragent = userAgent=(?P<browser>[^ (]+)
[user@splunk-indexer apps]$ cat search/local/props.conf

[Forcepoint:email]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Forcepoint mail relay
disabled = false
pulldown_type = true

I need to copy the app folder from my deployment server to the indexer and then paste it there?

Thanks

Re: CSV input, only headers

gcusello — Mon, 06 Jul 2020 05:29:56 GMT

Hi @zidoz ,

you're using the default csv props.conf and should be correct, could you share two or three rows of your file to check the props.conf?

Then you have to copy the props.conf both on UF and Indexers and then restart Spunk on the updated systems, you can copy it manually or using a Deployment server or a Master Node (if you have an Indexers Cluster).

Ciao.

Giuseppe

Re: CSV input, only headers

zidoz — Sun, 12 Jul 2020 08:19:17 GMT

Hi @gcusello

Sorry for the delayed response, it has been a crazy week.

I've check the props.conf file like you originally suggested - and it seems to work correctly

I copied the file to the UF under: C:\Program Files\SplunkUniversalForwarder\etc\apps\forcepoint

and to the indexer under: $SPLUNK_HOME/etc/apps/search

Splunk services were restarted on both systems, but the results are the same

Re: CSV input, only headers

gcusello — Sun, 12 Jul 2020 08:25:24 GMT

Hi @zidoz ,

what do you mean when you say: "it seems to work correctly" and "but the results are the same"?

it's Ok or not?

Anyway, could you share the header and some sample of you data?

Ciao.

Giuseppe

Re: CSV input, only headers

zidoz — Sun, 12 Jul 2020 09:04:56 GMT

Hi @gcusello ,

When I upload a file manually like you initially suggested, it is being parsed correctly.

But when I read the files from the UF I get only the headers, not the entire data of the file.

Below are some samples (I changes the domain and recipient names for security reasons):

Date & Time

From: Address

Envelope Sender

Sender Name

Sender Domain

Recipient Address

Recipient Domain

Subject

Action

Direction

Black/Whitelisted

Blocked Attachment Ext

Filtering Reason

Lexical Rule

Sender IP

Attachment File Type

Attachment Filename

Emb. Domain

Emb. Full URL

Virus Name

Date

Day of Week

Message Size

Spam Score

6/26/2020 13:46

overnightmillionaire@boosts.live

14973-1569-206245-3919-a.abc=XXXX.com@mail.boosts.live

Mind-Hacks

mail.boosts.live

a.abc@XXXX.com

XXXX.com

When Will You Get Your Big Break?

Discarded

Inbound

None

Spam

None

104.140.84.17

None

boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,bit.ly

http://boosts.live/3zxpsyVNkatoWAUfonYGVLGLyg4Alnq2QXAWdwcL0XxTnd5q,http://boosts.live/6c1499faf76704b7bf.jpg,http://boosts.live/70a1ef0532cd295f74.png,http://boosts.live/R8OtTI9UnRKhxwbKJB64Rl5_LdUjdv0K4DI9zwAZ4z36tME,http://boosts.live/a63b3ec82f405cddb9.jpg,http://boosts.live/e6d3a5f8ceb81257b9.png,http://boosts.live/sJeK_Fs85ipZcsgSXC3QrUo5fZZWPtH36iYMlQd52Z6KP0uk,http://www.boosts.live/9RG3JdpFlY8u4rZ8m1Mqx0H23SKZHqt1eAEPn_XyJkGmA8IB,https://bit.ly/3fMfMKR

None

26/06/2020

Fri

12667

6/26/2020 13:46

mind-hacks@boosts.live

14973-27306-26597-3919-a.abc=XXXX.com@mail.boosts.live

Overnight Millionaire

mail.boosts.live

a.abc@XXXX.com

XXXX.com

Manifest Your Much Deserved Money Overnight

Discarded

Inbound

None

Spam

None

104.140.84.17

None

boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,bit.ly

http://boosts.live/6c1499faf8f183a4a5.jpg,http://boosts.live/70a1ef053abaa4625f.png,http://boosts.live/a63b3ec82da286e4f0.jpg,http://boosts.live/e6d3a5f8c9c246ecef.png,http://boosts.live/mgt__Nvn-eVvImYlwKA7wRBeYYkcBB1owT_FG0m7Kbi8qZbs,http://boosts.live/pgSr11AGOuTpb3bhpX69GhdIZXrlC84HYI2kpyEa9x1Ovg3U,http://boosts.live/ubFMXfNq8EFDNT0_01tRGK8FBN-oBf_J8Xh82aZb0pzo7yFa,http://www.boosts.live/lKSG3d5-Rsr99F3D_-YYVKLu-20KxrLx-9IEPwjR4aT0Wp4F,https://bit.ly/3fMfMKR

None

26/06/2020

Fri

12665

6/26/2020 13:47

bobbragdon@csoonline.com

bounce+427efa.0bdb2c-a.abc=XXXX.com@csoonline.com

Bob Bragdon - CSO Virtual Events

csoonline.com

a.abc@XXXX.com

XXXX.com

Accepted

Inbound

None

Spam

None

146.20.191.20

None

csoonline.com,eventscloud.com,eventscloud.com,idg.com,idg.com,eventscloud.com,eventscloud.com,eventscloud.com,eventscloud.com,eventscloud.com,eventscloud.com

http://events.csoonline.com/newriskandsecurity/DO,http://na.eventscloud.com/emarketing/go.php?i=776370&e=YXJpay5lbGltZWxlY2hAb3Jib2dyYXBoLmNvbQ==&l=open,http://na.eventscloud.com/file_uploads/652bead785b85a67e3d5984dd4a4d709_button_RegNow_blue_MuseoSans.png,http://www.idg.com/idg-privacy-policy/,https://events.idg.com/event-series/the-new-risk-and-security-landscape/speakers/,https://na-admin.eventscloud.com/file_uploads/07fafcf8c1503186a6ce9b105a886998_SecurityNewRiskEmailHeader.jpg,https://na-admin.eventscloud.com/file_uploads/ebc8d3ac1acea5f6fb66b5a495bb4537_New_Risk_Speaker_Strip3.jpg,https://na.eventscloud.com/emarketing/go.php?i=776370&e=YXJpay5lbGltZWxlY2hAb3Jib2dyYXBoLmNvbQ==&l=http://events.csoonline.com/newriskandsecurity/DO,https://na.eventscloud.com/emarketing/go.php?i=776370&e=YXJpay5lbGltZWxlY2hAb3Jib2dyYXBoLmNvbQ==&l=http://www.idg.com/idg-privacy-policy/,https://na.eventscloud.com/emarketing/go.php?i=776370&e=YXJpay5lbGltZWxlY2hAb3Jib2dyYXBoLmNvbQ==&l=https://events.idg.com/event-series/the-new-risk-and-security-landscape/speakers/,https://na.eventscloud.com/emarketing/profile.php?id=9c430fad5a3ddaa0ec3d855c81dd9904fdbff482ef77c1422c6bf377377ac759bb21b12317d7654a4ebbbde94eae06cf-MjAyMC0wMiM1ZWY1ZmM0MTJiY2E2

None

26/06/2020

Fri

20286

-5.09

6/26/2020 13:47

mind-hacks@boosts.live

14973-1569-180978-3919-a.abc=XXXX.com@mail.boosts.live

Overnight Millionaire

mail.boosts.live

a.abc@XXXX.com

XXXX.com

Couch Potato Goes from 0 - 7,000/mo

Discarded

Inbound

None

Spam

None

104.140.84.17

None

boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,bit.ly

http://boosts.live/-8GQVgAXKPy_v03WrGO126ofOreEAAI9MphV_QrOmy4MQlPd,http://boosts.live/6c1499faf9f8427e2d.jpg,http://boosts.live/70a1ef05315be64256.png,http://boosts.live/VU22_58DN9JSppZGa3CJOdmUNmglsoy5WuDki-6tYbc9B__O,http://boosts.live/a63b3ec8251bece5ce.jpg,http://boosts.live/ch6M-mSwL1TYtDMuSd7NugN7mmde8lnbZZBRJYzzwUy1Rmdc,http://boosts.live/e6d3a5f8c0e4fb47bf.png,http://www.boosts.live/QS2gOre3BxvAMAsUNmbQn4n3x04Ly7PerU_GohzHC9p9SIL9,https://bit.ly/3fMfMKR

None

26/06/2020

Fri

12658