Getting Data In

CSV input, only headers

zidoz
Observer

Hi all,

 

I've configured a universal forwarder on Windows server to monitor a folder with csv files.

These files are logs from our mail relay system, so they are being written regularly.

I can see the files in my Splunk Search head, but only the title of the columns, not the data itselfsplunk.JPG

I've configured the sourcetype as CSV, added crcSalt=<SOURCE> to the inputs configuration on the Windows Server.

 

Does anyone have any idea why I'm only getting the headers?

 

Thanks all

Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @zidoz ,

could you share your props.conf?

Probably the problem is on the props.conf file, where do you have it?

Remember that ingesting csv files, props.conf must be on UF and Indexer (or Heavy Forwarder when present).

Ciao.

Giuseppe

0 Karma

zidoz
Observer

Hi @gcusello ,

 

I have a props.conf file on the UF under the path below:

C:\Program Files\SplunkUniversalForwarder\etc\system\default

And I have one on the indexer under:

/opt/splunk/etc/system/default

 

Which one would you like to see?

 

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @zidoz ,

files in $SPLUNK_HOME/etc/system default cnnot be changed!

this means that you haven't a props.conf.

So download a copy of the csv file and following the guided web procedure [Settings -- Add data] find the correct configuration for you input.

Then copy this pros.conf file on Indexer and on Universal Forwarder, not on default folder: create your own app.

Then restart Splunk on both the systems.

For more infos see at https://docs.splunk.com/Documentation/Splunk/8.0.4/Data/Extractfieldsfromfileswithstructureddata

Ciao.

Giuseppe

0 Karma

zidoz
Observer

Hi @gcusello ,

 

I think I copied the right prop.conf file to both the indexer and UF, but I'm still getting only the headers.

The data I copied is attached below:

# Version 7.3.4

[splunkd]
EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) - (?P<event_message>.+)

[scheduler]
EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) - (?P<event_message>.+)

[splunk_web_service]
EXTRACT-useragent = userAgent=(?P<browser>[^ (]+)
[user@splunk-indexer apps]$ cat search/local/props.conf

[Forcepoint:email]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Forcepoint mail relay
disabled = false
pulldown_type = true

 

I need to copy the app folder from my deployment server to the indexer and then paste it there?

Thanks

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @zidoz ,

you're using the default csv props.conf and should be correct, could you share two or three rows of your file to check the props.conf?

Then you have to copy the props.conf both on UF and Indexers and then restart Spunk on the updated systems, you can copy it manually or using a Deployment server or a Master Node (if you have an Indexers Cluster).

Ciao.

Giuseppe

 

0 Karma

zidoz
Observer

Hi @gcusello 

 

Sorry for the delayed response, it has been a crazy week.

I've check the props.conf file like you originally suggested - and it seems to work correctly

I copied the file to the UF under: C:\Program Files\SplunkUniversalForwarder\etc\apps\forcepoint

and to the indexer under: $SPLUNK_HOME/etc/apps/search

Splunk services were restarted on both systems, but the results are the same

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @zidoz ,

what do you mean when you say: "it seems to work correctly" and "but the results are the same"?

it's Ok or not?

Anyway, could you share the header and some sample of you data?

Ciao.

Giuseppe

0 Karma

zidoz
Observer

Hi @gcusello ,

When I upload a file manually like you initially suggested, it is being parsed correctly.

But when I read the files from the UF I get only the headers, not the entire data of the file.

 

Below are some samples (I changes the domain and recipient names for security reasons):

Date & TimeFrom: AddressEnvelope SenderSender NameSender DomainRecipient AddressRecipient DomainSubjectActionDirectionBlack/WhitelistedBlocked Attachment ExtFiltering ReasonLexical RuleSender IPAttachment File TypeAttachment FilenameEmb. DomainEmb. Full URLVirus NameDateDay of WeekMessage SizeSpam Score
6/26/2020 13:46overnightmillionaire@boosts.live14973-1569-206245-3919-a.abc=XXXX.com@mail.boosts.liveMind-Hacksmail.boosts.livea.abc@XXXX.comXXXX.comWhen Will You Get Your Big Break?DiscardedInboundNoneNoneSpamNone104.140.84.17NoneNoneboosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,bit.lyhttp://boosts.live/3zxpsyVNkatoWAUfonYGVLGLyg4Alnq2QXAWdwcL0XxTnd5q,http://boosts.live/6c1499faf7670...None26/06/2020Fri126670
6/26/2020 13:46mind-hacks@boosts.live14973-27306-26597-3919-a.abc=XXXX.com@mail.boosts.liveOvernight Millionairemail.boosts.livea.abc@XXXX.comXXXX.comManifest Your Much Deserved Money OvernightDiscardedInboundNoneNoneSpamNone104.140.84.17NoneNoneboosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,bit.lyhttp://boosts.live/6c1499faf8f183a4a5.jpg,http://boosts.live/70a1ef053abaa4625f.png,http://boosts.li...None26/06/2020Fri126650
6/26/2020 13:47bobbragdon@csoonline.combounce+427efa.0bdb2c-a.abc=XXXX.com@csoonline.comBob Bragdon - CSO Virtual Eventscsoonline.coma.abc@XXXX.comXXXX.comRegister now for CSO’s Virtual Conference, The New Risk and Security LandscapeAcceptedInboundNoneNoneSpamNone146.20.191.20NoneNonecsoonline.com,eventscloud.com,eventscloud.com,idg.com,idg.com,eventscloud.com,eventscloud.com,eventscloud.com,eventscloud.com,eventscloud.com,eventscloud.comhttp://events.csoonline.com/newriskandsecurity/DO,http://na.eventscloud.com/emarketing/go.php?i=7763...None26/06/2020Fri20286-5.09
6/26/2020 13:47mind-hacks@boosts.live14973-1569-180978-3919-a.abc=XXXX.com@mail.boosts.liveOvernight Millionairemail.boosts.livea.abc@XXXX.comXXXX.comCouch Potato Goes from 0 - 7,000/moDiscardedInboundNoneNoneSpamNone104.140.84.17NoneNoneboosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,bit.lyhttp://boosts.live/-8GQVgAXKPy_v03WrGO126ofOreEAAI9MphV_QrOmy4MQlPd,http://boosts.live/6c1499faf9f84...None26/06/2020Fri126580
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...