Hi all,
I've configured a universal forwarder on Windows server to monitor a folder with csv files.
These files are logs from our mail relay system, so they are being written regularly.
I can see the files in my Splunk Search head, but only the title of the columns, not the data itself
I've configured the sourcetype as CSV, added crcSalt=<SOURCE> to the inputs configuration on the Windows Server.
Does anyone have any idea why I'm only getting the headers?
Thanks all
Hi @zidoz ,
could you share your props.conf?
Probably the problem is on the props.conf file, where do you have it?
Remember that ingesting csv files, props.conf must be on UF and Indexer (or Heavy Forwarder when present).
Ciao.
Giuseppe
Hi @zidoz ,
files in $SPLUNK_HOME/etc/system default cnnot be changed!
this means that you haven't a props.conf.
So download a copy of the csv file and following the guided web procedure [Settings -- Add data] find the correct configuration for you input.
Then copy this pros.conf file on Indexer and on Universal Forwarder, not on default folder: create your own app.
Then restart Splunk on both the systems.
For more infos see at https://docs.splunk.com/Documentation/Splunk/8.0.4/Data/Extractfieldsfromfileswithstructureddata
Ciao.
Giuseppe
Hi @gcusello ,
I think I copied the right prop.conf file to both the indexer and UF, but I'm still getting only the headers.
The data I copied is attached below:
# Version 7.3.4
[splunkd]
EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) - (?P<event_message>.+)
[scheduler]
EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) - (?P<event_message>.+)
[splunk_web_service]
EXTRACT-useragent = userAgent=(?P<browser>[^ (]+)
[user@splunk-indexer apps]$ cat search/local/props.conf
[Forcepoint:email]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Forcepoint mail relay
disabled = false
pulldown_type = true
I need to copy the app folder from my deployment server to the indexer and then paste it there?
Thanks
Hi @zidoz ,
you're using the default csv props.conf and should be correct, could you share two or three rows of your file to check the props.conf?
Then you have to copy the props.conf both on UF and Indexers and then restart Spunk on the updated systems, you can copy it manually or using a Deployment server or a Master Node (if you have an Indexers Cluster).
Ciao.
Giuseppe
Hi @gcusello
Sorry for the delayed response, it has been a crazy week.
I've check the props.conf file like you originally suggested - and it seems to work correctly
I copied the file to the UF under: C:\Program Files\SplunkUniversalForwarder\etc\apps\forcepoint
and to the indexer under: $SPLUNK_HOME/etc/apps/search
Splunk services were restarted on both systems, but the results are the same
Hi @zidoz ,
what do you mean when you say: "it seems to work correctly" and "but the results are the same"?
it's Ok or not?
Anyway, could you share the header and some sample of you data?
Ciao.
Giuseppe
Hi @gcusello ,
When I upload a file manually like you initially suggested, it is being parsed correctly.
But when I read the files from the UF I get only the headers, not the entire data of the file.
Below are some samples (I changes the domain and recipient names for security reasons):
Date & Time | From: Address | Envelope Sender | Sender Name | Sender Domain | Recipient Address | Recipient Domain | Subject | Action | Direction | Black/Whitelisted | Blocked Attachment Ext | Filtering Reason | Lexical Rule | Sender IP | Attachment File Type | Attachment Filename | Emb. Domain | Emb. Full URL | Virus Name | Date | Day of Week | Message Size | Spam Score |
6/26/2020 13:46 | overnightmillionaire@boosts.live | 14973-1569-206245-3919-a.abc=XXXX.com@mail.boosts.live | Mind-Hacks | mail.boosts.live | a.abc@XXXX.com | XXXX.com | When Will You Get Your Big Break? | Discarded | Inbound | None | None | Spam | None | 104.140.84.17 | None | None | boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,bit.ly | http://boosts.live/3zxpsyVNkatoWAUfonYGVLGLyg4Alnq2QXAWdwcL0XxTnd5q,http://boosts.live/6c1499faf7670... | None | 26/06/2020 | Fri | 12667 | 0 |
6/26/2020 13:46 | mind-hacks@boosts.live | 14973-27306-26597-3919-a.abc=XXXX.com@mail.boosts.live | Overnight Millionaire | mail.boosts.live | a.abc@XXXX.com | XXXX.com | Manifest Your Much Deserved Money Overnight | Discarded | Inbound | None | None | Spam | None | 104.140.84.17 | None | None | boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,bit.ly | http://boosts.live/6c1499faf8f183a4a5.jpg,http://boosts.live/70a1ef053abaa4625f.png,http://boosts.li... | None | 26/06/2020 | Fri | 12665 | 0 |
6/26/2020 13:47 | bobbragdon@csoonline.com | bounce+427efa.0bdb2c-a.abc=XXXX.com@csoonline.com | Bob Bragdon - CSO Virtual Events | csoonline.com | a.abc@XXXX.com | XXXX.com | Register now for CSO’s Virtual Conference, The New Risk and Security Landscape | Accepted | Inbound | None | None | Spam | None | 146.20.191.20 | None | None | csoonline.com,eventscloud.com,eventscloud.com,idg.com,idg.com,eventscloud.com,eventscloud.com,eventscloud.com,eventscloud.com,eventscloud.com,eventscloud.com | http://events.csoonline.com/newriskandsecurity/DO,http://na.eventscloud.com/emarketing/go.php?i=7763... | None | 26/06/2020 | Fri | 20286 | -5.09 |
6/26/2020 13:47 | mind-hacks@boosts.live | 14973-1569-180978-3919-a.abc=XXXX.com@mail.boosts.live | Overnight Millionaire | mail.boosts.live | a.abc@XXXX.com | XXXX.com | Couch Potato Goes from 0 - 7,000/mo | Discarded | Inbound | None | None | Spam | None | 104.140.84.17 | None | None | boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,boosts.live,bit.ly | http://boosts.live/-8GQVgAXKPy_v03WrGO126ofOreEAAI9MphV_QrOmy4MQlPd,http://boosts.live/6c1499faf9f84... | None | 26/06/2020 | Fri | 12658 | 0 |