Getting Data In

CSV file not getting indexed in correct format through UF but parses correctly through WEB UI?

vrmandadi
Builder

alt textHas any one installed Splunk UF on Kali linux and faced any issues?.We have Splunk UF(7.1.1) installed on Kali linux and monitoring a path as below.The csv file is not coming in right format from the forwarder but when tried uploading in test environment through WEB UI(Settings-->Add Data--->Upload file ) shows the correct format

Below is the path of the csv file

/home/reports/8e20594b-282a-493e-ad9a-dc69e0ac676c.csv and I am using the monitor stanza as below

[monitor:///home/reports/*.csv]
recursive = true
index = main
sourcetype = rf
initCrcLength = 1024
crcSalt =

props.conf

SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
KV_MODE=none
category=Structured
disabled=false
pulldown_type=true
TIMESTAMP_FIELDS=Timestamp
HEADER_FIELD_LINE_NUMBER=1

0 Karma

FrankVl
Ultra Champion

To double check: you have that props.conf deployed on your universal forwarder as well as your indexer(s)? Normally UFs don't do much with props.conf of course, but INDEXED_EXTRACTIONS are one of the exceptions to that.

0 Karma

bjoernjensen
Contributor

Hey,

is your test environment also Kali Linux? If yes, I would start and try to monitor there too. The input assitant (WEB UI) as some rare effects (shouldlinemerge).

Looking at wrong-format.png, splunk might take the wrong delimiter/quote char. You could try to define these explicitly:

HEADER_FIELD_DELIMITER
FIELD_DELIMITER
FIELD_QUOTE 

https://docs.splunk.com/Documentation/Splunk/7.1.1/Admin/Propsconf

Could you post what the file looks like? Maybe check on Kali Linux something like cat -T yourfile.csv
https://www.if-not-true-then-false.com/2011/linux-display-show-file-contents-tabs-line-breaks-non-pr...

Cheerz,
Björn

0 Karma

vrmandadi
Builder

The WEB UI is not a kali linux.The timestamp field shows none and I think it is not picking the time as mentioned in the props. Below is the format of the file after running the command

cat -T yourfile.csv

IP,Hostname,Port,Port Protocol,CVSS,Severity,Solution Type,NVT Name,Summary,Specific Result,NVT OID,CVEs,Task ID,Task Name,Timestamp,Result ID,Impact,Solution,Affected Software/OS,Vulnerability Insight,Vulnerability Detection Method,Product Detection Result,BIDs,CERTs,Other References 10.22.19.1,,,,0.0,Log,"","CPE Inventory","This routine uses information collected by other routines about
CPE identities (http://cpe.mitre.org/) of operating systems, services and
applications detected during the scan.","10.22.19.1|cpe:/o:cisco",1.3.6.1.4.1.25623.1.0.810002,"NOCVE",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,0100f392-2d3e-4c39-b7db-45b2b1674018,"","","","","
Details:
CPE Inventory
(OID: 1.3.6.1.4.1.25623.1.0.810002)
Version used: $Revision: 8140 $
","","","",""
10.22.19.2,,,,0.0,Log,"","CPE Inventory","This routine uses information collected by other routines about
CPE identities (http://cpe.mitre.org/) of operating systems, services and
applications detected during the scan.","10.22.19.2|cpe:/o:cisco",1.3.6.1.4.1.25623.1.0.810002,"NOCVE",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,993222f0-69af-4454-b20f-d7ae8fc041f5,"","","","","
Details:
CPE Inventory
(OID: 1.3.6.1.4.1.25623.1.0.810002)
Version used: $Revision: 8140 $
","","","",""
10.22.19.3,,,,0.0,Log,"","CPE Inventory","This routine uses information collected by other routines about
CPE identities (http://cpe.mitre.org/) of operating systems, services and
applications detected during the scan.","10.22.19.3|cpe:/o:cisco",1.3.6.1.4.1.25623.1.0.810002,"NOCVE",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,9452ff23-4c0a-4962-a81c-25d43064f956,"","","","","
Details:
CPE Inventory
(OID: 1.3.6.1.4.1.25623.1.0.810002)
Version used: $Revision: 8140 $
","","","",""
10.22.19.1,,,,0.0,Log,"","ICMP Timestamp Detection","The remote host responded to an ICMP timestamp request.
The Timestamp Reply is an ICMP message which replies to a Timestamp message. It consists
of the originating timestamp sent by the sender of the Timestamp as well as a receive
timestamp and a transmit timestamp. This information could theoretically be used to
exploit weak time-based random number generators in other services.","Vulnerability was detected according to the Vulnerability Detection Method.",1.3.6.1.4.1.25623.1.0.103190,"CVE-1999-0524",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,75d32440-245d-4c9f-83ed-eca8980aff16,"","","","","
Details:
ICMP Timestamp Detection
(OID: 1.3.6.1.4.1.25623.1.0.103190)
Version used: $Revision: 10411 $
","","","CB-K15/1514, CB-K14/0632, DFN-CERT-2014-0658","http://www.ietf.org/rfc/rfc0792.txt"
10.22.19.2,,,,0.0,Log,"","ICMP Timestamp Detection","The remote host responded to an ICMP timestamp request.
The Timestamp Reply is an ICMP message which replies to a Timestamp message. It consists
of the originating timestamp sent by the sender of the Timestamp as well as a receive
timestamp and a transmit timestamp. This information could theoretically be used to
exploit weak time-based random number generators in other services.","Vulnerability was detected according to the Vulnerability Detection Method.",1.3.6.1.4.1.25623.1.0.103190,"CVE-1999-0524",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,d6f5ad45-4d5e-4deb-8619-a2f85b329097,"","","","","
Details:
ICMP Timestamp Detection
(OID: 1.3.6.1.4.1.25623.1.0.103190)
Version used: $Revision: 10411 $
","","","CB-K15/1514, CB-K14/0632, DFN-CERT-2014-0658","http://www.ietf.org/rfc/rfc0792.txt"
10.22.19.3,,,,0.0,Log,"","ICMP Timestamp Detection","The remote host responded to an ICMP timestamp request.
The Timestamp Reply is an ICMP message which replies to a Timestamp message. It consists
of the originating timestamp sent by the sender of the Timestamp as well as a receive
timestamp and a transmit timestamp. This information could theoretically be used to
exploit weak time-based random number generators in other services.","Vulnerability was detected according to the Vulnerability Detection Method.",1.3.6.1.4.1.25623.1.0.103190,"CVE-1999-0524",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,cfbbd926-6eef-4e27-a068-6c803cf9e76b,"","","","","
Details:
ICMP Timestamp Detection
(OID: 1.3.6.1.4.1.25623.1.0.103190)
Version used: $Revision: 10411 $
","","","CB-K15/1514, CB-K14/0632, DFN-CERT-2014-0658","http://www.ietf.org/rfc/rfc0792.txt"
10.22.19.1,,,,0.0,Log,"","OS Detection Consolidation and Reporting","This script consolidates the OS information detected by several NVTs and tries to find the best matching OS.

Furthermore it reports all previously collected information leading to this best matching OS. It also reports possible additional information
which might help to improve the OS detection.

If any of this information is wrong or could be improved please consider to report these to the references community portal.","Best matching OS:...........................

0 Karma

dflodstrom
Builder

Can you give an example of what is different when you ingest this file with a UF? Are the fields not being parsed properly?

0 Karma

vrmandadi
Builder

I have attached screen shot in the question of images which worked and did not work

0 Karma

dflodstrom
Builder

It looks like your events are many lines. Is that true in the source file? You may need a custom LINE_BREAKER if these are multi-line events.

0 Karma

vrmandadi
Builder

Yes there are many lines.The props.conf is not picking for this file

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...