Getting Data In

CSV file not getting indexed in correct format through UF but parses correctly through WEB UI?

Builder

alt textHas any one installed Splunk UF on Kali linux and faced any issues?.We have Splunk UF(7.1.1) installed on Kali linux and monitoring a path as below.The csv file is not coming in right format from the forwarder but when tried uploading in test environment through WEB UI(Settings-->Add Data--->Upload file ) shows the correct format

Below is the path of the csv file

/home/reports/8e20594b-282a-493e-ad9a-dc69e0ac676c.csv and I am using the monitor stanza as below

[monitor:///home/reports/*.csv]
recursive = true
index = main
sourcetype = rf
initCrcLength = 1024
crcSalt =

props.conf

SHOULDLINEMERGE=false
NO
BINARYCHECK=true
CHARSET=UTF-8
INDEXED
EXTRACTIONS=csv
KVMODE=none
category=Structured
disabled=false
pulldown
type=true
TIMESTAMPFIELDS=Timestamp
HEADER
FIELDLINENUMBER=1

0 Karma

Ultra Champion

To double check: you have that props.conf deployed on your universal forwarder as well as your indexer(s)? Normally UFs don't do much with props.conf of course, but INDEXED_EXTRACTIONS are one of the exceptions to that.

0 Karma

Contributor

Hey,

is your test environment also Kali Linux? If yes, I would start and try to monitor there too. The input assitant (WEB UI) as some rare effects (shouldlinemerge).

Looking at wrong-format.png, splunk might take the wrong delimiter/quote char. You could try to define these explicitly:

HEADER_FIELD_DELIMITER
FIELD_DELIMITER
FIELD_QUOTE 

https://docs.splunk.com/Documentation/Splunk/7.1.1/Admin/Propsconf

Could you post what the file looks like? Maybe check on Kali Linux something like cat -T yourfile.csv
https://www.if-not-true-then-false.com/2011/linux-display-show-file-contents-tabs-line-breaks-non-pr...

Cheerz,
Björn

0 Karma

Builder

The WEB UI is not a kali linux.The timestamp field shows none and I think it is not picking the time as mentioned in the props. Below is the format of the file after running the command

cat -T yourfile.csv

IP,Hostname,Port,Port Protocol,CVSS,Severity,Solution Type,NVT Name,Summary,Specific Result,NVT OID,CVEs,Task ID,Task Name,Timestamp,Result ID,Impact,Solution,Affected Software/OS,Vulnerability Insight,Vulnerability Detection Method,Product Detection Result,BIDs,CERTs,Other References 10.22.19.1,,,,0.0,Log,"","CPE Inventory","This routine uses information collected by other routines about
CPE identities (http://cpe.mitre.org/) of operating systems, services and
applications detected during the scan.","10.22.19.1|cpe:/o:cisco",1.3.6.1.4.1.25623.1.0.810002,"NOCVE",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,0100f392-2d3e-4c39-b7db-45b2b1674018,"","","","","
Details:
CPE Inventory
(OID: 1.3.6.1.4.1.25623.1.0.810002)
Version used: $Revision: 8140 $
","","","",""
10.22.19.2,,,,0.0,Log,"","CPE Inventory","This routine uses information collected by other routines about
CPE identities (http://cpe.mitre.org/) of operating systems, services and
applications detected during the scan.","10.22.19.2|cpe:/o:cisco",1.3.6.1.4.1.25623.1.0.810002,"NOCVE",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,993222f0-69af-4454-b20f-d7ae8fc041f5,"","","","","
Details:
CPE Inventory
(OID: 1.3.6.1.4.1.25623.1.0.810002)
Version used: $Revision: 8140 $
","","","",""
10.22.19.3,,,,0.0,Log,"","CPE Inventory","This routine uses information collected by other routines about
CPE identities (http://cpe.mitre.org/) of operating systems, services and
applications detected during the scan.","10.22.19.3|cpe:/o:cisco",1.3.6.1.4.1.25623.1.0.810002,"NOCVE",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,9452ff23-4c0a-4962-a81c-25d43064f956,"","","","","
Details:
CPE Inventory
(OID: 1.3.6.1.4.1.25623.1.0.810002)
Version used: $Revision: 8140 $
","","","",""
10.22.19.1,,,,0.0,Log,"","ICMP Timestamp Detection","The remote host responded to an ICMP timestamp request.
The Timestamp Reply is an ICMP message which replies to a Timestamp message. It consists
of the originating timestamp sent by the sender of the Timestamp as well as a receive
timestamp and a transmit timestamp. This information could theoretically be used to
exploit weak time-based random number generators in other services.","Vulnerability was detected according to the Vulnerability Detection Method.",1.3.6.1.4.1.25623.1.0.103190,"CVE-1999-0524",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,75d32440-245d-4c9f-83ed-eca8980aff16,"","","","","
Details:
ICMP Timestamp Detection
(OID: 1.3.6.1.4.1.25623.1.0.103190)
Version used: $Revision: 10411 $
","","","CB-K15/1514, CB-K14/0632, DFN-CERT-2014-0658","http://www.ietf.org/rfc/rfc0792.txt"
10.22.19.2,,,,0.0,Log,"","ICMP Timestamp Detection","The remote host responded to an ICMP timestamp request.
The Timestamp Reply is an ICMP message which replies to a Timestamp message. It consists
of the originating timestamp sent by the sender of the Timestamp as well as a receive
timestamp and a transmit timestamp. This information could theoretically be used to
exploit weak time-based random number generators in other services.","Vulnerability was detected according to the Vulnerability Detection Method.",1.3.6.1.4.1.25623.1.0.103190,"CVE-1999-0524",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,d6f5ad45-4d5e-4deb-8619-a2f85b329097,"","","","","
Details:
ICMP Timestamp Detection
(OID: 1.3.6.1.4.1.25623.1.0.103190)
Version used: $Revision: 10411 $
","","","CB-K15/1514, CB-K14/0632, DFN-CERT-2014-0658","http://www.ietf.org/rfc/rfc0792.txt"
10.22.19.3,,,,0.0,Log,"","ICMP Timestamp Detection","The remote host responded to an ICMP timestamp request.
The Timestamp Reply is an ICMP message which replies to a Timestamp message. It consists
of the originating timestamp sent by the sender of the Timestamp as well as a receive
timestamp and a transmit timestamp. This information could theoretically be used to
exploit weak time-based random number generators in other services.","Vulnerability was detected according to the Vulnerability Detection Method.",1.3.6.1.4.1.25623.1.0.103190,"CVE-1999-0524",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,cfbbd926-6eef-4e27-a068-6c803cf9e76b,"","","","","
Details:
ICMP Timestamp Detection
(OID: 1.3.6.1.4.1.25623.1.0.103190)
Version used: $Revision: 10411 $
","","","CB-K15/1514, CB-K14/0632, DFN-CERT-2014-0658","http://www.ietf.org/rfc/rfc0792.txt"
10.22.19.1,,,,0.0,Log,"","OS Detection Consolidation and Reporting","This script consolidates the OS information detected by several NVTs and tries to find the best matching OS.

Furthermore it reports all previously collected information leading to this best matching OS. It also reports possible additional information
which might help to improve the OS detection.

If any of this information is wrong or could be improved please consider to report these to the references community portal.","Best matching OS:...........................

0 Karma

Builder

Can you give an example of what is different when you ingest this file with a UF? Are the fields not being parsed properly?

0 Karma

Builder

I have attached screen shot in the question of images which worked and did not work

0 Karma

Builder

It looks like your events are many lines. Is that true in the source file? You may need a custom LINE_BREAKER if these are multi-line events.

0 Karma

Builder

Yes there are many lines.The props.conf is not picking for this file

0 Karma