Splunk Search

Indexing a log file comes up with empty results.

agregory23
New Member

Hello-

I am trying to index some files in a directory local to the splunk server (/tmp/risqiq/). I can see in the S.O.S that it it searching the log files that I want with since it has an "Action Status" of "finished reading" and a "Read_status" of "read". I put them in their own index but when I go look in the index there are 0 events in the index and its 1kb in size. I believe that its becase of my event type and the fact that its parsing the file and determining that there is nothing in there. I have tried both the continuous import mode as well as the direct file import.

I have tried to have Splunk auto detect the data type but that does not work. The log contents do show in the preview so I know it can access the files and folder. It has the date highlighted (which is not what I want). The problem is the log file is one giant line with events buried in to it. I believe each of the events begin with "id". I have also tried to search with a regex search of "\b\w[id]\b".

I have tried the following settings in the advanced mode of data preview for props.conf (I dont have shell access to the server itself):

# your settings
BREAK_ONLY_BEFORE=\b\w[id]\b
MAX_TIMESTAMP_LOOKAHEAD=150
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false

# set by detected source type
TRUNCATE=0
index=riskiq
pulldown_type=1

Here is a snippet from the log:

{"startDateInclusive":"2015-04-19T19:04:39.000-0700","endDateExclusive":"2015-04-19T20:04:39.000-0700","totalResults":2669,"resources":[{"id":249138,"url":"http://www.oneclicktools.com/oct/RingtoneConverterIcon.gif","hostname":"www.oneclicktools.com","rank":2147483647,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":49,"description":"Binary file MD5 found in AV(21%)","contentType":"image/gif","httpResponseCode":200,"detectedAt":"2010-01-01T00:00:00.000-0800","lastSeenAt":"2015-04-19T05:37:40.000-0700","entries":[{"type":"GSBMalware","matchType":"DOMAIN","url":"oneclicktools.com/"},{"type":"CymruMalware","matchType":"HOST","id":34761,"url":"http://www.oneclicktools.com/cdwriter15.exe","description":"Binary file MD5 found in AV(21%)","detectedAt":"2012-12-25T10:45:49.000-0800"}]},{"id":1130541,"url":"http://www.nutsvolts.com/","hostname":"www.nutsvolts.com","ip":"76.12.26.68","asn":"20021","rank":747367,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":49,"description":"Binary file detected by AV(21%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2010-01-01T00:00:00.000-0800","lastSeenAt":"2015-04-19T20:00:55.000-0700","entries":[{"type":"GSBMalware","matchType":"DOMAIN","url":"nutsvolts.com/"},{"type":"VirusTotal","matchType":"HOST","id":7770421,"url":"http://www.nutsvolts.com/uploads/magazine_downloads/workshop55.zip","description":"Binary file detected by AV(21%). {Avast=Win32:Malware-gen, Avira=TR/Graftor.101096.2, Comodo=UnclassifiedMalware, GData=Archive.Trojan.Agent.MV9FTA, Ikarus=Backdoor.Poison, Jiangmin=Backdoor/Poison.adbb, K7AntiVirus=Riskware ( 0040eff71 ), K7GW=Riskware ( 0040eff71 ), McAfee=RDN/Generic.dx!d2b, McAfee-GW-Edition=RDN/Generic.dx!d2b, Symantec=Trojan.Gen.2, TrendMicro-HouseCall=Suspicious_GEN.F47V0106}. md5=05dea7390de10298c277683f4d75646f","detectedAt":"2013-07-11T13:47:00.000-0700"}]},{"id":2692729,"url":"http://files.informer.com/siinst.exe","hostname":"files.informer.com","ip":"208.88.224.211","asn":"40824","rank":849,"phishing":false,"malware":true,"spam":false,"matchType":"URL","score":54,"contentType":"application/octet-stream","httpResponseCode":200,"detectedAt":"2011-02-20T06:51:54.000-0800","lastSeenAt":"2015-04-19T16:33:50.000-0700","entries":[{"type":"Malc0de","matchType":"URL","id":1860233,"url":"http://files.informer.com/siinst.exe","description":"MD5: 3ad2f4b0b0ce0875da4dd58bced17db9, IP: 208.88.224.211, Country: us, ASN: 40824","detectedAt":"2011-11-09T06:20:00.000-0800"},{"type":"VirusTotal","matchType":"URL","id":20347336,"url":"http://files.informer.com/siinst.exe","description":"Binary file detected by AV(4%). {Ikarus=Win32.SuspectCrc, TrendMicro-HouseCall=Suspicious_GEN.F47V1021}. md5=e49ef284df6d6516c0bf8851c76d081e","detectedAt":"2014-09-24T09:51:34.000-0700"}]},{"id":3795387,"url":"http://files.brothersoft.com/internet/miscellaneous/brothersoftextreme_ct2776682.exe","hostname":"files.brothersoft.com","ip":"65.49.92.216","asn":"6939","rank":7440,"phishing":false,"malware":true,"spam":false,"matchType":"URL","score":77,"description":"Binary file detected by AV(7%)","contentType":"application/octet-stream","httpResponseCode":200,"detectedAt":"2011-04-28T17:07:14.000-0700","lastSeenAt":"2015-04-19T08:31:19.000-0700","entries":[{"type":"GSBMalware","matchType":"HOST","url":"files.brothersoft.com/"},{"type":"RiskIQ","matchType":"URL","id":330552,"url":"http://files.brothersoft.com/internet/miscellaneous/BrotherSoftExtreme_CT2776682.exe","description":"Confidence: 75. FakeSoftwareUpdate: alert:Your Flash Player may be out of date","detectedAt":"2014-08-23T12:01:31.000-0700"},{"type":"CymruMalware","matchType":"HOST","id":1589,"url":"http://files.brothersoft.com/security/monitoring_software/spyagent7.zip","description":"Binary file MD5 found in AV(20%)","detectedAt":"2011-10-04T07:24:54.000-0700"},{"type":"VirusTotal","matchType":"URL","id":448557,"url":"http://files.brothersoft.com/internet/miscellaneous/BrotherSoftExtreme_CT2776682.exe","description":"Binary file detected by AV(7%). {Cyren=W32/A-68608136!Eldorado, DrWeb=Program.BrotherSoft.4, F-Prot=W32/A-68608136!Eldorado, NANO-Antivirus=Riskware.Win32.BrotherSoft.diumlo}. md5=b02a24d94306e494994ef41b55be7d07","detectedAt":"2011-09-21T09:25:34.000-0700"}]},{"id":8119183,"url":"http://www.ascentive.com/","hostname":"www.ascentive.com","ip":"64.62.158.147","asn":"6939","rank":81185,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":45,"description":"Binary file MD5 found in AV(16%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-07T22:46:15.000-0700","lastSeenAt":"2015-04-18T22:55:39.000-0700","entries":[{"type":"CymruMalware","matchType":"HOST","id":713,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Binary file MD5 found in AV(16%)","detectedAt":"2011-09-19T13:20:42.000-0700"},{"type":"VirusTotal","matchType":"HOST","id":22960038,"url":"http://www.ascentive.com/run/download?service=SpeedScan&debug=0&loadlink=http%3A%2F%2Fwww.ascentive.com%2Frun%2Fclick%2F%403040667546237%2Fproducts%2FSP_finallyfastpc%2Fselect_finallyfastpc_speedscan_download_op1.html%3Ftheme%3Dselect_speedscan_finallyfastpc_download%26plan1id%3D%26orderpackage1id%3DSSCN1YEAR29%26plan1c%3D%26upsell_code%3D%26popuppage%3D%26display%3D%26referredby%3D%403040667546237%26loadlink%3D","description":"Binary file detected by AV(11%). {ESET-NOD32=a variant of Win32/Ascentive, Kingsoft=VIRUS_UNKNOWN, McAfee=Artemis!B18049048917, McAfee-GW-Edition=Artemis, TrendMicro-HouseCall=Suspicious_GEN.F47V1029, Zillya=Trojan.Patched.Win32.80083}. md5=b18049048917e904f3c1a544d869abe1","detectedAt":"2014-11-09T22:16:08.000-0800"}]},{"id":8119273,"url":"http://www.ascentive.com/partners/affiliate-program/","hostname":"www.ascentive.com","ip":"64.62.158.147","asn":"6939","rank":81185,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":45,"description":"Binary file MD5 found in AV(16%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-07T22:46:34.000-0700","lastSeenAt":"2015-04-19T19:14:23.000-0700","entries":[{"type":"CymruMalware","matchType":"HOST","id":713,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Binary file MD5 found in AV(16%)","detectedAt":"2011-09-19T13:20:42.000-0700"},{"type":"VirusTotal","matchType":"HOST","id":13017376,"url":"http://www.ascentive.com/run/download?service=SpeedScan&debug=0&loadlink=http://www.ascentive.com/run/click/@2022810294200/products/SP/select_finallyfast_download_op1.html?theme%3Dselect_finallyfast_partner_download%26plan1id%3D%26orderpackage1id%3DFFST1YR29%26plan1c%3D%26upsell_code%3Dff2%26popuppage%3D%26display%3D%26referredby%3D@2022810294200%26c1%3D03_59531325_093feaa1-e60c-4ae0-a3c9-514147985d45%26loadlink%3D","description":"Binary file detected by AV(16%). {Comodo=Application.Win32.Conduit.~A, DrWeb=Adware.Conduit.6, ESET-NOD32=Win32/Toolbar.Conduit.R, Malwarebytes=PUP.Optional.Conduit.A, McAfee=Artemis!43583D1CCFCE, McAfee-GW-Edition=Artemis!43583D1CCFCE, Symantec=Trojan.ADH.2, TrendMicro-HouseCall=TROJ_GEN.F47V0305}. md5=43583d1ccfce25d7c422f4ecdafa662b","detectedAt":"2014-03-05T15:14:26.000-0800"}]},{"id":8145893,"url":"http://knorrnetworks.net/","hostname":"knorrnetworks.net","ip":"75.145.143.169","asn":"33491","rank":2147483647,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":30,"description":"Binary file detected by AV(22%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-08T13:45:33.000-0700","lastSeenAt":"2015-04-18T20:36:28.000-0700","entries":[{"type":"RiskIQ","matchType":"HOST","id":87799,"url":"http://knorrnetworks.net/SUPPORT/VNC/vnc-4.0-x86_win32.exe","description":"Confidence: 100. Binary file detected by AV(14%)","detectedAt":"2011-08-08T14:02:00.000-0700"},{"type":"VirusTotal","matchType":"HOST","id":47853,"url":"http://knorrnetworks.net/kn.exe","description":"Binary file detected by AV(22%). {Antiy-AVL=Trojan[RemoteAdmin:not-a-virus]/Win32.WinVNC-based, Baidu-International=HackTool.Win32.WinVNC.AsK, CMC=RemoteAdmin.Win32.WinVNC-based!O, Comodo=UnclassifiedMalware, Kaspersky=not-a-virus:RemoteAdmin.Win32.WinVNC-based.c, Malwarebytes=PUP.Radmin, McAfee=Artemis!193A45B47AD8, McAfee-GW-Edition=Heuristic.BehavesLike.Win32.Suspicious-BAY.G, NANO-Antivirus=Riskware.Win32.WinVNC-based.cqjdov, Symantec=WS.Reputation.1, TheHacker=Trojan/Buzus.bpyo}. md5=193a45b47ad85b15c39cfc32d0d88164","detectedAt":"2011-08-07T10:47:00.000-0700"}]},{"id":8278779,"url":"http://www.ascentive.com/about_us/ascentive/press_releases.php","hostname":"www.ascentive.com","ip":"64.62.158.147","asn":"6939","rank":81185,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":34,"description":"Binary file MD5 found in AV(16%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-11T21:26:19.000-0700","lastSeenAt":"2015-04-19T10:38:00.000-0700","entries":[{"type":"RiskIQ","matchType":"HOST","id":87457,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Confidence: 100. Binary file MD5 found in AV(16%)","detectedAt":"2011-08-07T14:52:10.000-0700"},{"type":"CymruMalware","matchType":"HOST","id":713,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Binary file MD5 found in AV(16%)","detectedAt":"2011-09-19T13:20:42.000-0700"},{"type":"VirusTotal","matchType":"HOST",

Does anyone have any ideas on what the best way to go about getting these log files into Splunk?

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

If you use the "upload data" wizard, and select Structured>_Json you get the first example.
The second example uses LINE_BREAKER and will break at the point designated...
Splunk understands JSON and will extract the field for you. First one indexes the fields second one doesn't.

 [indexed_json]
    INDEXED_EXTRACTIONS = json
    KV_MODE = none
    NO_BINARY_CHECK = true
    category = Structured
    description = JavaScript Object Notation format. For more information, visit http://json.org/
    disabled = false
    pulldown_type = true

    [JSON_LB]
    LINE_BREAKER = ([\r\n]+){"startDateInclusive":
    NO_BINARY_CHECK = true
    SHOULD_LINEMERGE = false
    category = Custom
    pulldown_type = true
With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

agregory23
New Member

I got a little closer with:

INDEXED_EXTRACTIONS=json

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...