Splunk Search

Preventing rex from automatically retrieving the value from the next field when the Wi-Fi country name is empty

sateesh250795
Loves-to-Learn Lots

Hi all,

rex "WifiCountryDetails\W+(?<WifiCountryDetails>[\w*\s*]+)"

We r using the above Rex for getting the Wi-Fi country details... But the problem is while fetching the data, if the Wi-Fi country name is empty it automatically gathers the next field value in it..

But if the wificountrydetails are empty it has to show empty in the data,

please let me know how to achieve it.

Labels (1)
0 Karma

inventsekar
SplunkTrust
SplunkTrust

Hi @sateesh250795 ..

1)Can you provide some sample logs..

2) After the "WifiCountryDetails" field, do you have a "," or ":", etc ?!?!?

let me assume it is a comma (",")...then, lets try this rex:

rex "WifiCountryDetails\W+(?<WifiCountryDetails>[\w*\s*]+)\,"

 

0 Karma

sateesh250795
Loves-to-Learn Lots

178.197.202.134 SECURITY_MONITORING_AUTOMATION CAMERA_SYSTEMS VML4 VML4030 1697109816450 INFO 51D29C77A022A KU5TFBKH-1700-316-114351851 - statusInfo {"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"","BatPercent":100,"BatTech":"Rechargeable","ChargerTech":"QuickCharger","ChargingState":"Off","Bat1Volt":7.931,"Temperature":24,"Battery1CaliVoltage":7.931,"Streamed":53395,"UserStreamed":1650,"MotionStreamed":7561,"IRLEDsOn":21054,"PoweredOn":11029385,"CameraOnline":11029337,"CameraOffline":48,"WifiConnectionCount":2,"WifiConnectionAttempts":2,"PIREvents":66,"FailedStreams":1,"FailedUpgrades":0,"SnapshotCount":0,"LogFrequency":2,"CriticalBatStatus":0,"ISPOn":57564,"TimeAtPlug":11025490,"TimeAtUnPlug":10979418,"PercentAtPlug":95,"PercentAtUnPlug":100,"ISPWatchdogCount":0,"ISPWatchdogCount2":0,"SecsPerPercentCurr":0,"SecsPerPercentAvg":0,"PirOorEvents":0,"DdrFailCnt":0,"carrierFw":"Generic","SignalStrengthIndicator":2,"SystemVersionString":"1.096.3.2_1.8.65.0_22","ID":534,"SystemFirmwareVersion":"1.096.3.2_30_0bad0df","HardwareVersion":"H6","SystemSerialNumber":"51D29C77A022A","TimeStamp":"2023-10-12T11:23:36+0000"}

Tags (1)
0 Karma

inventsekar
SplunkTrust
SplunkTrust

Hi @sateesh250795 .... may we know if you got answer for your question please, thanks. 

0 Karma

yuanliu
SplunkTrust
SplunkTrust

As I always warn people, do not treat structured data as text.  Using Splunk's built-in function with JSON, XML, and so on is much more robust and saves you tons of headaches in future maintenance.

All you need to do is to extract the conformant JSON into its own field.  Then apply spath. (An alternative new syntax is fromjson.)

 

| rex "^[^{]+\s*(?<json>{.+})"
| spath input=json

 

It's that simple.  Your sample log will give you everything in that JSON.

fieldnamefieldvalue
Bat1Volt7.931
BatPercent100
BatTechRechargeable
Battery1CaliVoltage7.931
CameraOffline48
CameraOnline11029337
ChargerTechQuickCharger
ChargingStateOff
CriticalBatStatus0
DdrFailCnt0
FailedStreams1
FailedUpgrades0
HardwareRevisionH6
HardwareVersionH6
ID534
IRLEDsOn21054
ISPOn57564
ISPWatchdogCount0
ISPWatchdogCount20
LogFrequency2
MotionStreamed7561
PIREvents66
PercentAtPlug95
PercentAtUnPlug100
PirOorEvents0
PoweredOn11029385
SecsPerPercentAvg0
SecsPerPercentCurr0
SignalStrengthIndicator2
SnapshotCount0
Streamed53395
SystemFirmwareVersion1.096.3.2_30_0bad0df
SystemSerialNumber51D29C77A022A
SystemVersionString1.096.3.2_1.8.65.0_22
Temperature24
TimeAtPlug11025490
TimeAtUnPlug10979418
TimeStamp2023-10-12T11:23:36+0000
Typestatus
UserStreamed1650
WifiConnectionAttempts2
WifiConnectionCount2
WifiCountryDetails 
carrierFwGeneric
json{"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"","BatPercent":100,"BatTech":"Rechargeable","ChargerTech":"QuickCharger","ChargingState":"Off","Bat1Volt":7.931,"Temperature":24,"Battery1CaliVoltage":7.931,"Streamed":53395,"UserStreamed":1650,"MotionStreamed":7561,"IRLEDsOn":21054,"PoweredOn":11029385,"CameraOnline":11029337,"CameraOffline":48,"WifiConnectionCount":2,"WifiConnectionAttempts":2,"PIREvents":66,"FailedStreams":1,"FailedUpgrades":0,"SnapshotCount":0,"LogFrequency":2,"CriticalBatStatus":0,"ISPOn":57564,"TimeAtPlug":11025490,"TimeAtUnPlug":10979418,"PercentAtPlug":95,"PercentAtUnPlug":100,"ISPWatchdogCount":0,"ISPWatchdogCount2":0,"SecsPerPercentCurr":0,"SecsPerPercentAvg":0,"PirOorEvents":0,"DdrFailCnt":0,"carrierFw":"Generic","SignalStrengthIndicator":2,"SystemVersionString":"1.096.3.2_1.8.65.0_22","ID":534,"SystemFirmwareVersion":"1.096.3.2_30_0bad0df","HardwareVersion":"H6","SystemSerialNumber":"51D29C77A022A","TimeStamp":"2023-10-12T11:23:36+0000"}

Here is an emulation you can play with and compare with real data

 

| makeresults
| eval _raw="178.197.202.134 SECURITY_MONITORING_AUTOMATION CAMERA_SYSTEMS VML4 VML4030 1697109816450 INFO 51D29C77A022A KU5TFBKH-1700-316-114351851 - statusInfo {\"Type\":\"status\",\"HardwareRevision\":\"H6\",\"WifiCountryDetails\":\"\",\"BatPercent\":100,\"BatTech\":\"Rechargeable\",\"ChargerTech\":\"QuickCharger\",\"ChargingState\":\"Off\",\"Bat1Volt\":7.931,\"Temperature\":24,\"Battery1CaliVoltage\":7.931,\"Streamed\":53395,\"UserStreamed\":1650,\"MotionStreamed\":7561,\"IRLEDsOn\":21054,\"PoweredOn\":11029385,\"CameraOnline\":11029337,\"CameraOffline\":48,\"WifiConnectionCount\":2,\"WifiConnectionAttempts\":2,\"PIREvents\":66,\"FailedStreams\":1,\"FailedUpgrades\":0,\"SnapshotCount\":0,\"LogFrequency\":2,\"CriticalBatStatus\":0,\"ISPOn\":57564,\"TimeAtPlug\":11025490,\"TimeAtUnPlug\":10979418,\"PercentAtPlug\":95,\"PercentAtUnPlug\":100,\"ISPWatchdogCount\":0,\"ISPWatchdogCount2\":0,\"SecsPerPercentCurr\":0,\"SecsPerPercentAvg\":0,\"PirOorEvents\":0,\"DdrFailCnt\":0,\"carrierFw\":\"Generic\",\"SignalStrengthIndicator\":2,\"SystemVersionString\":\"1.096.3.2_1.8.65.0_22\",\"ID\":534,\"SystemFirmwareVersion\":\"1.096.3.2_30_0bad0df\",\"HardwareVersion\":\"H6\",\"SystemSerialNumber\":\"51D29C77A022A\",\"TimeStamp\":\"2023-10-12T11:23:36+0000\"}"
``` data emulation above ```

 

 

Tags (1)

PickleRick
SplunkTrust
SplunkTrust

While I wholehaeartedly agree about not using text-based tools for structured data, the problem is that you can't use spath for defining extractions - you can only use it in search. And json or xml kv-extractions work only if the whole event is a well-formed structure. You can't use it if you have - for example - a json event with a syslog header.

There is an open idea about it somewhere on ideas.splunk.com

yuanliu
SplunkTrust
SplunkTrust

extractions work only if the whole event is a well-formed structure. You can't use it if you have - for example - a json event with a syslog header.

There is an open idea about it somewhere on ideas.splunk.com


I remember seeing you posting a link in Slack but can't find it in a hurry.  Maybe more people could vote for such.  Be honest, though, seeing all kinds of "smart" schemes developers come up to mix JSON with plain text in this board, it can be tricky to implement a generally application solution.

One workaround for situations like this is to make a data-specific regex field extraction to extract the compliant part for each sourcetype so this regex doesn't have to be included in search commands.  This makes easier code maintenance.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Found it.

https://ideas.splunk.com/ideas/EID-I-208

And I don't think it even needs to cover so many use cases. So far we can't do the one, most popular - a structured data blob forwarded wthin a syslog message.

I wiuldn't expect such feature to be too automatic. It could just go with "if the sourcetype has KV_MODE set the STRUCTURED_PART (by default (.*) ) defines which part of the raw event is subjected to KV extraction". And that's it. You could do your own regex to capture everything after the syslog header or whatever you need. Simple yet flexible. Like LINE_BREAKER or TOKENIZER

yuanliu
SplunkTrust
SplunkTrust

That STRUCTURED_PART could be entered as a comment.  I think it is more practical than a general solution.

0 Karma

inventsekar
SplunkTrust
SplunkTrust

rex-empty-string.jpg

the rex command, in my previous reply.. works perfect. pls check it. 

 

As you are  a new member, karma points / upvotes are appreciated, pls help those who help you, with your karma points. if any reply solved your issue, pls accept it as "Accept as solution", thanks. 

0 Karma

inventsekar
SplunkTrust
SplunkTrust

Lets try this (i have tested this on my laptop, its working fine)

source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty" 
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>\w+)\"" | table _raw WifiCountryDetails

 

 

0 Karma

sateesh250795
Loves-to-Learn Lots

If we r doing it for stats its not working.... giving the same results as before .... can u pls try for stats count by wificountrydetails instead of table  

Tags (1)
0 Karma

inventsekar
SplunkTrust
SplunkTrust
source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty" 
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>\w+)\"" 
| stats count by WifiCountryDetails

working fine, as suggested previously, check screenshot below..  (pls note the rex command closely.. i have edited ur rex little bit.. i added the ("\"\:\") before rex and (\") after the rex). 

rex-empty-string-stats.jpg

0 Karma

sateesh250795
Loves-to-Learn Lots

I did exactly same but the again its fetching the next field.. pls look into the screenshoterror regex.png

Tags (1)
0 Karma

inventsekar
SplunkTrust
SplunkTrust

please take the screenshot along with the SPL command also seen inside the image (zoom down or keep the browser window small, so big portion can be copied to the image)

0 Karma

sateesh250795
Loves-to-Learn Lots

pls have a lookregex error1.png

Tags (1)
0 Karma

inventsekar
SplunkTrust
SplunkTrust
source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty" 
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>(\w+|\S))\"" 
| stats count by WifiCountryDetails

Pls check the above SPL.. i checked it and its working fine. 

to troubleshoot further.. can you pls copy paste two sample logs, one with country and one without country. 

(pls remove ip address, important details from the sample logs.. just copy paste the line like these..

{"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"","BatPercent":100,"BatTech":"Rechargeable"

{"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"india","BatPercent":100,"BatTech":"Rechargeable"

)

0 Karma

inventsekar
SplunkTrust
SplunkTrust

Hi Splunk New beginners:
For absolute beginners...Splunk newbie learning videos..
https://www.youtube.com/@SiemNewbies101/playlists

 

I have added 25 small videos of rex... Completely for Splunk newbies and beginners.
hope this helps somebody, thanks.

0 Karma

inventsekar
SplunkTrust
SplunkTrust

Pls check this SPL:

source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty" 
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>(\w+|\S))\"" 
| eval WifiCountryDetails = if(isnull(WifiCountryDetails) OR len(WifiCountryDetails)==0, "No Country Name", WifiCountryDetails)
| stats count by WifiCountryDetails

image-no-county-name.jpg

 

0 Karma

sateesh250795
Loves-to-Learn Lots

No. its not working😑. again its taking the next field value... and we r gathering the count by using stats count by wificountrydetails... 

Tags (1)
0 Karma

sateesh250795
Loves-to-Learn Lots

there is : in the place of wificountry details ..... its has to fetch as it is.. but it is fetching next field value batpercent in the place of wificountry details.

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...