Splunk Search

Can fromjson be used in props.conf or transforms.conf

jwhughes58
Contributor

I'm working with these events

 

Oct  3 17:11:23 hostname Tetration Alert[1485]: [ERR] {"keyId":"keyId","eventTime":"1696266370000","alertTime":"1696266682583","alertText":"Missing Syslog heartbeats, it might be down","severity":"HIGH","tenantId":"0","type":"CONNECTOR","alertDetails":"{\"Appliance ID\":\"applianceId\",\"Connector ID\":\"connectorId\",\"Connector IP\":\"1.1.1.1/24\",\"Name\":\"SYSLOG\",\"Type\":\"SYSLOG\",\"Deep Link\":\"host.tetrationanalytics.com/#/connectors/details/SYSLOG?id=syslog_id\",\"Last checkin at\":\"Oct 02 2023 16.55.25 PM UTC\"}","rootScopeId":"rootScopeId"}
Oct  3 17:11:23 hostname Tetration Alert[1485]: [ERR] {"keyId":"keyId","eventTime":"1696266370000","alertTime":"1696266682583","alertText":"Missing Email heartbeats, it might be down","severity":"HIGH","tenantId":"0","type":"CONNECTOR","alertDetails":"{\"Appliance ID\":\"applianceId\",\"Connector ID\":\"connectorId\",\"Connector IP\":\"1.1.1.1/24\",\"Name\":\"EMAIL\",\"Type\":\"EMAIL\",\"Deep Link\":\"host.tetrationanalytics.com/#/connectors/details/EMAIL?id=6467c9b6379aa00e64072f57\",\"Last checkin at\":\"Oct 02 2023 16.55.25 PM UTC\"}","rootScopeId":"rootScopeId"}
Oct  3 09:57:52 hostname Tetration Alert[1393]: [DEBUG] {"keyId":"Test_Key_ID_2023-09-29 09:57:52.73850357 +0000 UTC m=+13322248.433593601","alertText":"Tetration Test Alert","alertNotes":"TestAlert","severity":"LOW","alertDetails":"This is a test of your Tetration Alerts Notifier (TAN) configuration. If you received this then you are ready to start receiving notifications via TAN."}

 

I set my_json to all the json.  I then use fromjson to pull out the nvps.  I then use fromjson on alertDetails since it is nested in the json.  I can do this from the CLI using

 

index=main sourcetype="my_sourcetype"
| fromjson csw_json
| fromjson alertDetails

 

I need to be able to use that in a props or transforms conf file.  Are these commands able to do that?

I tried this in the transforms.conf after extracting myAlertDetail

 

[stanza_name]
REGEX = "(?<_KEY_1>[^"]*)":"(?<_VAL_1>.*)"
SOURCE_KEY = myAlertDetail

 

I get {\ and the test message.  According to regex101.com the regex should pull everything, but it doesn't in Splunk.  Thus the question about fromjson.

Splunk 9.0.4 on Linux

TIA,

Joe

Labels (2)
Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

No, you can't use search commands in fields definition. You can create calculated fields but they are only limited to what you could normally put in an eval statement.

With key-value extraction done using regex (as you tried with _KEY_1 and _VAL_1 groups it's tricky to properly capture the data, you use the structure of the json object and you might hit they limit on key-value pairs extracted (100 by default if I remember correctly).

Unfortunately Splunk has no way of telling it to start kv extraction from a given point within an event - it always tries to "consume" whole event. So it works well if the _raw field as a whole is just one big json object but can't handle cases like "json sent with a syslog header". It's a shame really and I think I even posted an idea about that on ideas.splunk.com. Worth upvoting.

jwhughes58
Contributor

@PickleRickThanks.  I was afraid of that when I couldn't find anything in the documentation.  What is your idea so I can upvote it?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

https://ideas.splunk.com/ideas/EID-I-208

It turns out it was not my idea, I just upvoted and commented it from my old account because it had already been there when I wanted to create it 🙂

 

0 Karma

jwhughes58
Contributor

@PickleRickThanks.  I've upvoted the idea.

0 Karma
Get Updates on the Splunk Community!

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out &gt;&gt; As our brave ...

Customer Experience | Call for Stories: Your 2023 Journey with Splunk!

Share your Splunk journey: Splunk is committed to supporting our customers toward success. As the year draws ...

Infographic provides the TL;DR for the 2023 Splunk Career Impact Report

We’ve been shouting it from the rooftops! The findings from the 2023 Splunk Career Impact Report showing that ...