Can fromjson be used in props.conf or transforms.conf

[jwhughes58]

I'm working with these events

Oct  3 17:11:23 hostname Tetration Alert[1485]: [ERR] {"keyId":"keyId","eventTime":"1696266370000","alertTime":"1696266682583","alertText":"Missing Syslog heartbeats, it might be down","severity":"HIGH","tenantId":"0","type":"CONNECTOR","alertDetails":"{\"Appliance ID\":\"applianceId\",\"Connector ID\":\"connectorId\",\"Connector IP\":\"1.1.1.1/24\",\"Name\":\"SYSLOG\",\"Type\":\"SYSLOG\",\"Deep Link\":\"host.tetrationanalytics.com/#/connectors/details/SYSLOG?id=syslog_id\",\"Last checkin at\":\"Oct 02 2023 16.55.25 PM UTC\"}","rootScopeId":"rootScopeId"}
Oct  3 17:11:23 hostname Tetration Alert[1485]: [ERR] {"keyId":"keyId","eventTime":"1696266370000","alertTime":"1696266682583","alertText":"Missing Email heartbeats, it might be down","severity":"HIGH","tenantId":"0","type":"CONNECTOR","alertDetails":"{\"Appliance ID\":\"applianceId\",\"Connector ID\":\"connectorId\",\"Connector IP\":\"1.1.1.1/24\",\"Name\":\"EMAIL\",\"Type\":\"EMAIL\",\"Deep Link\":\"host.tetrationanalytics.com/#/connectors/details/EMAIL?id=6467c9b6379aa00e64072f57\",\"Last checkin at\":\"Oct 02 2023 16.55.25 PM UTC\"}","rootScopeId":"rootScopeId"}
Oct  3 09:57:52 hostname Tetration Alert[1393]: [DEBUG] {"keyId":"Test_Key_ID_2023-09-29 09:57:52.73850357 +0000 UTC m=+13322248.433593601","alertText":"Tetration Test Alert","alertNotes":"TestAlert","severity":"LOW","alertDetails":"This is a test of your Tetration Alerts Notifier (TAN) configuration. If you received this then you are ready to start receiving notifications via TAN."}

I set my_json to all the json.  I then use fromjson to pull out the nvps.  I then use fromjson on alertDetails since it is nested in the json.  I can do this from the CLI using

index=main sourcetype="my_sourcetype"
| fromjson csw_json
| fromjson alertDetails

I need to be able to use that in a props or transforms conf file.  Are these commands able to do that?

I tried this in the transforms.conf after extracting myAlertDetail

[stanza_name]
REGEX = "(?<_KEY_1>[^"]*)":"(?<_VAL_1>.*)"
SOURCE_KEY = myAlertDetail

I get {\ and the test message.  According to regex101.com the regex should pull everything, but it doesn't in Splunk.  Thus the question about fromjson.

Splunk 9.0.4 on Linux

TIA,

Joe

[PickleRick]

No, you can't use search commands in fields definition. You can create calculated fields but they are only limited to what you could normally put in an eval statement.

With key-value extraction done using regex (as you tried with _KEY_1 and _VAL_1 groups it's tricky to properly capture the data, you use the structure of the json object and you might hit they limit on key-value pairs extracted (100 by default if I remember correctly).

Unfortunately Splunk has no way of telling it to start kv extraction from a given point within an event - it always tries to "consume" whole event. So it works well if the _raw field as a whole is just one big json object but can't handle cases like "json sent with a syslog header". It's a shame really and I think I even posted an idea about that on ideas.splunk.com. Worth upvoting.

[jwhughes58]

@PickleRickThanks.  I was afraid of that when I couldn't find anything in the documentation.  What is your idea so I can upvote it?

[PickleRick]

https://ideas.splunk.com/ideas/EID-I-208

It turns out it was not my idea, I just upvoted and commented it from my old account because it had already been there when I wanted to create it 🙂

[jwhughes58]

@PickleRickThanks.  I've upvoted the idea.