rex expression does not work in curl

[Sukisen1981]

I have a simple search on a text pad, like this index=text|rex field=_raw "ApplicationRegistry-(?<text>.*)" max_match=0 |table source,sourcetype,text
This works in web UI but does not work with curlcurl -ku admin:admin https://192.168.1.4:8089/servicesNS/admin/search/jobs/export --data-urlencode search=“search index%3Dtext%7Crex%20field%3D_raw%20"ApplicationRegistry-(%3F.*)"%20max_match%3D0%20%7C%20table%20host%2Csource%2Csourcetype%2Ctext” -d output_mode=json

However, if i remove the rex expression and just table source,host,sourcetype the below command works

curl -ku admin:admin https://192.168.1.4:8089/servicesNS/admin/search/search/jobs/export -d search=“search index%3Dtext%20%7C%20table%20host%2Csource%2Csourcetype” -d output_mode=json

Why is curl failing with rex? i receive an error - The system cannot find the file specified.
I have looked at many answers here, replace -d with this or that etc. etc. but ot does not work

[DavidHourani · Accepted solution]

Hi @Sukisen1981,

This is working for me :

curl -ku admin:admin https://127.0.0.1:8089/servicesNS/admin/search/search/jobs/export -d search='search index="_audit" user=* |head 1 | rex field=_raw "user=(?<text>[^\s,]*)" | table source text'  -d output_mode=json 

and this as well :

curl -ku admin:admin https://127.0.0.1:8089/servicesNS/admin/search/search/jobs/export -d search='search index="_audit" user=* |head 1 |rex field=_raw "Audit:(?<text>.*)" | table source text'  -d output_mode=json

Could you please try and let me know if it works for you. Also which version of Splunk are you currently
using ? It could be a problem specific to your current version.

Cheers,
David

[Sukisen1981]

Receive error 'head' is not recognized as an internal or external command, in both cases,once again I am on splunk 6.63, is that having any effect?
It is getting a bit irritating now, for the moment we are managing without the rex fields but we really wanted to have the regex fields to be extracted through splunk before using the API JSON output in the downstream UI system...

[DavidHourani]

I'm testing on the same version now, working on linux.. What distro are you using ? you can get rid of the head 1 if you like it's just for reducing the number of events. Could you try it on your splunk GUI first, run the command with the exact user you are using for the curl command see what it does and if it gives results. If it's working on GUI and you're using admin for CLI it should work 🙂
Also could be a problem with the path for your curl command if you're using windows..

[Sukisen1981]

Hi @DavidHourani - I am on windows so I need to replace the single quote after the first appearance of the word 'search' with double quotes,but that does not still solve the original error message.
It is a problem with the path in windows for sure - the message is The system cannot find the file specified. But what is the solution :)?
The rex works in the UI , does not really matter if the rex returns results or not, it should not error out in the UI and that's not happening in the UI

[DavidHourani]

what you're facing is a windows related problem with Curl, not a Splunk problem, what you could do is write a small script that ads the query in a variable and send the variable instead of typing the search 🙂

[Sukisen1981]

Hi @DavidHourani - Is there any pointer to the script you are referring to? I realize that this is a windows specific issue , but alas I can not ask the client to use Linux just for this issue.
I have posted the same question on stack today, awaiting answers from from bash experts over there and also trying to install cygwin.
On this not being a splunk issue, I disagree - No where in the curl docs that splunk mention something like this can occur whilst using windows, and well, a major product vendor can not provide features that give issues on one of the most widely used OS in the world.
But that is besides the point, thanks for your reply I am now investigating other alternatives to make this work on windows. Once again, many thanks

[DavidHourani]

Hi @Sukisen1981,
You're welcome, yeah I agree on your point of view about Windows/Splunk issues. Problem is that not all commands used for Linux and then re-adapted for Windows are integrated in the same way so escaping characters doesn't always work as you'd expect it. Cygwin might solve the problem for you as well (Let me know if it does 🙂 ).

As for the scripts, have a look here :
https://linuxhint.com/curl_bash_examples/
The idea is to put the whole Splunk query in a variable and reuse the variable name in the curl command.

[Sukisen1981]

Hi @DavidHourani -
Cygwin works and ultimately of course I wanted to eventually use the REST api edn points on 8089 , wrote a small python script of 5-6 lines and it works.
Many thanks for your help , I am gonna up vote a couple of your comments since you really digged into this

[DavidHourani]

Awesome news ! Would really appreciate it if you accept the answer as well 🙂
Cheers,
David

[Sukisen1981]

@DavidHourani - done 🙂 🙂

[DavidHourani]

Awesome news ! Would really appreciate it if you accept the answer as well!
Cheers,
David

[codebuilder]

Replace your opening and closing quotation marks with single ticks to define the contents as a literal string.

 curl -ku admin:admin https://192.168.1.4:8089/servicesNS/admin/search/search/jobs/export -d search='search index=text|rex field=_raw \"ApplicationRegistry-(?<text>.*)\" max_match=0 |table source,sourcetype,text' -d output_mode=json

Then it should work for you.

[Sukisen1981]

does not work. I receive the same message in CMD

[chris_barrett]

Try the example found here and see if it works - https://docs.splunk.com/Documentation/Splunk/7.2.6/RESTREF/RESTsearch#search.2Fjobs.2Fexport

I do wonder if the asterisk not being escaped/encoded in your test is the issue?

[Sukisen1981]

Hi @chris_barrett and @harsmarvania57 . To make things more generic I have now used the audit command so that we can all replicate the issue. So, this works -
curl -ku admin:admin https://192.168.1.5:8089/servicesNS/admin/search/search/jobs/export -d search=“search index="_audit"|table source,host,text” -d output_mode=json

In the splunk web UI this works -
index="_audit"|rex field=_raw "Audit:(?.*)" |table source,host,text
But if i use curl to fetch the rex, it fails. So this does not work

 curl -ku admin:admin https://192.168.1.5:8089/servicesNS/admin/search/search/jobs/export -d search=“index="_audit"|rex field=_raw "Audit:(?<text>.*)" |table source,host,text" |table source,host,text” -d output_mode=json

I know its about escaping/writing the regex inside the curl command, but what is the way to do that?
I did refer to this answer here - https://answers.splunk.com/answers/495305/why-is-rex-field-not-producing-results-when-used-i.html
But this does not work, curious to know if you guys try the _audit index with the rex as stated above are you receiving results from the curl command?

[harsmarvania57]

Maybe noob question and not relevant but still asking, are you using curl on Linux or on Windows ?

[Sukisen1981]

I am using windows

[harsmarvania57]

Hi,

Can you please try below command

curl -ku admin:admin https://192.168.1.4:8089/servicesNS/admin/search/search/jobs/export -d search="search index=text|rex field=_raw \"ApplicationRegistry-(?<text>.*)\" max_match=0 |table source,sourcetype,text" -d output_mode=json

[Sukisen1981]

Tried and received the same error message. I can see that you are trying to escape the regular expression, I guess its all on how to escape / treat the regex as a regex

[Sukisen1981]

BTW does the splunk version play a role? I am on 6.6.3