topic Re: How to add data from an external API files? in Deployment Architecture

How to add data from an external API files?

maha110192 — Sun, 20 Feb 2022 02:42:45 GMT

Hello splunkies!

I'm trying to be and admin and I'm trying to add data manually to my inputs.conf, please see my scenario:

path: /logfiles/syslog/log.txt

The output from a script that contacts an internal REST API. There are two kinds of requests in this file:

1 . http://localhost:8080/api/requests/xTraining.json API shows data from the non-production host and should be written to

Index = API-NPTraining

2. http://localhost:8080/api/requests/Training.json api shows data from the production host and should be written to

index = API-PTraining

Both should use sourcetype ss:training

Data in this file will rotate daily to log.txt.1020, log.txt.1021...etc

I have my stanzas like this

#first stanza

[monitor:///logfiles/syslog/log*.txt]

disabled = 0

host = http://localhost:8080/api/requests/xTraining.json

index = API-NPTraining

sourcetype = ss:training

# second Stanza

[monitor:///logfiles/syslog/log*.txt]

disabled = 0

host = http://localhost:8080/api/requests/Training.json

index = API-PTraining

sourcetype = ss:training

What am I missing? Am I wrong in something?

thank you.

Re: How to add data from an external API files?

gcusello — Sun, 20 Feb 2022 09:17:45 GMT

Hi @maha110192,

there an error:

path file name is /logfiles/syslog/log.txt, so in the monitor stanza you have to put

[monitor:///logfiles/syslog/log.txt]

in this way you don't take the rotated files but only the first file.

if you want to take also the rotated files (that aren't indexed twice also changine filename), you have to use a different monitor stanza

[monitor:///logfiles/syslog/log.txt.*]

using your monitor stanza you take only the first file, not the others.

In addition I don't like a so long string, with special chars for hostname: it's very difficoult to use in searches.

At least, you have to use different indexes between production and not production logs only if they have diferent retention and/or different access rights, otherwise you could put them in the same index, recognizing them by host.

Ciao.

Giuseppe

Re: How to add data from an external API files?

maha110192 — Sun, 20 Feb 2022 19:26:00 GMT

Hi @gcusello,

Thank you for helping me. I can see my mistake. Also, I have different rights for prod, non-prod. So, I need to keep both.

Regarding the long host string, Can I just put in this way?

host = /api/request/xTraining.json

or still being so long?

Best regards,

Re: How to add data from an external API files?

gcusello — Mon, 21 Feb 2022 06:31:08 GMT

Hi @maha110192,

ok for your need.

About the host value, I hint to avoid special chars as "/", usually host is the hostname of the server (using the usual convention for your servers) or a name as "Production_Server" or something similar.

If this answer soves your request, please accept it for the other people of Community or tell me how I can help you.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: How to add data from an external API files?

maha110192 — Mon, 21 Feb 2022 08:34:30 GMT

Hello Giuseppe,

I agree with you about the host name. But I’m worried that I’m missing this part of the exercise:

“The output from a script that contacts an internal REST API. There are two kinds of requests in this file:

1 . http://localhost:8080/api/requests/xTraining.json API shows data from the non-production host“

I don’t know if I am referring well to this APIs or if I need to add something else. Or if have to create an output.conf

can you help me to clarify the situation here?

thanks in advance,

and for sure those karma points are yours. 😁

thanks,

Re: How to add data from an external API files?

gcusello — Mon, 21 Feb 2022 12:57:07 GMT

Hi @maha110192,

outputs.conf is to send data to an indexer not for API configuration.

About the hostname, even if I don't like it, follow the test instructions.

I am not an expert in APIs, but I think that you have to refer well to this APIs.

Ciao.

Giuseppe

Re: How to add data from an external API files?

maha110192 — Mon, 21 Feb 2022 15:23:01 GMT

Thank you very much @gcusello . I really appreciate it your help.

Best regards,