Data to TCP JSON source it not captured

Srinath Dayala

I am trying to send data to Cribl stream→ TCP Json source using curl command, and i can see the data is coming to Cribl worker node on the given port ( verified with tcpdump) but the same data is not being captured in the TCP JSON source. Any settings i am missing here?

Jon Rust

Sharing your TCP JSON source config, and the curl command you're attempting would be a big help

Srinath Dayala

Here is the TCP JSON source config:

{
"id": "testsource",
"disabled": false,
"sendToRoutes": true,
"pqEnabled": false,
"streamtags": [],
"host": "0.0.0.0",
"tls": {
"disabled": true
},
"ipWhitelistRegex": "/.*/",
"maxActiveCxn": 1000,
"enableProxyHeader": false,
"enableLoadBalancing": false,
"authType": "manual",
"authToken": "<<token>>",
"type": "tcpjson",
"port": 10070,
"environment": "Development"
}

And the curl command: Both commands did not work

Jon Rust

If you're going to use curl I think you'll want to use the HTTP source. TCP JSON is not intended for HTTP conversations.

Jon Rust

For TCP JSON, the payload would be something like the following:

{"authToken": "yourtoken"}
{"message": "hello world", "morefields": "foobar"}
{"message": "another payload", "morefields": "foobar"}

Srinath Dayala

I want to validate the TCP JSON source but i can only test with curl command. I don't know any other method to test it. The first curl command in my previous post should work i think but it is not.

I have tried the below but no luck. Do you know the correct example with curl?

Jon Rust

Try using netcat (nc) instead of curl. Curl is an HTTP tool. TCP JSON doesn't provide an HTTP endpoint. Create a file like below:

$ cat payload.txt
{"authToken": "yourtoken"}
{"message": "hello world", "morefields": "foobar"}
{"message": "another payload", "morefields": "foobar"}

Then pipe that into netcat pointing to your Cribl instance and port

EDIT: But I am curious what you're going to be sending to the TCP JSON endpoint. It is not a typical source type, and as I've pointed out, HTTP delivery will not work. For that you'd want the HTTP source type.

Srinath Dayala

I have tried that also and can see that data in tcpdump but not in Cribl capture. Not sure what is going wrong.

Srinath Dayala

Finally, received a sample event in Cribl. Thanks for the answers. I will test more in deep.

ebfrater@yahoo.com

Srinath, what was your solve? We are having issues of seeing things in the live capture, we are getting a bad gateway after some time. We are using file monitor and trying to send to a different file location in cribl and seeing NO movement besides queueing.

Srinath Dayala

My use case is different than yours. May be there are some worker process issues on Cribl.