We have updated our Terms of Service, Code of Conduct, and Addendum.

Data to TCP JSON source it not captured

I am trying to send data to Cribl stream→ TCP Json source using curl command, and i can see the data is coming to Cribl worker node on the given port ( verified with tcpdump) but the same data is not being captured in the TCP JSON source. Any settings i am missing here?

Tagged:

Answers

  • Jon Rust
    Jon Rust Posts: 458 mod

    Sharing your TCP JSON source config, and the curl command you're attempting would be a big help

  • Srinath Dayala
    Srinath Dayala Posts: 12

    Here is the TCP JSON source config:

    {
    "id": "testsource",
    "disabled": false,
    "sendToRoutes": true,
    "pqEnabled": false,
    "streamtags": [],
    "host": "0.0.0.0",
    "tls": {
    "disabled": true
    },
    "ipWhitelistRegex": "/.*/",
    "maxActiveCxn": 1000,
    "enableProxyHeader": false,
    "enableLoadBalancing": false,
    "authType": "manual",
    "authToken": "<<token>>",
    "type": "tcpjson",
    "port": 10070,
    "environment": "Development"
    }

    And the curl command: Both commands did not work

  • Jon Rust
    Jon Rust Posts: 458 mod

    If you're going to use curl I think you'll want to use the HTTP source. TCP JSON is not intended for HTTP conversations.

  • Jon Rust
    Jon Rust Posts: 458 mod
    edited November 18

    For TCP JSON, the payload would be something like the following:

    {"authToken": "yourtoken"}
    {"message": "hello world", "morefields": "foobar"}
    {"message": "another payload", "morefields": "foobar"}
    

  • Srinath Dayala
    Srinath Dayala Posts: 12

    I want to validate the TCP JSON source but i can only test with curl command. I don't know any other method to test it. The first curl command in my previous post should work i think but it is not.

    I have tried the below but no luck. Do you know the correct example with curl?

  • Jon Rust
    Jon Rust Posts: 458 mod
    edited November 18

    Try using netcat (nc) instead of curl. Curl is an HTTP tool. TCP JSON doesn't provide an HTTP endpoint. Create a file like below:

    $ cat payload.txt
    {"authToken": "yourtoken"}
    {"message": "hello world", "morefields": "foobar"}
    {"message": "another payload", "morefields": "foobar"}
    
    

    Then pipe that into netcat pointing to your Cribl instance and port

    EDIT: But I am curious what you're going to be sending to the TCP JSON endpoint. It is not a typical source type, and as I've pointed out, HTTP delivery will not work. For that you'd want the HTTP source type.

  • Srinath Dayala
    Srinath Dayala Posts: 12

    I have tried that also and can see that data in tcpdump but not in Cribl capture. Not sure what is going wrong.

  • Srinath Dayala
    Srinath Dayala Posts: 12

    Finally, received a sample event in Cribl. Thanks for the answers. I will test more in deep.

  • Srinath, what was your solve? We are having issues of seeing things in the live capture, we are getting a bad gateway after some time. We are using file monitor and trying to send to a different file location in cribl and seeing NO movement besides queueing.

  • Srinath Dayala
    Srinath Dayala Posts: 12

    My use case is different than yours. May be there are some worker process issues on Cribl.