We have updated our Terms of Service, Code of Conduct, and Addendum.

Redis filtering issue with unexpected results and need help on response handling. Any suggestions?

Hi All, got a question in regards to Redis. We encounter issues, that filtering isn't working as expected. We use Redis as a whitelist, meaning that we extract a field from the raw log and using the EXISTS command to check if the key (build as a combination of strings) exists in Redis. If it does not exists in Redis the return value is `'0'` and we will drop the event. What we discovered is, that events that should be dropped are available at the destination. My questions are: What is / Is there a return value from Redis in case Redis was not able to process the query (EXISTS command)? Proabably not `'0'` , but if there is a response I somehow need to catch it. That was the only thing that came to my mind could cause the issue. Do you have some other ideas? Cheers, Mario

Answers

  • Jon Rust
    Jon Rust Posts: 458 mod

    Can you share your exact filter in the Drop function? Are you comparing with `0` or `'0'`?

  • Rob Franz
    Rob Franz Posts: 33 mod

    Hi Jon, filter is: `__asset_exist=='0'`

  • Jon Rust
    Jon Rust Posts: 458 mod

    are you sure it's coming back as a string?

  • Jon Rust
    Jon Rust Posts: 458 mod

    i'd validate the type of var

  • Rob Franz
    Rob Franz Posts: 33 mod

    It is a Number, but it shouldn't matter, because: `0 == '0'` > `true` `0 === '0'` > `false`

  • Jon Rust
    Jon Rust Posts: 458 mod

    can you share the pipeline?

  • Rob Franz
    Rob Franz Posts: 33 mod

    See screenshot. What you can't see is the RegExtract function. Already changed the drop filter to a more conservative one. Before it was as described: `__asset_exist == '0' && __asset_exist_lower == '0'`

  • Rob Franz
    Rob Franz Posts: 33 mod

    Also this route is running for over a month with ~120m events per day, but there are only a few (<20) suspicious events. That is why I was asking this: """ What is / Is there a return value from Redis in case Redis was not able to process the query (EXISTS command)? Proabably not `'0'` , but if there is a response I somehow need to catch it. """