Query on simplifying JSON to CSV conversion.

hgoolya

If I am reading the docs correctly, when converting from json to CSV, I have to define all the fields. Since all of the fields are really already known, is there a way to output all the fields first without having to define all of them? We are working on converting some Azure logs, and there are LOTs of properties.field(s). Hoping there is another way without us having to define all of these in the "Fields to serialize" field within the Serialize function.

Jon Rust

All fields must be listed, in the order you require. Copy-pasta works for the list. Possibly in your analysis tool of choice you can get a list?

hgoolya

ok, tahnks

hgoolya

thanks

Jon Rust

create a temp function to serialize into K=V and another to use regex to pull all the field names. In the preview window, copy-pasta the list

Jon Rust

hold on, trying it out

Jon Rust

```{ "conf": { "output": "default", "streamtags": [], "groups": {}, "asyncFuncTimeout": 1000, "functions": [ { "filter": "true", "conf": { "mode": "extract", "type": "json", "srcField": "raw", "dstField": "_raw" }, "id": "serde" }, { "filter": "true", "conf": { "type": "kvp", "fields": [ "!", "!cribl_breaker", "" ], "dstField": "raw", "cleanFields": false, "srcField": "_raw" }, "id": "serialize" }, { "filter": "true", "conf": { "source": "_raw", "iterations": 100, "overwrite": false, "regex": "/(?<fields>[-a-zA-Z0-9]+)=/g" }, "id": "regex_extract" }, { "filter": "true", "conf": { "add": [ { "disabled": false, "name": "fields", "value": "fields.join(',')" } ] }, "id": "eval" } ] }, "id": "list_fields" }```

Jon Rust

» my sample data started as stringified JSON » i parsed it into actual JSON » then serialized into K=V » then regex extract for all field names (strings leading up to `=`), with global flag set » then eval to join the fields array into a list

Jon Rust

obviously nested json would present a problem, but then i don't know how you'd plan on making that CSV in the first place

hgoolya

great. Let me give that a try

hgoolya

the nested is going to be the issue

Jon Rust

how would you do nested in CSV?

hgoolya

not sure. I think we are doing this for another source, so I need to look. We will tackle this next week