Solving practical problems with jq

Posted on Views:

As a developer, you will sometimes face weird problems, it is important to come up with reliable and repeatable ways to solve this problems, so when such problems come up again, you would be able to find a solution easier. As for myself, one of the tools that I found most useful on Unix-like system is jq, which is a tool to process json files. Let me demonstrate how I use this tool to solve some problems I encountered during work.

Problem: Convert a JSON file to CSV

Example JSON

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[
  {
    "title": "This is a song",
    "artist": "This is an artist",
    "album": "This is an album",
    "year": 1989
  },
  {
    "title": "This is a song",
    "artist": "This is an artist",
    "album": "This is an album",
    "year": 1989
  },
  {
    "title": "This is a song",
    "artist": "This is an artist",
    "album": "This is an album",
    "year": 1989
  }
]

JQ code to generate csv:

1
jq -r '(.[0] | keys_unsorted) as $keys | ([$keys] + map([.[ $keys[] ]])) [] | @csv'

Resulting CSV:

1
2
3
4
"title","artist","album","year"
"This is a song","This is an artist","This is an album",1989
"This is a song","This is an artist","This is an album",1989
"This is a song","This is an artist","This is an album",1989

Problem: Aggregate JSON object.

Example JSON:

1
2
3
4
5
{
  "A": [{ "Name": "A1" }, { "Name": "A2" }],
  "B": [{ "Name": "B1" }, { "Name": "B2" }],
  "C": [{ "Name": "C" }]
}

The goal is to produce something like below:

1
{ "A": ["A1", "A2"], "B": ["B1", "B2"], "C": ["C"] }

It transform the object and aggregate (or compress?) them by “Name” property, I know this can be easily done with JavaScript, but jq and bash seems like more widely used and will come in handy when JavaScript is not available.

The jq code I came up with is as follows:

1
jq '[keys_unsorted[] as $k|{($k): [.[$k][]|.Name]}]|add'

References:

  • https://stackoverflow.com/a/32967407