JSON data

How to work with JSON data

There are a number of command line tools available to help you work with JSON data.

  1. jq (GitHub)
    • jq is a lightweight and flexible command-line JSON processor.
    • jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.
    • jq is written in portable C, and it has zero runtime dependencies. You can download a single binary, scp it to a far away machine of the same type, and expect it to work.
    • jq can mangle the data format that you have into the one that you want with very little effort, and the program to do so is often shorter and simpler than you’d expect.
    • You can download jq from here (tutorial, manual).
    For example you can use:
    @systemoutput("curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.[0]'")
    
  2. GROQ (GitHub)
    • Graph-Relational Object Queries (GROQ) is a query language (like SQL, but different) which is designed to work directly on JSON documents.
    • Its primary design goals are expressive filtering, joining of several documents into a single response, and shaping the response to fit the client application.
    • The idea behind GROQ is to be able to describe exactly what information your application needs, potentially joining together information from several sets of documents, then stitching together a very specific response with only the exact fields you need.
    • It basically lets you write queries you can quickly filter and then reformat JSON documents to get them into the most convenient shape.
    • GROQ was developed by Sanity.io (where it’s used as the primary query language). It’s open source and it gives us built-in ways to use it in JavaScript and the command line on any JSON source.
    • You can install using npm with npm install -g groq-cli
    For example you can use:
    @system("curl -o todos.json https://jsonplaceholder.typicode.com/todos")
    @systemoutput("cat todos.json | groq '*[completed == true]{title, userId}' --pretty")
    @system("rm todos.json")
    
    or alternatively,
    @systemoutput("curl -sS https://jsonplaceholder.typicode.com/todos | groq '*[completed == true]{title, userId}' --pretty")
    

Other tools you might find useful include:

  • Jshon - Jshon parses, reads and creates JSON. It is designed to be as usable as possible from within the shell and replaces fragile adhoc parsers made from grep/sed/awk as well as heavyweight one-line parsers made from perl/python. Requires Jansson;
  • RapidJSON - A fast JSON parser/generator for C++ with both SAX/DOM style API;
  • JSON for modern C++;
  • jsonv.sh - A Bash command line tool for converting JSON to CSV;
  • JSON.sh - a json parser written in shell, compatible with ash, bash, dash and zsh;
  • JSON.awk - A practical JSON parser written in awk;
  • Python JSON Module - Python JSON encoder and decoder
  • ijson - Iterative JSON parser with a standard Python iterator interface;
  • jtc - cli tool to extract, manipulate and transform source JSON;
  • underscore-cli - Command-line utility-belt for hacking JSON and Javascript;
  • Eat - Command-line tool for converting anything to JSON.
  • gron - transforms JSON into discrete assignments to make it easier to grep for what you want and see the absolute 'path' to it. It eases the exploration of APIs that return large blobs of JSON but have terrible documentation;
  • jo - a small utility to create JSON objects or arrays from the shell;
  • json - a fast CLI tool for working with JSON. It is a single-file node.js script with no external deps (other than node.js itself);
  • Record Query - A tool for doing record analysis and transformation;
  • Record Stream - commandline tools for slicing and dicing JSON records;
  • JSON lambda (jl) - Functional sed for JSON (a tiny functional language for querying and manipulating JSON);
  • jp - command is a command line interface to JMESPath, an expression language for manipulating JSON;
  • json-command - JSON command line processing toolkit;
  • JSON-grep (GitHub) - command line tool and API for parsing JSON documents based on logical expressions;
  • jsed - command line JSON editor.