Workshop Punk API - Fetching JSON and Parsing JSON

From Juneday education
Jump to: navigation, search

Introduction

In this workshop, we'll write code for connecting to a web server and requesting some resource (like an HTML page or even fetching JSON if the web server acts as a Web API). We'll show you the syntax, so that you understand what the example code provided with the Lab 3 code does. You will see how to connect, read and print the result to standard out.

Since we'll discover that simply printing JSON to standard out isn't very helpful for a real Java program, we'll have to convert the String with JSON we got from the server to some kind of Java objects. The process of interpreting and converting JSON to Java objects is called "parsing". We'll use the API org.json for this (and we'll download the necessary stuff in a JAR called org.json.jar).

Prerequisites

Make sure you have the following commands installed:

  • curl
  • jq
  • wget
  • java sdk (javac, java)

Getting started with the workshop

Start as usual by creating a new directory for the workshop, e.g. mkdir json-workshop and cd to that directory. This will be the root directory of your application created in this workshop.

Download the files and run the script to fetch the org.json jar file (it will end up in lib/).

Files to download:

Create the directory tree for this exercise:

$ mkdir -p se/itu/brewdog/{beer,main}

Move the files to their destination:

$ mv Beer.java se/itu/brewdog/beer/
$ mv FetchJson.java se/itu/brewdog/main/

Run tree and get an idea of the directory layout:

.
├── download_json_jar.sh
└── se
    └── itu
        └── brewdog
            ├── beer
            │   └── Beer.java
            └── main
                └── FetchJson.java

You will work with the class called se/itu/brewdog/main/FetchJson.java with the main method.

Investigate also the class se/itu/brewdog/beer/Beer.java . This is the class representing a Beer and the data for different Beer objects is what we will fetch from the Punk API.

But first, we'll make the script download_json_jar.sh executable:

$ chmod u+x download_json_jar.sh

Now we can run it later.

Investigate the Punk API documentation

Go to the Punk API documentation page and read about how to fetch some data about beer.

Questions - write down the answers as a comment in the FetchJson class

  • What is the Root endpoint for the Punk API web api?
  • If you don't provide any parameters to a query, what will you get?
  • Look at the example response and try to understand the JSON structure
  • Run the following in a terminal curl 'https://api.punkapi.com/v2/beers?brewed_before=11-2012&abv_gt=6' (note the single quotes which are missing on the documentation web page!)
    • What did you get?
    • If you want a prettier output (and have jq installed, do the following instead:
      curl 'https://api.punkapi.com/v2/beers?brewed_before=11-2012&abv_gt=6' | jq '.'
  • What parameter should you use if you only want beers with alcohol level above some value?
  • Fetch all beers (skip the brewed_before) with an alcohol above 31 abv
    • You can filter out only name and abv by piping the output to jq and you can use the following weird syntax:
      curl 'your api url' | jq '.[]|"\(.name) \(.abv)"'
  • Fetch a random beer
    • What beer did you get?
    • What food is this beer good with?

Connecting to the Punk API

Let's now write some code for connecting to the Punk API web api from our Java class. In order to do that, we'll use some classes from the Java API which you probably haven't seen until now. So, we need some imports:

import java.io.BufferedReader;     // For reading the response
import java.io.IOException;        // If connecting or reading goes wrong
import java.io.InputStreamReader;  // Also for reading
import java.net.HttpURLConnection; // For connecting to a web server
import java.net.URL;               // Specifies a URL to a web server
import java.util.stream.Collectors;// Useful for collecting lines and converting them to a List

We also need a variable to store the end point (the starting point of the web api URLs) and the query (the parameters):

  private static final String END_POINT = "https://api.punkapi.com/v2/";
  private static final String QUERY = "beers?page=1&per_page=10";

Let's write crappy code in this first attempt and do everything in the main method!

The main method should have the following steps:

  • Write out a status message to standard out saying that we'll connect to the Punk API and what URL you will use
    • END_POINT + QUERY
  • Start a try-catch
    • Create a URL object (name the reference url) and feed END_POINT + QUERY to its constuctor
    • Create a new StringBuilder called response (for the Punk API response with its JSON)
    • Create a new BufferedReader called reader, and give its constructor a new InputStreamReader and give that one's constructor url.openStream()
    • Start a for-each-loop on a String line in the list created by reader.lines().collect(Collectors.toList())
      • append each line to response
    • Print response to standard out
  • catch (IOException e) and print the message to stderr

Expand using link to the right to see the complete main method (cheater! ;-) ).

Here's the complete source for se.itu.brewdog.main.FetchJson

package se.itu.brewdog.main;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.util.stream.Collectors;

public class FetchJson {

  private static final String END_POINT = "https://api.punkapi.com/v2/";
  private static final String QUERY = "beers?page=1&per_page=10";
  
  public static void main(String[] args) {
    
    System.out.println("Connecting to the punk api");
    System.out.println("Using this url: " + END_POINT + QUERY);

    try {
      URL url = new URL(END_POINT + QUERY);
      StringBuilder response = new StringBuilder();
      BufferedReader reader =
        new BufferedReader(new InputStreamReader
                           (url.openStream()));

      for(String line : reader.lines().collect(Collectors.toList())) {
        response.append(line);
      }
      System.out.println(response);
    } catch (IOException e) {
      System.err.println("Error fetching JSON: " + e.getMessage());
    }
    
  }

}

Your class should be part of the package se.itu.brewdog.main.

To compile and run your class, do the following:

$ javac se/itu/brewdog/main/FetchJson.java && java se.itu.brewdog.main.FetchJson

Now, the program prints a lot of JSON to standard out. That's not very user-friendly, is it?

Next, we'll read the JSON from the api as before, but before we print anything, we'll create Beer objects from each object representing a beer in the JSON. This process is called to parse JSON. We'll use a library called org.json for this. Since this is not part of the standard Java API, we'll have to download the necessary classes etc, in a JAR file, which we will put on our class path.

Trouble shooting

If you run into a javax.net.ssl.SSLHandshakeException, it means your computer's Java installation doesn't trust Punk API's SSL certificate. You can add their certificate as a trusted one to your Java installation by following this tutorial. If this takes too long, revisit this after the workshop and pair up with a class mate who doesn't have this problem, for the rest of the workshop.

If the keytool tutorial didn't work for you, you can try to programatically disable key checks as shown here.

Another solution is found here on StackOverflow.

Parsing the JSON response using org.json

Introduction and prerequisites

We expect that you have seen the following video lectures (and used our wiki pages with information and exercises on the topic) prior to taking this workshop.

In this part, we will not print the actual JSON string, but first convert it (parse it) to a List<Beer>. In order to do so, we need to use the org.json library described in the video lecture and wiki pages above.

If you want to read up on how to connect to web servers from Java, then look at Oracle's tutorial on working with URLs.

Writing a method which takes a JSON String and returns a List<Beer>

First we need a few new import statements to our main class:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.util.stream.Collectors;

import java.util.ArrayList;          // a list of beers from the parser
//import java.util.Arrays;           // not used
import java.util.List;               // a list of beers from the parser

import org.json.*;                   // all the json classes for parsing

import se.itu.brewdog.beer.Beer;     // the actual beer class

Write a new method (static, since it is called from main) called static List<Beer> parse(String json) . This should be called from the main with the response.toString() (instead of printing response to stdout).

These are the steps to perform inside the method:

  • Create a new JSONArray with the method's argument json as the argument to the constructor
    • This is because the JSON from the Punk API has an outer array surrounding all beer objects
  • Create a List<Beer> beers as a new ArrayList<>()
    • This is the list we will populate with the result from parsing the big JSON array
  • Create a for loop from 0 to the JSONArray's length -1
    • for(int i = 0; i < jsonArray.length(); i++)

This is what we will do in the loop:

  • Declare JSONObject jsonObject and assign it the JSONArray's index number i
    • JSONObject jsonObject = jsonArray.getJSONObject(i);
  • Get the name of the beer from this JSONObject
    • String name = jsonObject.getString("name");
  • Do the same with the description and alcohol:
String description = jsonObject.getString("description");
double alcohol = jsonObject.getDouble("abv");
  • The suitable foods is special, it too is a JSONArray:
JSONArray foodsJson = (JSONArray)jsonObject.get("food_pairing");
  • We need an array list of Strings to store the suitable food tips:
ArrayList<String> foodStrings = new ArrayList<>();
  • Loop through the JSONArray of foods and put in the array list:
for(int j = 0; j < foodsJson.length(); j++) {
  foodStrings.add(foodsJson.getString(j));
}
  • Create a Beer from the data you have parsed out, and add it to the beers list:
beers.add(new Beer(name,
                   description,
                   alcohol,
                   foodStrings)
         );
  • return beers!

How did we know to use getDouble() for the alcohol value? We looked at the JSON for one beer:

{
    "id": 192,
    "name": "Punk IPA 2007 - 2010",
    "tagline": "Post Modern Classic. Spiky. Tropical. Hoppy.",
    "first_brewed": "04/2007",
    "description": "Our flagship beer that kick started the craft beer bla bla hipster bla",
    "image_url": "https://images.punkapi.com/v2/192.png",
    "abv": 6.0,
    ... etc etc
}

In JSON, types are implicit, so that a value surrounded by double quotes are of type String, for instance, and values without quotes representing numbers are either integers or floating point numbers. This is exactly what parsing is all about! We have an alien format (JSON) and want to get the values into the world of Java. So we kindly ask the parser to getDouble() if we want a Java double in our Java universe. And we use getString() if we want a Java String type value etc.

Now, from your main method (just before the catch(IOException), change the code that previously printed the whole JSON String to call the parse method and loop through the resulting Beer list and print one Beer at the time.

Suggested solution:

for (Beer beer : parse(response.toString())) {
  System.out.println("===================");
  System.out.println(beer);
}

Done!

You should see something like this (note that on windows/cygwin, you need to use ; as the separator for the class path):

$ javac -cp lib/org.json.jar:. se/itu/brewdog/main/FetchJson.java && java -cp lib/org.json.jar:. se.itu.brewdog.main.FetchJson
Connecting to the punk api
Using this url: https://api.punkapi.com/v2/beers?page=1&per_page=10
===================
Buzz
"A light, crisp and bitter IPA brewed with English and American hops. A small batch brewed only once."
4.50% abv
Suitable to drink with:
* Spicy chicken tikka masala
* Grilled chicken quesadilla
* Caramel toffee cake
===================
Trashy Blonde
"A titillating, neurotic, peroxide punk of a Pale Ale. Combining attitude, style, substance, and a little bit of low self esteem for good measure; what would your mother say? The seductive lure of the sassy passion fruit hop proves too much to resist. All that is even before we get onto the fact that there are no additives, preservatives, pasteurization or strings attached. All wrapped up with the customary BrewDog bite and imaginative twist."
4.10% abv
Suitable to drink with:
* Fresh crab with lemon
* Garlic butter dipping sauce
* Goats cheese salad
* Creamy lemon bar doused in powdered sugar
===================
Berliner Weisse With Yuzu - B-Sides
"Japanese citrus fruit intensifies the sour nature of this German classic."
4.20% abv
Suitable to drink with:
* Smoked chicken wings
* Miso ramen
* Yuzu cheesecake
===================
Pilsen Lager
"Our Unleash the Yeast series was an epic experiment into the differences in aroma and flavour provided by switching up your yeast. We brewed up a wort with a light caramel note and some toasty biscuit flavour, and hopped it with Amarillo and Centennial for a citrusy bitterness. Everything else is down to the yeast. Pilsner yeast ferments with no fruity esters or spicy phenols, although it can add a hint of butterscotch."
6.30% abv
Suitable to drink with:
* Spicy crab cakes
* Spicy cucumber and carrot Thai salad
* Sweet filled dumplings
===================
Avery Brown Dredge
"An Imperial Pilsner in collaboration with beer writers. Tradition. Homage. Revolution. We wanted to showcase the awesome backbone of the Czech brewing tradition, the noble Saaz hop, and also tip our hats to the modern beers that rock our world, and the people who make them."
7.20% abv
Suitable to drink with:
* Vietnamese squid salad
* Chargrilled corn on the cob with paprika butter
* Strawberry and rhubarb pie
===================
Electric India
"Re-brewed as a spring seasonal, this beer – which appeared originally as an Equity Punk shareholder creation – retains its trademark spicy, fruity edge. A perfect blend of Belgian Saison and US IPA, crushed peppercorns and heather honey are also added to produce a genuinely unique beer."
5.20% abv
Suitable to drink with:
* Mussels with a garlic and herb sauce
* Crab melt sandwich
* Shortbread cookies
===================
AB:12
"An Imperial Black Belgian Ale aged in old Invergordon Scotch whisky barrels with mountains of raspberries, tayberries and blackberries in each cask. Decadent but light and dry, this beer would make a fantastic base for ageing on pretty much any dark fruit - we used raspberries, tayberries and blackberries beause they were local."
11.20% abv
Suitable to drink with:
* Tandoori lamb with pomegranate
* Beef Wellington with a red wine jus
* Raspberry chocolate torte
===================
Fake Lager
"Fake is the new black. Fake is where it is at. Fake Art, fake brands, fake breasts, and fake lager. We want to play our part in the ugly fallout from the Lager Dream. Say hello to Fake Lager – a zesty, floral 21st century faux masterpiece with added BrewDog bitterness."
4.70% abv
Suitable to drink with:
* Fried crab cakes with avocado salsa
* Spicy shredded pork roll with hot dipping sauce
* Key lime pie
===================
AB:07
"Whisky cask-aged imperial scotch ale. Beer perfect for when the rain is coming sideways. Liquorice, plum and raisin temper the warming alcohol, producing a beer capable of holding back the Scottish chill."
12.50% abv
Suitable to drink with:
* Kedgeree
* Scotch broth with sourdough bread
* Clootie dumpling
===================
Bramling X
"Good old Bramling Cross is elegant, refined, assured, (boring) and understated. Understated that is unless you hop the living daylights out of a beer with it. This is Bramling Cross re-invented and re-imagined, and shows just what can be done with English hops if you use enough of them. Poor Bramling Cross normally gets lost in a woeful stream of conformist brown ales made by sleepy cask ale brewers. But not anymore. This beer shows that British hops do have some soul, and is a fruity riot of blackberries, pears, and plums. Reminds me of the bramble, apple and ginger jam my grandmother used to make."
7.50% abv
Suitable to drink with:
* Warm blackberry pie
* Vinegar doused fish and chips
* Aromatic korma curry with lemon and garlic naan

Expand using link to the right to see the complete FetchJson class (cheater! ;-) ).

package se.itu.brewdog.main;

import java.io.BufferedReader; import java.io.IOException; import java.io.InputStreamReader; import java.net.URL; import java.util.stream.Collectors;

import java.util.ArrayList; import java.util.Arrays; import java.util.List;

import org.json.*;

import se.itu.brewdog.beer.Beer;

public class FetchJson {

 private static final String END_POINT = "https://api.punkapi.com/v2/";
 private static final String QUERY = "beers?page=1&per_page=10";
 
 public static void main(String[] args) {
   
   System.out.println("Connecting to the punk api");
   System.out.println("Using this url: " + END_POINT + QUERY);
   try {
     URL url = new URL(END_POINT + QUERY);
     StringBuilder response = new StringBuilder();
     BufferedReader reader =
       new BufferedReader(new InputStreamReader
                          (url.openStream()));
     for(String line : reader.lines().collect(Collectors.toList())) {
       response.append(line);
     }
     for (Beer beer : parse(response.toString())) {
       System.out.println("===================");
       System.out.println(beer);        
     }
   } catch (IOException e) {
     System.err.println("Error fetching JSON: " + e.getMessage());
   }
   
 }
 static List<Beer> parse(String json) {
   JSONArray jsonArray = new JSONArray(json);
   List<Beer> beers = new ArrayList<>();
   
   for(int i = 0; i < jsonArray.length(); i++) {
     
     JSONObject jsonObject = jsonArray.getJSONObject(i);
     String name = jsonObject.getString("name");
     String description = jsonObject.getString("description");
     double alcohol = jsonObject.getDouble("abv");
     JSONArray foodsJson = (JSONArray)jsonObject.get("food_pairing");
     ArrayList<String> foodStrings = new ArrayList<>();
     for(int j = 0; j < foodsJson.length(); j++) {
       foodStrings.add(foodsJson.getString(j));
     }
     beers.add(new Beer(name,
                        description,
                        alcohol,
                        foodStrings)
               );
     /*
       beers.add(new Beer("Buzz",
       "A light, crisp and bitter IPA brewed with English" +
       " and American hops. A small batch brewed only once.",
       4.5,
       Arrays.asList(new String[] {"Spicy chicken tikka masala",
       "Grilled chicken quesadilla",
       "Caramel toffee cake"})
       )
       );
     */
   }
   return beers;
 }
 

}

Refactoring

Now, what have we said about ten kilometer long main methods and single-file applications? Nothing good, we hope!

Our program isn't very useful, as we have designed it now. It always does the same thing, and stuff is so tightly coupled that we can hardly re-use any part at all. It might be questionable whether this counts as a program at all. It is more of a proof-of-concept showing that we can connect to a web api, fetch JSON, and transform the JSON string to Java objects. The same JSON every time, though.

Typically, this is not what applications or programs look like. Rather, an application is typically configurable and flexible and let's the user use the application for similar but different tasks. Imagine if your browser could only load one web page, and you needed to install another browser if you wanted to visit yet another page on the www. Or if there was one ls command per file or directory to list. Want to list the contents of your home directory? Better run ls-home! Want to list the temp directory? Better run ls-tmp!

Outside academia, programs act more like your browser and the ls command. You give arguments to the program at start-up and the programs act accordingly. You can control the settings of your browser to manage how much cache space to use, what prefered language to send to the web servers, how long to store cookies (if you want to accept cookies at all) etc, etc.

Let's refactor our beer fetcher, so that we can run it with different queries every time, and let's move the networking to its own class (for fetching the JSON) and the parsing to its own class! This way, we are more likely to reuse the components or replace them with other components. Such flexibility is very hard if everything is done in one or two huge methods.

Create a new package net

We'll start by creating a new package se/itu/brewdog/net and create a new class there, se.itu.brewdog.net.JSONFetcher.

In this class, create a method (non-static) fetch(String query) which returns a String with the JSON fetched from the Punk API using the query argument together with the root endpoint of the api. You can use most of the code that was in the main method previously. You may choose if the method should declare that it throws IOException (leaving fetch-problems to the caller to handle) or if you want to try and handle such exceptions locally in your method (think about how to handle this and what to return to the caller if an exception occurs!).

Also, move the import statements required for the networking to the new class, and also move the two static final strings from the main class to the new class. Change the name of the final static QUERY constant to DEFAULT_QUERY in the new class. The main method (if exceptions are handled in the fetch() method!) now becomes instead:

  public static void main(String[] args) {

    String query = "";
    if (args.length != 0) {
      query = args[0];
    }
    JSONFetcher fetcher = new JSONFetcher();
    String json = fetcher.fetch(query);

    for (Beer beer : parse(json)) {
      System.out.println("===================");
      System.out.println(beer);        
    }
  }

You need to import se.itu.brewdog.net.JSONFetcher in your main class.

Run your program without arguments and with the following argument: "beers?abv_gt=31":

$ javac -cp lib/org.json.jar:. se/itu/brewdog/main/FetchJson.java && java -cp lib/org.json.jar:. se.itu.brewdog.main.FetchJson "beers?abv_gt=31"
Connecting to the punk api
Using this url: https://api.punkapi.com/v2/beers?abv_gt=31
===================
The End Of History
"The End of History: The name derives from the famous work of philosopher Francis Fukuyama, this is to beer what democracy is to history. Complexity defined. Floral, grapefruit, caramel and cloves are intensified by boozy heat."
55.00% abv
Suitable to drink with:
* Roasted wood pigeon with black pudding
* Pan seared venison fillet with juniper sauce
* Apricot coconut cake
===================
Sink The Bismarck!
"This is IPA amplified, the most evocative style of the craft beer resistance with the volume cranked off the scale. Kettle hopped, dry hopped then freeze hopped for a deep fruit, resinous and spicy aroma. A full on attack on your taste buds ensues as the incredibly smooth liquid delivers a crescendo of malt, sweet honey, hop oils and a torpedo of hop bitterness that lasts and lasts."
41.00% abv
Suitable to drink with:
* Charred apricot salad
* Whole baked reblochon
* Salted caramel crème brûlée
===================
Tactical Nuclear Penguin
"This beer is about pushing the boundaries, it is about taking innovation in beer to a whole new level. Dark and decadent, plum, treacle and roast coffee are amplified beyond any stout you've had before."
32.00% abv
Suitable to drink with:
* Lobster thermidor
* Pan fried Foie Gras
* Vanilla bean white chocolate

Crete a new package se.itu.brewdog.json

Create a new directory se/itu/brewdog/json and there, create the class se.itu.brewdog.json.BeerParser.

In this class, import everything needed for the JSON parsing, and put the complete parse method in the class but make the method public.

The full se.itu.brewdog.main.FetchJson definition now becomes:

package se.itu.brewdog.main;

import se.itu.brewdog.beer.Beer;
import se.itu.brewdog.net.JSONFetcher;
import static se.itu.brewdog.json.BeerParser.parse;

public class FetchJson {

  
  public static void main(String[] args) {

    String query = "";
    if (args.length != 0) {
      query = args[0];
    }
    JSONFetcher fetcher = new JSONFetcher();
    String json = fetcher.fetch(query);

    for (Beer beer : parse(json)) {
      System.out.println("===================");
      System.out.println(beer);        
    }
  }

}

But now the name of the main class is missleading. To the user, the program fetches beers, not JSON! The user couldn't (and shouldn't have to) care less if JSON or XML or anything else is used to fetch beers. So let's change the name of the main class to:

se.itu.brewdog.main.BrewdogBeerLookup. Change the name of the source code and class.

Compile and run (note that both the source code and the class name has changed in the command line):

$ javac -cp lib/org.json.jar:. se/itu/brewdog/main/BrewdogBeerLookup.java && java -cp lib/org.json.jar:. se.itu.brewdog.main.BrewdogBeerLookup "beers?abv_gt=31"

Voluntary challenge as an exercise

It is a bit not user-friendly to require that the argument to the program include the "beers?" part of the request.

You may move the word "beers" to the root endpoint. But the user should then include the "?" first in the argument. This is better, but not perfect. The problem is that these are all valid query URLs to the api:

So, sometimes "beers" is followed by nothing (all beers), sometimes followed by "?" (search criteria) and sometimes followed by "/".

Create a solution which works with all cases and is easy to understand by the user of the program.

Source code