Never confuse education with intelligence, you can have a PhD and still be an idiot.
- Richard Feynman -

Assignment:Exposing data over http

From Juneday education
Revision as of 16:10, 24 May 2018 by Rikard (Talk | contribs) (FAQ for the assignments: Added bonus about web frontend to the web API)

Jump to: navigation, search


In this three-part assignment you will create an open web API for publishing data from the Systembolaget product line. The web API will allow for searches via HTTP/REST. It is extremely important that you read this page and make sure you understand the goal of the three-part assignment. Also read up on the technology listed below.

Background and goals of the assignment

The Swedish alcohol monopoly Systembolaget publishes an "API" with all their products in their product line. The "API" consists of a large (huge, really) XML file. The XML file is updated every day. While this is helpful for developers, the problem is that in order to create an app accessing the product catalog, the developer needs to download the complete XML file. The XML file from Systembolaget can be found here.

The idea behind this three-part assignment, is to create a web service which allows us to query for products according to some criteria. We haven't found such a web API with Systembolaget, so let's create one ourselves! When we have such a web API, we can build apps and services around it, and even allow others to access the data in this new way. If you plan to take an Android course later, you may very well write an app fetching data from your own web API.

We have provided a database with the products from the XML file for you. Optimally, this database should be updated every day, but we'll disregard that in this assignment. So your task boils down to writing the web backend for the data and writing some GUI which will connect to the backend and fetch data from it using various queries.

Note about the XML file

The XML file was the reason for us to come up with the idea behind this assignment in three parts. So, from your perspective, it is ancient history. The data from the XML file are already in a database. So you will not see the XML file, neither do you have to mention the XML file in your system reports and descriptions. We mention it here only as the motivation for creating a web API in the first place. Systembolaget should actually have offered a web API instead of the XML file. But as they didn't, we borrowed their XML file and converted it to a database, so that you can use JDBC and SQLite3 as the back-end for your web API.

Preparing yourself for the assignments

The three sub assignments will cover a whole lot of acronyms and techie buzzwords. Therefore, we strongly encourage you to read up on and orient yourselves on the technologies involved. Below is a list of resources explaining some of the terms involved:

Each of the sub assignments will explain the application of various technologies further, as will the lectures and workshops accompanying these assignments.

Videos introducting the assignment and sub assignments

Start with these videos:

We have also gathered all videos for the three assignments here: Exposing_data_labs_-_Student_material_and_resources.

Strategy for the web API

A REST web API works as a web server, listening for request and responding with data formatted in some way. It is common to use some standard format for the data, such as JSON or XML. In this assignment, we'll settle for JSON, because it sounds modern and nice.

So what about the actual data, then? Well, this assignment is given as part of the university course "Programmeringsteknik och databaser" (Programming and databases), so why not program against a database? We have fetched the XML file with the products, and inserted (most of) the data into a database for you. It is possible to use this assignment in other courses too, but it was produced with this particular course in mind, so a lot of references will be tied to that course in the documentation and descriptions found here.

So, given a database with all the products, you will write the web API using Java Servlet technology. But as you are learning about SQL and databases in the course, we will start with the backend without a real connection to the database. It is not uncommon to start working on a part of a system before all parts are ready. We'll instruct you to create a fake database access (which simply pretends to fetch data from somewhere and returns a list of product objects as if they really were fetched from the database). Then you can focus on making sure the web API works (and don't care where the actual data comes from).

There are various ways to test the web API which we will show you. All this will be done in the first sub assignment, Assignment:Exposing_data_over_http_lab1_Web_API (in the source code referred to as simply "Lab1").

In the second sub assignment, Assignment:Exposing_data_over_http_lab2_GUI_Client (in the source code referred to as simply "Lab2"), you will create a small GUI client using Java Swing (in later editions of this assignment we will move on to JavaFX but this year we'll settle with Swing - apologies for that). The idea of this GUI client is to create an app with a graphical user interface which lets the user search for (or, rather, give filter criteria) products and display the products satisfying the filter criteria in a table-like view. Of course, this GUI client will later get the data from the web API, but in the second sub assignment, we'll again create fake data. This time, we'll fake data so that we don't have to focus on networking and parsing JSON (JSON is the format of the data we'll get from the web API). We want to show you that you can write a GUI which gets products from somewhere and displays them in some kind of table component. When the GUI is functioning, layout-wise, we will also show you that you can create the correct query strings to later send to the web API. This means that you can test your GUI and verify that it creates the correct query strings based on the user's input in the criteria input text fields.

Finally, we'll tie everything together in the third sub assignment, Assignment:Exposing_data_over_http_lab3_integrate_the_subsystems. In this third, and last, sub assignment, you'll switch out the fake data component in the web API and replace it with data from the database, using JDBC. At this point, if you have followed the course's database lectures, and exercises, you know enough SQL to create a query for fetching all product data from the database. If you have followed our instructions for lab1, then the switch will be really easy and no code will have to be changed in the actual Servlet. You will get to focus on writing the database component (but don't worry, we'll provide a lot of code to get you started).

On the client-side, we'll switch from using fake data to creating a component which actually uses the query string produced by the GUI and connects to the web API and requests products according to the query criteria (e.g. min_price=100&max_price=200&min_alcohol=50). Since you have finished and tested the GUI in lab2, we can here focus on how to connect to a web API from a Java application, read the data and convert the JSON string to Java objects (which will be presented in the GUI).

If we have the time, we'll also demonstrate an Android app, which will do the same thing; present the user with options for specifying some criteria, fetch JSON from the web API and present it in some tabular widget in the app.

Why did we design the assignment this way?

We wanted to give you a three-part assignment which involves four parts:

  • Java programming which is a little more complicated than the introduction course
  • Database technology - A database which you can fetch data from using SQL
  • Java programming for a GUI application
  • Database connectivity - Connecting a Java application to a database

Other than that, we wanted the assignment to cover these features:

  • Data in the database should be real data, and not some made up boring stuff
  • Data should be exposed from the database to applications in some standard way
  • APIs at different levels should be used
    • Application frameworks (like JDBC, Swing, An API which you write in part yourselves, e.g. for handling Product objects)
    • A web API which lets clients query for data matching some criteria and responds with JSON over HTTP
  • A description of a larger system overview
    • architecture
    • some design patterns
    • system integration (e.g. using JSON as the data format between systems, JDBC between an application and a database)
  • Something you can keep after the course and use in future courses or projects
  • Some buzzword compliance so that you can say that you have worked with REST, HTTP, Servlets, JSON etc
  • A system with a few subsystems distributed and networked with a larger scale than the typical school exercise in programming
  • Something that resembles a real-world application with client-server or frontend/backend
  • Data transfer using some standard data format like JSON

What to do now

OK, so how do we get started? We suggest that you start with lab1, Assignment:Exposing_data_over_http_lab1_Web_API and get the web API functional. Then you can move on to lab2, Assignment:Exposing_data_over_http_lab2_GUI_Client, and after that, finally wrap it up in lab3 Assignment:Exposing_data_over_http_lab3_integrate_the_subsystems.

Good luck and happy reading!

// Rikard and Henrik


Some students are bound to think that the three assignment parts, lab1, lab2 and lab3 are too easy, since we give away lots of code and there's only small parts for you to write yourself. This is something we're aware of. One of the pedagogical ideas behind that, is that you will need to read and understand a lot of some one else's code (namely our code), which is very common for a programmer. In fact, it isn't very common to start from scratch and write a whole system from bottom up.

Also, since we wrote much of the system for you, we were able to end up with a much larger system than we would have if you were to write everything from scratch. It takes time to both learn how to design such a system, how to code it and then also code it. So another reason for our giving away a lot of code was to allow for a slightly larger system as a whole than would be the case if we didn't give you anything. A great part of the assignments are the reports with a description of the system as a whole which we require you to hand in apart from the code itself. This way, we think, we will force you to read and understand the system with client-server style access to the web API, the connection to the database back-end, and the higher level protocols (HTTP) and formats (JSON) which we think is at least as important a skill as is the actual Java coding!

However, some students will think the three assignments are hard enough or even very hard, even considering the small amount of coding required on their behalf. In order to handle the spectrum of difficulties, we have focused a lot at supervision for those who need help in getting started and in understanding the system on a larger scale. We have also provided a lot of video lectures to help understanding the new concepts introduced in the assignments.

For those who think it is really too simple to pass the coding parts, we have provided optional challenges in each of the three assignments. We'll summarize the challenges here since they are detailed on each assignment page. And please beware that these challenges are optional and not in anyway required. They simply serve as something meaningful to do for those who weren't challenged enough by the original assignments.

Lab 1 challenges

  • Use a CSV file with product data as source for the fake products in FakeProductLine
  • Include the type property of the products in the JSON (add it in JsonExporter)
  • Expand the list of parameters the Web API accepts to include also
    • name (could be used for filtering on a substring of the product name)
    • product_group (filtering on a named product group)

Lab 2 challenges

  • Add a checkbox "I am over 20" which unlocks the text fields for alcohol
  • Use a clickable link for testing the API URL the GUI produces
  • If you added name and product_group as valid parameters in Lab1, add fields for them in the GUI too

Lab 3 challenges

  • Write your own JDBC code (replace DBHelper with your own stuff) for the server-side
  • Write your own Client for fetching JSON and your own parser for JSON to List<Product> on the client-side
  • Super-challenge: As a much harder challenge - Write an application which fetches the XML from Systembolaget and parses it and replaces or updates data in the database

Alternative user interface - web front-end (Bonus on demand)

On popular demand, we've added an alternative client (user interface) for the web API you are writing in these three assignments.

If you too feel that Swing seems a wee bit outdated, and want to use a browser as the client when interfacing with the back-end (the web API with the servlet and database), then you can check this out:

This is not a formal part of the assignments, and serves more as a bonus chapter, showing you how you can use HTML and JavaScript to write a small front-end to your own web API. The web API will be finished after you finish "Lab 3".

FAQ for the assignments

See our page: Exposing data over http FAQ, where we have collected frequently asked questions etc regarding the three sub-assignments for the Exposing data over HTTP lab.

Terms and concepts


« PreviousBook TOCNext »