Java:Language - SPI

From Juneday education
Jump to: navigation, search

Introduction

Java's Service Provider Interface is a mechanism for dynamically load services from third parties.

SPI works like a plug-in architecture where you can add plug-ins for services in your application, and have your application dynamically load and use services written by other people (or by yourself).

A typical example of SPI is Java's JDBC API. If your application uses a database, you communicate with the database via Java's standard java.sql API, mostly using interfaces defined there. Some of the interfaces include:

  • java.sql.Connection - the connection to the database
  • java.sql.Statement - a statement representing SQL you want to issue to the database
  • java.sql.ResultSet - the resultset you get from a query (typically a SELECT)

How come you can use interfaces defined in Java's API to communicate with the database? The answer is that you need a driver from the database vendor, in the form of a JAR file that you download from the vendor. If you have the JAR file with the driver on your class path, then the driver will register itself with java.sql.DriverManager using the Java SPI mechanism.

This allows for developers to program against a database using standard JDBC code from Java's API in java.sql, but dynamically get vendor specific drivers that implement the interfaces in java.sql.

Prior to Java 6, when there was no SPI, developers could do the same, but then they had to programmatically load the vendor's driver which would then register itself with java.sql.DriverManager. Typically it would look something like:

Connection con; // java.sql.Connection interface type reference variable

static {
  try {
    Class.forName("org.sqlite.JDBC");
    con = DriverManager.getConnection("jdbc:sqlite:some_database.db");
  } catch (SQLException e) {
    System.err.println("Connection error: " + e.getMessage());
    // throw some exception
  }
}

The code above would trigger the class org.sqlite.JDBC to be loaded into the JVM. When this happens, a static block in the class would execute, and register itself as the provider of concrete implementations of the JDBC interfaces, for instance java.sql.Connection for the protocol e.g. jdbc:sqlite.

The DriverManager class has a map which keeps track of what concrete driver to use for various vendor databases such as jdbc:sqlite, jdbc:mysql etc.

Since JDBC version 4, vendors use SPI for loading their drivers. With SPI, developers don't need to programmatically load the vendor's driver. It is enough for the vendor's JAR file to be on the class path, for DriverManager to find and load it.

Example of an SPI capable application

We have written an example using SPI for an application which loads data from a file (either a CSV file or a Json file), and parses the contents to Java objects.

The application should create Student objects from data in text files. The text files can be either a CSV (comma-separated values) or JSON files (Json is a text file format for data).

To show how the SPI mechanism works, the application knows nothing about how to parse CSV or JSON. To do the actual parsing from CSV or JSON, the application has service providers in the form of two JAR files. We'll start with showing the text file contents and the Student Java class (which is what we want the parsers to create from the text files).

The text files with data

$ cat students.csv 
Nils Nilsson,nisse@mail.se
Petra Nilsson,petra@mail.se
Bengt Bengtsson,bengan@mail.se
Beata Bond,bea@mail.se

$ file students.csv
students.csv: ASCII text
$ cat students.json
[
  {
    "name" : "Nils Nilsson",
    "mail" : "nisse@mail.se"
  },
  {
    "name" : "Petra Nilsson",
    "mail" : "petra@mail.se"
  },
  {
    "name" : "Bengt Bengtsson",
    "mail" : "bengan@mail.se"
  },
  {
    "name" : "Beata Bond",
    "mail" : "bea@mail.se"
  }
]

$ file students.json
students.json: ASCII text
package org.progund.student;

public class Student {
  private String name;
  private String mail;

  public Student(String name, String mail) {
    this.name = name;
    this.mail = mail;
  }

  public String name() {
    return name;
  }

  public String mail() {
    return mail;
  }

  public String toString() {
    return name + " <" + mail + ">";
  }
}

The application

The application's main file looks like this:

package org.progund.student;

import java.util.List;

public class StudentDemo {

  public static void main(String[] args) {

    if (args.length != 1) {
      System.err.println("you must provide a filename");
      System.exit(1);
    }

    List<Student> students = StudentParserManager.getStudents(args[0]);
    if (students == null) {
      System.out.println("No students found.");
      System.exit(2);
    }

    for (Student s : students) {
      System.out.println(s);
    }
  }
}

The application asks the StudentParserManager class to get a list of students from the provided file name. This application only works for files with the suffix .csv or the suffix .json. This is to show you a simple way for a provider to decide if it can parse the file or not. In reality, we wouldn't depend on the file name in order to decide this, but this is a simplified example.

The StudentParserManager looks like this:

package org.progund.student;

import java.util.List;
import java.util.ArrayList;
import java.util.ServiceLoader;

public class StudentParserManager {

  private static ServiceLoader<StudentParser> loader
    = ServiceLoader.load(StudentParser.class);
  
  public static List<Student> getStudents(String fileName) {

    List<Student> students = new ArrayList<>();

    for (StudentParser parser : loader) {
      students = parser.parseStudents(fileName);
      if (students != null) {
        return students;
      }
    }
    return students;
  }

}

The key feature of StudentParserManager, is that it uses java.util.ServiceLoader to find all external providers that implement the interface org.progund.student.StudentParser. It loops through all such providers it can find, and the first one that can parse the file (given as argument), will be used and the resulting List<Student> will be returned to the caller.

The org.progund.StudentParser interface is the service that providers should provide. You can think of it as the JDBC interfaces that database vendors should provide implementations for. Here's the code for org.progund.student.StudentParser:

package org.progund.student;

import java.util.List;

public interface StudentParser {
  List<Student> parseStudents(String fileName);
}

This means that the providers of parsers, e.g. a parser for CSV and a parser for JSON, need to implement this interface in order to be a provider for this service.

This is the full application. Without any providers, it cannot parse any type of file. This is true also about JDBC, without providers in the form of drivers from database vendors, you cannot do anything with JDBC.

Next, we'll look at the provider for the service in the form of a CSV parser.

A provider for CSV parsing

We can pretend that a third party vendor decides to offer us a plug-in in the form of a provider for the StudentParser service. So the vendor needs to implement the StudentParser interface in a concrete class which does the parsing of a CSV file (provided as the argument to the parseStudents(String fileName) method.

The provider needs the class files for both org.progund.student.Student and org.progund.student.StudentParser, in order to be able to compile its implementation of the service.

Here's the directory layout of this vendor for the CSV parser:

.
|-- com
|   `-- bar
|       `-- parser
`-- org
    `-- progund
        `-- student
            |-- Student.class
            `-- StudentParser.class

The vendor will put its implementation in the package com.bar.parser and call it StudentCsvParser:

package com.bar.parser;

import java.io.IOException;
import java.io.File;
import java.io.BufferedReader;
import java.io.FileReader;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.stream.Collectors;
import java.util.List;
import java.util.ArrayList;

import org.progund.student.Student;
import org.progund.student.StudentParser;

public class StudentCsvParser implements StudentParser {

  @Override
  public List<Student> parseStudents(String fileName) {

    if (!fileName.endsWith(".csv")) {
      System.err.println("File name must end with .csv: " + fileName);
      return null;
    }
    
    List<Student> students = new ArrayList<Student>();

    try {
      students = Files.lines(Paths.get(fileName))
        .map(s -> new Student(s.split(",")[0], s.split(",")[1]))
        .collect(Collectors.toList());
    } catch (IOException e) {
      System.err.println("can't parse " + fileName + ": " + e.getMessage());
    }
    
    return students;
  }
}

It's not important that you understand how the parsing works, in order to understand our SPI example. Just trust us that it parses a CSV file to a List<Student>. We have topic pages and exercises for using java.nio and functional streams on the wiki, for those interested.

Now, it is time to package the vendor's implementation to a JAR file.

Packaging the CSV parser service implementation

Before we create the JAR file with the service implementation, we need to add a special directory with a special file in order to signal that this JAR file will contain a service for org.progund.student.StudentParser. Create the directory tree META-INF/services/. In that directory, create a text file with the same name as the class with the service interface, i.e. org.progund.student.StudentParser.

The content of that file should be the full name of the implementation class:

$ cat META-INF/services/org.progund.student.StudentParser 
com.bar.parser.StudentCsvParser

So, the file is META-INF/services/org.progund.student.StudentParser, and the text inside the file is com.bar.parser.StudentCsvParser .

With the JAR file on the class path, the application will dynamically find and load this provider's implementation.

The actual JAR file is created like this:

$ jar cvf com.bar.parser.csv.jar com META-INF
added manifest
adding: com/(in = 0) (out= 0)(stored 0%)
adding: com/bar/(in = 0) (out= 0)(stored 0%)
adding: com/bar/parser/(in = 0) (out= 0)(stored 0%)
adding: com/bar/parser/StudentCsvParser.java(in = 1066) (out= 454)(deflated 57%)
adding: com/bar/parser/StudentCsvParser.class(in = 2532) (out= 1192)(deflated 52%)
ignoring entry META-INF/
adding: META-INF/services/(in = 0) (out= 0)(stored 0%)
adding: META-INF/services/org.progund.student.StudentParser(in = 31) (out= 30)(deflated 3%)

Running the application with the provider's JAR file on the class path

Now we can run the application with the provider's JAR file on the class path, and give students.csv as the argument:

$ java -cp .:providers/csv/com.bar.parser.csv.jar org.progund.student.StudentDemo students.csv
parsing using csv parser
Nils Nilsson <nisse@mail.se>
Petra Nilsson <petra@mail.se>
Bengt Bengtsson <bengan@mail.se>
Beata Bond <bea@mail.se>

If we try to run it with the JSON file as the argument, the following happens:

$ java -cp .:providers/csv/com.bar.parser.csv.jar org.progund.student.StudentDemo students.json
File name must end with .csv: students.json
No students found.

That's expected. We don't have a provider for JSON parsing, yet.

Creating a provider for JSON parsing

Another company, Foo, decides to create an implementation of the service interface, for parsing students from a JSON file. This is the directory layout for that implementation:

.
|-- com
|   `-- foo
|       `-- parser
|           |-- StudentJsonParser.class
|           `-- StudentJsonParser.java
|-- MANIFEST.MF
|-- META-INF
|   `-- services
|       `-- org.progund.student.StudentParser
|-- org
|   `-- progund
|       `-- student
|           |-- Student.class
|           `-- StudentParser.class
`-- org.json.jar

Note that the provider vendor needs org.json.jar in order to compile its parser.

To compile the implementation com/foo/parser/StudentJsonParser.java we need both . (current directory) and org.json.jar on the class path. Remember that you separate paths on windows using ; and on Mac/Linux using :. We'll show you both command lines here.

Mac/Linux:

$ javac -cp org.json.jar:. com/foo/parser/StudentJsonParser.java

Windows:

$ javac -cp "org.json.jar;." com/foo/parser/StudentJsonParser.java

Next, we need to create a file called MANIFEST.MF to point out the org.json.jar which we will include in the providers jar-file (since its implementation depends on that JAR file!):

$ cat MANIFEST.MF
Manifest-Version: 1.0
Created-By: Foo
Class-Path: ./org.json.jar

After this, we need the META-INF directory with the file pointing out the service implementation:

$ cat META-INF/services/org.progund.student.StudentParser 
com.foo.parser.StudentJsonParser

Same as for the other vendor's implementation. The file is called META-INF/services/org.progund.student.StudentParser and the text content in it is the name of the implementing class, com.foo.parser.StudentJsonParser .

This is the source code for the parser implementation:

package com.foo.parser;

import java.util.List;
import java.util.ArrayList;

import org.json.*;

import java.io.IOException;

import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;

import org.progund.student.Student;
import org.progund.student.StudentParser;

public class StudentJsonParser implements StudentParser {

  public List<Student> parseStudents(String fileName) {
    if (!fileName.endsWith(".json")) {
      System.err.println("File must end with .json: " + fileName);
      return null;
    }
    System.out.println("parsing using json parser");
    List<Student> students = null;
    try {
      JSONArray jsonArray
        = new JSONArray(new String(Files.readAllBytes(Paths.get(fileName)),
                                 StandardCharsets.UTF_8));
      students = new ArrayList<>();
      for(int i = 0; i < jsonArray.length(); i++) {
        JSONObject jsonObject = jsonArray.getJSONObject(i);
        String name = jsonObject.getString("name");
        String mail = jsonObject.getString("mail");
        students.add(new Student(name, mail));
      }
    } catch (IOException e) {
      System.err.println("can't parse json in " + fileName + ": " +
                         e.getMessage());
    }
    return students;
  }
}

Again, you don't need to understand the parser code, in order to understand how the SPI example works. But we have topic pages and exercises on this wiki for how to parse JSON from Java.

The command for creating this vendor's JAR file is slightly different, because we are including our MANIFEST.MF file and also the complete org.json.jar in the vendor's JAR file:

$ jar cvfm com.foo.parser.json.jar MANIFEST.MF META-INF/ com/ org.json.jar 
added manifest
ignoring entry META-INF/
adding: META-INF/services/(in = 0) (out= 0)(stored 0%)
adding: META-INF/services/org.progund.student.StudentParser(in = 32) (out= 31)(deflated 3%)
adding: com/(in = 0) (out= 0)(stored 0%)
adding: com/foo/(in = 0) (out= 0)(stored 0%)
adding: com/foo/parser/(in = 0) (out= 0)(stored 0%)
adding: com/foo/parser/StudentJsonParser.class(in = 1991) (out= 1082)(deflated 45%)
adding: com/foo/parser/StudentJsonParser.java(in = 1289) (out= 522)(deflated 59%)
adding: org.json.jar(in = 61749) (out= 58931)(deflated 4%)

The contents of the resulting JAR file is:

$ jar tf com.foo.parser.json.jar
META-INF/
META-INF/MANIFEST.MF
META-INF/services/
META-INF/services/org.progund.student.StudentParser
com/
com/foo/
com/foo/parser/
com/foo/parser/StudentJsonParser.class
com/foo/parser/StudentJsonParser.java
org.json.jar

Running the application with both providers' JAR files on the class path

Let's run the application with both vendors' JAR files on the class path. Both with students.csv as the argument, and with students.json as the argument and see what happens:

$ java -cp providers/json/com.foo.parser.json.jar:.:providers/csv/com.bar.parser.csv.jar org.progund.student.StudentDemo students.csv
File must end with .json: students.csv
parsing using csv parser
Nils Nilsson <nisse@mail.se>
Petra Nilsson <petra@mail.se>
Bengt Bengtsson <bengan@mail.se>
Beata Bond <bea@mail.se>
$ java -cp providers/json/com.foo.parser.json.jar:.:providers/csv/com.bar.parser.csv.jar org.progund.student.StudentDemo students.json
parsing using json parser
Nils Nilsson <nisse@mail.se>
Petra Nilsson <petra@mail.se>
Bengt Bengtsson <bengan@mail.se>
Beata Bond <bea@mail.se>

The first printout when given the argument of students.csv, contained a line saying: File must end with .json: students.csv. This is from the json parser. It just so happened that Java's ServiceLoader found the JSON parser before the CSV parser (we have no guarantee for in which order providers are found), and the JSON parser said that it can't parse files that don't end in ".json".

Conclusion

This example hopefully showed you:

  • The basic architecture and idea behind SPI
  • How to define a Service interface
  • How to load all implementing providers
  • How to write an implementation of a Service interface
    • How to package it in a JAR file
  • How to run an application which loads service providers from JAR files on the class path

Links

Further reading

Source code

Lecture video lectures

  • TODO

Lecture slides

  • TODO

Where to go next

  • To be decided - this is a stand-alone topic page for now