Difference between revisions of "ITIC:Networks and protocols"

From Juneday education
Jump to: navigation, search
(Further reading: some more links)
(HTTP(S): Example headers)
 
Line 127: Line 127:
  
 
HTTP stands for HyperText Transfer Protocol. Hyper text, is text with links to other documents (like the blue links on a web page, or links in some help systems to other parts of the help system).
 
HTTP stands for HyperText Transfer Protocol. Hyper text, is text with links to other documents (like the blue links on a web page, or links in some help systems to other parts of the help system).
 +
 +
Here are some example HTTP responses (showing only the ''headers'', which carries the metadata about the response):
 +
 +
Error code 503 (Backend fetch failed):
 +
<pre>
 +
$ HEAD 'https://www.gu.se/english/about_the_university/staff/?languageId=100001&userId=xljunj&departmentId=107812'
 +
503 Backend fetch failed
 +
Connection: close
 +
Date: Thu, 12 Sep 2019 08:03:49 GMT
 +
Via: 1.1 varnish (Varnish/5.2)
 +
Age: 0
 +
Retry-After: 5
 +
Server: nginx/1.12.2
 +
Content-Type: text/html; charset=utf-8
 +
Client-Date: Thu, 12 Sep 2019 08:03:49 GMT
 +
Client-Peer: 130.241.151.114:443
 +
Client-Response-Num: 1
 +
Client-SSL-Cert-Issuer: /C=NL/ST=Noord-Holland/L=Amsterdam/O=TERENA/CN=TERENA SSL CA 3
 +
Client-SSL-Cert-Subject: /C=SE/L=G\xC3\xB6teborg/O=G\xC3\xB6teborgs universitet/CN=www.gu.se
 +
Client-SSL-Cipher: ECDHE-RSA-AES128-SHA256
 +
Client-SSL-Socket-Class: IO::Socket::SSL
 +
Grace: none
 +
X-Varnish: 520028186
 +
</pre>
 +
 +
Error code 500 (Internal server error):
 +
<pre>
 +
$ HEAD 'https://ait.gu.se/sitemap.xml'
 +
500 Can't connect to ait.gu.se:443
 +
Content-Type: text/plain
 +
Client-Date: Thu, 12 Sep 2019 08:08:28 GMT
 +
Client-Warning: Internal response
 +
</pre>
 +
 +
Information code 404 (not found):
 +
<pre>
 +
$ HEAD 'http://wiki.juneday.se/research.html'
 +
404 Not Found
 +
Connection: close
 +
Date: Thu, 12 Sep 2019 08:09:46 GMT
 +
Server: Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips mod_auth_kerb/5.4 PHP/5.5.21
 +
Content-Type: text/html; charset=iso-8859-1
 +
Client-Date: Thu, 12 Sep 2019 08:09:46 GMT
 +
Client-Peer: 130.241.135.117:80
 +
Client-Response-Num: 1
 +
</pre>
 +
 +
Code 200 OK:
 +
<pre>
 +
$ HEAD 'http://juneday.se/sitemap.xml'
 +
200 OK
 +
Connection: close
 +
Date: Thu, 12 Sep 2019 08:10:12 GMT
 +
Accept-Ranges: bytes
 +
ETag: "d4545-346-585030b44af00"
 +
Server: Apache/2.2.22 (Debian)
 +
Content-Length: 838
 +
Content-Type: application/xml
 +
Last-Modified: Tue, 26 Mar 2019 17:56:44 GMT
 +
Client-Date: Thu, 12 Sep 2019 08:10:42 GMT
 +
Client-Peer: 89.18.105.40:80
 +
Client-Response-Num: 1
 +
</pre>
 +
 +
Code 301 (Moved permanently) - what makes you browser to redirect to a new location if a page has moved
 +
<pre>
 +
$ curl -I -L http://www.gu.se
 +
HTTP/1.1 301 Moved Permanently
 +
Server: nginx/1.12.2
 +
Date: Thu, 12 Sep 2019 09:10:08 GMT
 +
Content-Type: text/html
 +
Content-Length: 185
 +
Connection: keep-alive
 +
Location: https://www.gu.se/
 +
</pre>
  
 
===SSH===
 
===SSH===

Latest revision as of 11:11, 12 September 2019

Prerequisites and preparation

External literature

  • Kernigan - D Is for Digital: What a Well-Informed Person Should Know about Computers and Communications
    • Chapter: Part III - Communications, Internet, pages 117-160
  • Fröberg/Sandklef, Swedish compendium
    • Chapter: Ladda ned filer från webben, pages 40-44"

Introduction

Computers are connected in networks and can communicate with each other. Networks can be local in various degrees; same room or building, same organization, same country or internationally. In order to communicate, the computers need to use agreed upon and standardized ways for communication, which is called protocols. They also need to be assigned unique addresses, and know about each other's addresses to communicate (much like you need to know your friend's phone number in order to communicate over the phone, or the physical address if you are pen pals and want to communicate in writing).

When communicating with computers, you typically want to request some data or send some data (or both). Therefore, you will also learn about some formats for data sent across networks, such as XML, Json and CSV.

Wired networks

Computers can be connected via cables. It is possible to link two (nearby) computers via a direct cable from one network interface to another. But that's pretty limited. Instead, you can use a dedicated (often small) computer called a router, which connects all computers and routes the traffic between them. The router is often also used for assigning all computers in the network a unique address.

Wireless networks

Today, it is more common (only geezers like Henrik and Rikard use cables) to communicate wirelessly using a wireless router (sometimes called access point, hotspot or wireless gateway). Most broadband modems have an integrated wireless router, so you only need one device to get online wirelessly at home. Wireless computer communication is often called Wi-Fi (which is actually a trademark) and a wireless networks is usually referred to as a WLAN (Wireless Local Area Network).

A wireless router uses radio to relay communication between two or more clients. The router is often responsible for assigning unique addresses to the clients.

You can (and should) use encrypted connections between you computer and the router, so that people on the same network can't read your communication. There are still unencrypted networks around but we advice against connecting to them (unless in a controlled environment, like a school laboration). The reason we advice against it is of course security. Communicating without encryption means that anyone on the same network can see everything you do (including logging in using username and password on a plain HTTP web site).

If you must use an unencrypted WLAN, make sure that you understand the risks and that you use only HTTPS for sensitive stuff, or communicate over protocols with built-in encryption. More about encryption further on.

Names and numbers

In order for computers to communicate, they need a means of finding each other, an address on the network. For this purpose, every computer on the network is assigned an IP address. Typically, when you connect to a network (via cable or wirelessly), the first thing your computer does is to broadcast its hardware address of its network device. This is a unique number for every network card made (but it can be changed or spoofed). A server (an application on a network serving many clients over the network using some protocol) then pics up the broadcast, and assigns an available IP address to the computer. The computer then accepts the offered IP address and configures its network card to this IP number.

IP number (or IP address) stands for Internet Protocol address/number. The Internet Protocol (IP) is how computers communicate on networks (and the Internet, which is a large international network). The protocol sends packages over the network. A package contains a header and data. The header contains only metadata like what protocol to use for the actual transmission, where the package comes from and what the destination IP address is. Other protocols handle the actual transmission (typically TCP).

Private networks

When you connect to a router (e.g. a WLAN router), you will typically be assigned a private network IP address, which is local to your local area network. This IP will only be used for communication with the router and other computers on the same network. This was a good thing when IP version 4 was the standard, because it meant that local IP addresses needed not be universally unique (which would lead to a shortage as the number of computers grew). Instead, all computers on a local network will share one connection to the Internet using a gateway computer with two network interfaces, one to the local network (with a private IP address) and one to the Internet with a "public" (routable on the Internet) IP address. All requests from the local network will (from the Internet's point of view) come from the same IP address (that of the gateway).

IPv4 addresses consists of four 8 bit numbers separated by three dots, e.g. 10.0.116.35. Eight bit (unsigned) numbers range from 0 - 255.

Local/private IPv4 addresses typically come in the following ranges:

  • 10.0.0.0 – 10.255.255.255
  • 172.16.0.0 – 172.31.255.255
  • 192.168.0.0 – 192.168.255.255

IPv6 addresses are much "larger" and consists of eight 16 bit numbers (typically written in hexadecimal to save space!). This means 128 bits, which in turn means that we can have 2128 addresses! That's about 3.403×1038 addresses. If you don't remember exponentiations, that's more than 3 followed by 38 zeros. Compare that to the IPv4 addressing with 32 bits, which is 4 294 967 296 addresses (232). Rikard's laptop currently has the following IPv6 address for its network card: fe80::de1f:acad:148:d9b4 .

Domain name system

Now, we are humans and as such, we have a hard time remembering numbers. It's been said that a normal person can hold seven digits in its working memory. So, in order to remember addresses to computers, domain names were invented. A fully qualified domain name (FQDN) uniquely identifies a computer on the internet. An example of which is wiki.juneday.se. Now, the IP number for that computer is needed for your browser to be able to request a page from our web server. The IP number is currently 130.241.135.117. So, how can your browser get that information when you type in the FQDN?

The answer is the domain name system, DNS. Every connected computer, needs the IP number to at least one DNS (this is often provided by the router at the same time the computer gets its IP number). So, when you type in a FQDN in your browser, the browser asks the operating system's network service for the IP number for the FQDN. The operating system uses the DNS protocol to connect to the configured DNS, and queries the DNS server program for the IP address for the FQDN. The DNS responds to the request with the IP address (if there is one) for the FQDN, and passes this information on to the browser. Note: this is a simplified version of the story, in order to make a point about domain names and IP numbers.

You can try to use your computer's configured DNS by the use of some command line applications:

$ nslookup wiki.juneday.se
Server:		127.0.1.1
Address:	127.0.1.1#53

Non-authoritative answer:
Name:	wiki.juneday.se
Address: 130.241.135.117

$ host wiki.juneday.se
wiki.juneday.se has address 130.241.135.117

$ dig wiki.juneday.se

; <<>> DiG 9.10.3-P4-Ubuntu <<>> wiki.juneday.se
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58735
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;wiki.juneday.se.		IN	A

;; ANSWER SECTION:
wiki.juneday.se.	1152	IN	A	130.241.135.117

;; Query time: 2 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Thu Jun 27 12:39:42 CEST 2019
;; MSG SIZE  rcvd: 60

$ systemd-resolve wiki.juneday.se
wiki.juneday.se: 130.241.135.117

-- Information acquired via protocol DNS in 118.4ms.
-- Data is authenticated: no
$

Ports

A computer on the Internet can provide more than one service at the same time. A service is provided by running a special application called a server application (sometimes service). A computer can run many applications at the same time, so it can provide many services. The same computer can thus both act as a web server and a file server (for e.g. FTP). But if the computer has only one IP number, how can we distinguish between the services on that IP?

The answer is something called ports. A port is described as a number representing an end-point for network communication. You might think of it as an extension at a corporation's phone system. Or an apartment number in a house with many flats. How can you send a post card to someone at a street address which hosts e.g. 24 families? You could specify the apartment number on the address (the mailman should probably manage anyway as long as everyone has distinct names, but what if there are two John Smiths living at the same address?).

So, when you contact a server for a specific service, like HTTP, you need not only the IP number but also the port. Luckily for all of us, this is rather standardized. HTTP typically runs on port 80, and your browser knows this. So if you don't add the port number to your request, it will try with port 80 first (or 443 if you are using HTTPS instead of HTTP).

The port numbers can be any number between 0 and 65535. On most operating systems, ports up to 1024 requires administrator privileges to be opened. You open a port on your computer when you run a server application.

Here are some common standard ports for various services:

  • 20: File Transfer Protocol (FTP) (Data Transfer)
  • 21: File Transfer Protocol (FTP) (Command Control)
  • 22: Secure Shell (SSH)
  • 23: Telnet remote login service, unencrypted text messages
  • 25: Simple Mail Transfer Protocol (SMTP)
  • 53: Domain Name System (DNS)
  • 80: Hypertext Transfer Protocol (HTTP) (web)
  • 110: Post Office Protocol (POP3)
  • 123: Network Time Protocol (NTP)
  • 143: Internet Message Access Protocol (IMAP)
  • 194: Internet Relay Chat (IRC)
  • 443: HTTP Secure (HTTPS) "HTTP over TLS/SSL"

We'll look at some of those protocols below.

Protocols

Protocols are standards for how computers communicate when using certain services over a network.

DHCP

DHCP is used for dynamically assigning (and requesting) an IP address on a network. It is dynamic, because there is no guarantee that you will get the same IP-number every time you connect to the network.

DNS

The DNS protocol is used when communicating with a DNS server, typically looking up the IP address of a (fully qualified) domain name. DNS is a binary protocol, so you need to use some tool in order to communicate with a DNS, unless you are fluent in writing binary DNS queries. Here's a list of some of those tools:

  • nslookup
  • host
  • dig

HTTP(S)

HTTP is used for communicating with a web server. Typically a browser acts as the client and requests some resource (typically a web page or an image, or a web page and all of its images), and uses HTTP to contact a web server with a request for a specific resource. The web server then uses HTTP to send its response (some metadata and the actual data).

HTTPS is used for the same thing, but uses an encrypted communication line between the client and server.

HTTP stands for HyperText Transfer Protocol. Hyper text, is text with links to other documents (like the blue links on a web page, or links in some help systems to other parts of the help system).

Here are some example HTTP responses (showing only the headers, which carries the metadata about the response):

Error code 503 (Backend fetch failed):

$ HEAD 'https://www.gu.se/english/about_the_university/staff/?languageId=100001&userId=xljunj&departmentId=107812'
503 Backend fetch failed
Connection: close
Date: Thu, 12 Sep 2019 08:03:49 GMT
Via: 1.1 varnish (Varnish/5.2)
Age: 0
Retry-After: 5
Server: nginx/1.12.2
Content-Type: text/html; charset=utf-8
Client-Date: Thu, 12 Sep 2019 08:03:49 GMT
Client-Peer: 130.241.151.114:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=NL/ST=Noord-Holland/L=Amsterdam/O=TERENA/CN=TERENA SSL CA 3
Client-SSL-Cert-Subject: /C=SE/L=G\xC3\xB6teborg/O=G\xC3\xB6teborgs universitet/CN=www.gu.se
Client-SSL-Cipher: ECDHE-RSA-AES128-SHA256
Client-SSL-Socket-Class: IO::Socket::SSL
Grace: none
X-Varnish: 520028186

Error code 500 (Internal server error):

$ HEAD 'https://ait.gu.se/sitemap.xml'
500 Can't connect to ait.gu.se:443
Content-Type: text/plain
Client-Date: Thu, 12 Sep 2019 08:08:28 GMT
Client-Warning: Internal response

Information code 404 (not found):

$ HEAD 'http://wiki.juneday.se/research.html'
404 Not Found
Connection: close
Date: Thu, 12 Sep 2019 08:09:46 GMT
Server: Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips mod_auth_kerb/5.4 PHP/5.5.21
Content-Type: text/html; charset=iso-8859-1
Client-Date: Thu, 12 Sep 2019 08:09:46 GMT
Client-Peer: 130.241.135.117:80
Client-Response-Num: 1

Code 200 OK:

$ HEAD 'http://juneday.se/sitemap.xml'
200 OK
Connection: close
Date: Thu, 12 Sep 2019 08:10:12 GMT
Accept-Ranges: bytes
ETag: "d4545-346-585030b44af00"
Server: Apache/2.2.22 (Debian)
Content-Length: 838
Content-Type: application/xml
Last-Modified: Tue, 26 Mar 2019 17:56:44 GMT
Client-Date: Thu, 12 Sep 2019 08:10:42 GMT
Client-Peer: 89.18.105.40:80
Client-Response-Num: 1

Code 301 (Moved permanently) - what makes you browser to redirect to a new location if a page has moved

$ curl -I -L http://www.gu.se
HTTP/1.1 301 Moved Permanently
Server: nginx/1.12.2
Date: Thu, 12 Sep 2019 09:10:08 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://www.gu.se/

SSH

SSH is used for encrypted computer communication. Typically, you need a remote shell on a computer and use ssh to log into a remote computer. You will then get a shell on the remote computer and can work as if you were connected directly to the remote computer. SSH stands for Secure SHell. You can also use SSH to issue commands on a remote computer, and using SCP you can securely transfer files between two computers using SSH as the communication channel.

Some services use SSH for this kind of communication. For instance Github (a service for control versioning) allows you to configure SSH for pulling and pushing source code between you local repository and a central repository at github.com .

FTP

FTP - File transfer protocol

FTP stands for File Transfer Protocol, and was popular as the protocol for sharing files on file servers. FTP should be replaced by SFTP or SCP (which uses SSH). However, FTP is still in use in many organizations.

SMTP

SMTP stands for Simple Mail Transfer Protocol and is used for sending emails via email servers. If you have an app for email on your phone or computer, it probably uses SMTP for sending emails via your configured email server.

POP etc

POP is used for fetching emails. POP stands for Post Office Protocol, and is used for polling your email server for new mails. See SMTP above.

Encryption

Encryption allows you to communicate with computers without any one being able to read the communication in plain text. We will not go into how encryption works on a mathematical or technical level, but we will show you where and how you can use it. We would also like to point out that encryption hides what you are saying but not that you are communicatin, nor with whom you are communicating. It is important to be aware that using e.g. encrypted email communication does nothing to hide the metadata about when you are sending an email, and to whom you are sending an email. The same goes for using HTTPS on the web. There will still be traces and proof of your accessing a certain web site or service, but the actual communication will be encrypted.

If you want to make your activities even more private than simply using encryption, you should consider using a VPN (virtural private network), where you will be logging on to a network with encryption, and then all your communications will go via the virtual network, so that accessing a web site can't (easily) be connected to your person, since the traffic comes from a VPN rather than directly from your own network (at home or at school etc). Ideally, the only thing others can see, is that you are connected to a VPN, and that's it. If they want to get the metadata (what you did on the network and when), they need to get a warrant and obtain that information from the VPN provider. They could also get physical access to your computer and get traces of your activities from it.

Data

The common use of computer networks is to share information and data. When data is sent over the network (being exchanged between computers), we need to make sense of that data. For this reason, there are several formats for data that are commonly used on the Internet.

The data sent from one computer to another must be possible to interpret by the receiving computer. Let's say that two systems are connected over the Internet. One system needs some data about customers from the other system. How we encode and represent the data being sent, is of great importance, so that we can use the data correctly. Such data typically consists of metadata (data about the data, what the data represents) and raw data. Of course, the makers of the two systems can come up with their own proprietary data format so that the programmers know how to interpret the data being sent. But that limits the interoperability of systems and locks the customers in with one vendor. The two systems become dependent on each other, because of the special way data is encoded and represented.

An alternative to using proprietary data formats, is to use well-defined and commonly used formats such as XML, Json and CSV. (The latter is so simple, actually, so it hasn't even been standardized, but that's beside the point here.)

We'll look at some examples of all three formats here, in the context of customer data being exchanged between two business systems.

Customer data example

Let's start with the metadata, i.e. what the data stands for as a means of interpreting (and checking the validity of the data). Let's assume that the customer data needs the following fields of data about each customer:

  • Customer number (a six digit whole number)
  • Customer name (first name, last name as plain text)
  • Customer email (plain text email address)

Next, we'll look at how this data about customers could look in various formats.

XML

XML - eXtensible Markup Language

XML stands for eXtensible Markup Language, and is a well-structured plain-text data format. The basis of an XML document, is tags surrounding parts of data. Tags surround the data by having an opening tag and a closing tag. Tags can be nested in an hierarchical manner, but the rule is that a tag inside another tag must be closed before the surrounding tag closes. Don't worry, you will see examples soon! The start tag together with the data and end tag is called an element.

A start-tag starts with a less-than sign, then the tag name, possible attributes, then ends with a greater-than sign. This is what it could look like: <customers>. A start tag with attributes could look like this: <email type="work"> .

An end-tag starts with a less-than sign, followed by a forward slash, then the tag name again and ends with a greater-than sign: </customers>.

As you saw, the attributes (you can have more than one) consist of an attribute name, an equal sign and a value surrounded in double quotes: type="work" .

Now, as we said, tags can be nested to create a hierarchy. If we think about the first example, <customers>, the name suggest that there will be more than one customer in the data. That's not a coincident. This is rather typical of XML data. This allows us to enumerate a number of <customer> elements within the surrounding <customers> element:

<customers>
  <customer number="000001">
    <name>
      <firstname>Henrik</firstname>
      <lastname>Sandklef</lastname>
    </name>
    <email type="work">henke@gu.se</email>
    <email type="home">henke@henkeshome.se</email>
  </customer>
  <customer number="000002">
    <name>
      <firstname>Rikard</firstname>
      <lastname>Fröberg</lastname>
    </name>
    <email type="work">rille@gu.se</email>
    <email type="home">rille@rilleshome.se</email>
  </customer>
</customers>

Note that we arbitrarily chose that customer number be an attribute of the customer tag, but we could just us well have chosen to make it an element to be nested inside the customer element. A rule-of-thumb is to make metadata attributes and other data elements, but it is impossible to draw the line sometimes. There are some recommendations if you want to know more. From the rules of XML's point of view, this doesn't matter.

The very first line of an XML document should be an XML declaration which can look like this: <?xml version="1.0" encoding="UTF-8"?> .

The first opening tag of the document starts what is called the root element. So the root element of the example is the customers element (the whole list of customers comes inside this root element). There can only be one root element (this is another rule).

The full document with data for the two customers now becomes:

<?xml version="1.0" encoding="UTF-8"?>
<customers>
  <customer number="000001">
    <name>
      <firstname>Henrik</firstname>
      <lastname>Sandklef</lastname>
    </name>
    <email type="work">henke@gu.se</email>
    <email type="home">henke@henkeshome.se</email>
  </customer>
  <customer number="000002">
    <name>
      <firstname>Rikard</firstname>
      <lastname>Fröberg</lastname>
    </name>
    <email type="work">rille@gu.se</email>
    <email type="home">rille@rilleshome.se</email>
  </customer>
</customers>

Please look at the indentation of the document. While this isn't required, it is recommended, because one of the ideas behind XML data, is that both humans and computers will probably look at it. The same can be said about most data formats, and also about program source code!

If you want a deeper look at XML, we recommend this page here on the wiki: Introduction_to_XML. We suggest that you open that page in a new tab, read it and close it afterwards, so that you easily can find your way back here.

Json

Json stands for JavaScript Object Notation, and was created as a way to serialize objects in JavaScript. Today it is one of the most popular data formats for data sent over networks (this is not a scientifically backed up assessment, but rather one based on notions and observations).

The structure of a Json document, is less verbose than that of XML. This is because you don't repeat the element names in closing tags, but rather use curly braces instead. Json is build around blocks of name-value pairs. Pairs can be grouped using curly braces as blocks, and blocks can be nested. The correct term isn't "block" actually, it is Object. Json also defines the concept of "arrays" which means ordered lists of related data, grouped by square brackets and separated by commas. All names should be enclosed by double quotes.

Perhaps examples convey the ideas better than words!

A simple object with one name-value pair could look like this:

{
  "firstName": "Rikard"
}

A simple object with a few name-value pairs could look like this:

{
  "customerNumber": "000002"
  "firstName": "Rikard",
  "lastName": "Fröberg",
  "email": "rille@gu.se"
}

Note here that the values come inside double quotes. This is not a rule per se, it is used when the value represents a string and not a number. Numbers are written without double quotes, which signals to the receiving end that the value represents a number. There are also some literals that can be used to signify other types of data:

  • true Boolean true
  • false Boolean false
  • null Absence of an optional value

Arrays look like this:

{ "customers":[
  {
    "customerNumber": "000001"
    "firstName": "Henrik",
    "lastName": "Sandklef",
    "email": "henke@gu.se"
  },
  {
    "customerNumber": "000001"
    "firstName": "Rikard",
    "lastName": "Fröberg",
    "email": "rille@gu.se"
  }]
}

In the example above, we have a named array called customers with two object elements separated by comma. But the named array must reside inside an (anonymous) object. This is because a valid Json document is one of:

  • An object
  • An array
  • A string
  • A literal (true, false, null)

A perfectly valid Json document could have the same data as an unnamed array:

[
  {
    "customerNumber": "000001"
    "firstName": "Henrik",
    "lastName": "Sandklef",
    "email": "henke@gu.se"
  },
  {
    "customerNumber": "000001"
    "firstName": "Rikard",
    "lastName": "Fröberg",
    "email": "rille@gu.se"
  }
]

The only difference here, is that the array has no name, which may or may not be a problem for humans reading the document. And of course, the document becomes a little shorter, since the surrounding object braces aren't needed.

If you want to learn more about Json, we recommend this page here on the wiki: Introduction_to_JSON. As usual, we recommend that you open and read the page in a new tab, so that you simply can close that tab when you're done, and quickly find you way back here.

CSV

CSV stands for Comma-Separated Values, and is a simplistic plain-text format for data. Typically you have column names on the first row, separated by commas. Then one line per "item", with comma-separated values corresponding to the column order:

customer_number,first_name,last_name,email
000001,Henrik,Sandklef,henke@gu.se
000002,Rikard,Fröberg,rille@gu.se

One advantage of CSV files, is that they are small (and simple). They can often be opened by spreadsheet applications like LibreOffice Calc or Microsoft Excel (who will make the columns columns of a spreadsheet and put the values in cell rows under the correct column).

Problem comes, of course, when you have values that also contain commas. The common solution to that, is to surround values by double-quotes if they contain at least one comma. Most spreadsheet applications handle that. A draw-back is that it is harder to decide or figure out what types the values have. How would an application treat the customer numbers? As the numbers 1 and 2, or as the strings "000001" and "000002"? This can often be handled during the import of the CSV file, in an interactive dialog in the application.

Another advantage is that many databases can import CSV files (with the same caveat about types as mentioned above). But that actually goes for XML and Json too. Quite a few database engines can import CSV, XML and Json (and probably some can handle even more formats.

Try and create a CSV file and import it into LibreOffice Calc or MS Excel. If you don't have any spreadsheet application, you can install LibreOffice if you have the time. It is free software (and free of charge too). You could also try GNUmeric (also free software) or even Google Sheets on Drive (if you have a Gmail account).

Relation between network and application protocols and data formats

From a technical point of view, there are protocols for sending data over the network. At the lowest level, we have the IP protocol for creating packages of data to be transmitted over the Internet with source and destination IP addresses, and the actual data fragmented into a package.

Then we have the TCP protocol for how the actual communication takes place; error handling, keeping state about the connection, putting the packages together in the correct order, handling re-sending of packages if some are lost etc.

Above this level we have application or service protocols, like HTTP for applications talking to web servers. As the data part of HTTP, we can have data encoded as e.g. Json, XML och CSV.

This could be described by the following ASCII art:

+--------------------------------+
| HTTP                           |
| Headers                        |
| Data (can be Json)             |
+--------------------------------+
| TCP                            |
| manage packages, connections,  |
| ports, errors                  |
+--------------------------------+
| IP                             |
| Header with metadata           |
| Source IP                      |
| Destination IP                 |
| Binary data package (part of   |
| the data)                      |
+--------------------------------+

So, when you request a Json document from a web server, your browser uses HTTP to tell the server what it wants. Your browser uses the operating system to set up a connection to the web server and the data of your request is packaged in packages handled by the IP protocol. The TCP protocol helps with the communication between the client and server, and makes sure all packages arrive and that they are pieced together again in the correct order. The server sends a response in HTTP with some headers and the data as JSON. The response is also fragmented in packages sent via IP, and TCP. TCP helps your computer to puzzle together the packages with the response, request re-sending of missing packages etc.

One way of expressing this is to say that the data was sent as Json over HTTP which is a protocol used over TCP/IP.

We bring this up in hope of avoiding confusion about what is a protocol and what is a data format.

Links

Further reading

Workshop slides

The following APIs are used during the workshop:

Slides for the summary in-classroom lecture

Videos and presentations

English videos

Swedish videos

Navigation

The next page is Networks_and_protocols_-_Exercises.

« PreviousBook TOCNext »