Exercises on HTTP

These exercises are tested on Ubuntu, Fedora, MacOS and cygwin using netcat as installed using Juneday's installation scripts (Software Used). Please note that there might be small differences on how to use these commands on cygwin and mac os (as well as differences between different versions of bash on different mac os versions). Unfortunately there are a couple of programs called netcat/nc.

We try to keep the list below updated:

MacOS:
- HomeBrew: netcat is started using nc (alt /usr/local/bin/nc)
- MacPorts: netcat is started using nc6 (alt /opt/local/bin/nc6)
- Without the above the netcat exercises below are, as far as we know, not possible to do
Cygwin: nc6
Debian, Ubuntu, Fedora, RedHat: nc

A good way to figure out what to do if something below doesn't work on your computer, is to as the manual (e.g. man nc) or ask the command itself:

nc -help
OpenBSD netcat (Debian patchlevel 1.105-7ubuntu1)
This is nc from the netcat-openbsd package. An alternative nc is available
in the netcat-traditional package.
usage: nc [-46bCDdhjklnrStUuvZz] [-I length] [-i interval] [-O length]
	  [-P proxy_username] [-p source_port] [-q seconds] [-s source]
	  [-T toskeyword] [-V rtable] [-w timeout] [-X proxy_protocol]
	  [-x proxy_address[:port]] [destination] [port]
	Command Summary:
		-4		Use IPv4
		-6		Use IPv6
		-b		Allow broadcast
		-C		Send CRLF as line-ending
		-D		Enable the debug socket option
		-d		Detach from stdin
		-h		This help text
		-I length	TCP receive buffer length
		-i secs		Delay interval for lines sent, ports scanned
		-j		Use jumbo frame
		-k		Keep inbound sockets open for multiple connects
		-l		Listen mode, for inbound connects
		-n		Suppress name/port resolutions
		-O length	TCP send buffer length
		-P proxyuser	Username for proxy authentication
		-p port		Specify local port for remote connects
        	-q secs		quit after EOF on stdin and delay of secs
		-r		Randomize remote ports
		-S		Enable the TCP MD5 signature option
		-s addr		Local source address
		-T toskeyword	Set IP Type of Service
		-t		Answer TELNET negotiation
		-U		Use UNIX domain socket
		-u		UDP mode
		-V rtable	Specify alternate routing table
		-v		Verbose
		-w secs		Timeout for connects and final net reads
		-X proto	Proxy protocol: "4", "5" (SOCKS) or "connect"
		-x addr[:port]	Specify proxy address and port
		-Z		DCCP mode
		-z		Zero-I/O mode [used for scanning]
	Port numbers can be individual or ranges: lo-hi [inclusive]

Trouble shooting

If you run into trouble while running the various scripts on this page, you can look at our Bash FAQ for some common problems and their solutions.

Exercises

Find header of www.sunet.se

Use a command line program (curl, wget, telnet, nc (netcat), ..) to find the HTTP headers sent by www.sunet.se

Expand using link to the right to see a suggested solution/answer.

Curl:

$curl --head www.sunet.se
HTTP/1.1 301 Moved Permanently
Date: Sat, 18 Nov 2017 12:21:45 GMT
Server: Apache/2.4.7 (Ubuntu)
Location: https://www.sunet.se/
Content-Type: text/html; charset=iso-8859-1

Wget: Wget is a bit tougher to use to find the headers. Here's our suggested way to do find the headers:

$ wget --server-response www.sunet.se 
--2017-11-18 13:25:18--  http://www.sunet.se/
Resolving www.sunet.se (www.sunet.se)... 192.36.171.231, 2001:6b0:8:2::233, 2001:6b0:8:2::232
Connecting to www.sunet.se (www.sunet.se)|192.36.171.231|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 301 Moved Permanently
  Date: Sat, 18 Nov 2017 12:25:18 GMT
  Server: Apache/2.4.7 (Ubuntu)
  Location: https://www.sunet.se/
  Content-Length: 306
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html; charset=iso-8859-1
Location: https://www.sunet.se/ [following]
--2017-11-18 13:25:18--  https://www.sunet.se/
Connecting to www.sunet.se (www.sunet.se)|192.36.171.231|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Sat, 18 Nov 2017 12:25:19 GMT
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 11627
  Content-Type: text/html; charset=UTF-8
  Age: 0
  X-Cache: MISS
  X-Cache-Hits: 0
  Connection: keep-alive
  Accept-Ranges: bytes
Length: 11627 (11K) [text/html]
Saving to: ‘index.html’

index.html          100%[===================>]  11,35K  --.-KB/s    in 0,001s  

2017-11-18 13:25:19 (7,80 MB/s) - ‘index.html’ saved [64246]

telnet: You use telnet interactively so to make things clearer we've highlighted the stuff you should write:

$ telnet www.sunet.se 80
Trying 192.36.171.231...
Connected to www.sunet.se.
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.1 301 Moved Permanently
Date: Sat, 18 Nov 2017 12:28:42 GMT
Server: Apache/2.4.7 (Ubuntu)
Location: https:///
Connection: close
Content-Type: text/html; charset=iso-8859-1

Connection closed by foreign host.

netcat or nc: You use netcat interactively so to make things clearer we've highlighted the stuff you should write:

$ nc www.sunet.se 80
HEAD / HTTP/1.0

HTTP/1.1 301 Moved Permanently
Date: Sat, 18 Nov 2017 12:35:23 GMT
Server: Apache/2.4.7 (Ubuntu)
Location: https:///
Connection: close
Content-Type: text/html; charset=iso-8859-1

you may need to end the session by pressing Ctrl-d.

lwp-request -m HEAD (or HEAD):

$ HEAD http://www.sunet.se/
200 OK
Connection: close
Date: Sat, 18 Nov 2017 12:39:02 GMT
Age: 0
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Client-Date: Sat, 18 Nov 2017 12:39:02 GMT
Client-Peer: 192.36.171.231:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=NL/ST=Noord-Holland/L=Amsterdam/O=TERENA/CN=TERENA SSL CA 3
Client-SSL-Cert-Subject: /C=SE/ST=Stockholm/L=Stockholm/O=SUNET/CN=*.sunet.se
Client-SSL-Cipher: ECDHE-RSA-AES256-GCM-SHA384
Client-SSL-Socket-Class: IO::Socket::SSL
X-Cache: MISS
X-Cache-Hits: 0

Visit a simple HTML page

Point your browser to http://wiki.juneday.se/example-data/simple.html. View the source of the page.

Expand using link to the right to see a suggested solution/answer.

On chrome and firefox you press Ctrl-u

Use a command line program to fetch the same simple page

Use a command line program to download the same web page (http://wiki.juneday.se/example-data/simple.html) as you viewed with a browser.

Expand using link to the right to see a suggested solution/answer.

Wget:

$ wget http://wiki.juneday.se/example-data/simple.html
--2017-11-18 16:28:55--  http://wiki.juneday.se/example-data/simple.html
Resolving wiki.juneday.se (wiki.juneday.se)... 129.16.69.98
Connecting to wiki.juneday.se (wiki.juneday.se)|129.16.69.98|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 83 [text/html]
Saving to: ‘simple.html’

simple.html         100%[===================>]      83  --.-KB/s    in 0s      

2017-11-18 16:28:55 (3,76 MB/s) - ‘simple.html’ saved [83/83]

Curl:

$curl -o simple.html http://wiki.juneday.se/example-data/simple.html 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    83  100    83    0     0     83      0  0:00:01 --:--:--  0:00:01  1566

Compare the content of the source you viewed in the previous exercise with the content of the file simple.html. Are they similar?

Expand using link to the right to see a suggested solution/answer.

The contents should be the same:

<html>
<head><title>simple</title></head>
<body>
<p>Hello HTTP</p>
</body>
</html>

Netcat in listening mode

You can start up netcat in something called listening mode, which makes netcat listen for incoming connections (netcat acts like a server). A webserver also listens to incoming connections. The web server waits until some client (browser, curl ..) connects and then the two can begin to communicate (using http). You should start netcat (listening mode using port 8181) in one terminal. In a way this is faking be a proper web server.

Note: check the text above about netcat/nc to make sure you invoke netcat/nc using the correct name.

Expand using link to the right to see a suggested solution/answer.

$ nc -l -p 8181

Note: netcat/nc need to be started using different names on the platoforms Juneday supports. Here's a list:

MacOS:
- HomeBrew: netcat is started using nc (alt /usr/local/bin/nc)
- MacPorts: netcat is started using nc6 (alt /opt/local/bin/nc6)
- Without the above the netcat exercises below are, as far as we know, not possible to do
Cygwin: nc6
Debian, Ubuntu, Fedora, RedHat: nc

Connect to your server

You should now start up netcat in normal mode on port 8181 in another terminal, while keeping the listening netcat running. Note: You should have two terminals with one netcat process running in each terminal.

Expand using link to the right to see a suggested solution/answer.

$ nc localhost 8181

Send data between the two

Type in some data in one of the terminals and see what happends.

Expand using link to the right to see a suggested solution/answer.

Typing data in the listening netcat:

Hi there

the following can be seen in the client netcat:

Hi there

If we type something in the client:

I am a client

we should be able to see the same text in the listening netcat:

I am a client

Using netcat we can send data between processes. This is what is done with a normal webserver. Can we use netcat as a webserver? Checkc out the following exercises.

Write a script printing HTTP header and HTML content

Write a script that prints out a header, typically like this

HTTP/1.0 200 OK
Connection: close
Date: sat nov 18 21:15:21 CET 2017
Server: netcat special deal
Content-Type: text/html; charset=utf-8
Cache-Control: max-age=60

and some content, typically:

<html>
<head><title>Some web page title</title></head>
<body>
<p>Hello HTTP</p>
</body>
</html>

Invoke the program to make sure it works.

Expand using link to the right to see a suggested solution/answer.

Store the following text in a text file, typically called wserver.sh

#!/bin/bash



#
# Function to output header
#
header()
{
    echo "HTTP/1.0 200 OK"
    echo "Connection: close"
    echo "Date: $(date)"
    echo "Server: netcat special deal"
    echo "Content-Type: text/html; charset=utf-8"
    echo "Cache-Control: max-age=60"
    echo ""
    echo ""
}

#
# Function to outout content 
#
content()
{
    echo "<html>"
    echo "<head><title>Some web page title</title></head>"
    echo "<body>"
    echo "<p>Hello HTTP</p>"
    echo "</body>"
    echo "</html>"
}

CONTENT=$(content)
CLENGTH=$(echo $CONTENT | wc -c)
LENGTH=$(( $CLENGTH + 1  ))
header
echo ${CONTENT}
# close with EOF
exec 1>&-

Make sure to make the script executable: chmod a+x wserver.sh and then invoke the script:

$ ./wserver.sh
HTTP/1.0 200 OK
Connection: close
Date: lör nov 18 21:24:48 CET 2017
Server: netcat special deal
Content-Type: text/html; charset=utf-8
Cache-Control: max-age=60


<html>
<head><title>Some web page title</title></head>
<body>
<p>Hello HTTP</p>
</body>
</html>

The source above can be found here: wserver.sh or here wserver.sh if you want to view in your browser.

Start netcat in listning mode and use wserver.sh

We can now start netcat in listening mode on port 8181 and use wserver.sh to "answer" (output) text when someone connects. To start netcat in listnening mode we do:

$ nc -l -p 8181

but then we need to type in the HTTP response and the HTML content our selves. Let's use pipes instead:

$ ./wserver.sh | nc -l -p 8181

Connect to the above server using curl

Use curl to connect to localhost on port 8181. What do you see?

Expand using link to the right to see a suggested solution/answer.

$ curl localhost:8181

<html>
<head><title>Some web page title</title></head>
<body>
<p>Hello HTTP</p>
</body>
</html>

What we see is the content from wserver.sh. Cool isn't it :)

Connect to the above server using a browser

Start netcat in listening mode with wserver.sh as you did before. Now, instead of connecting using curl, you should use a browser. Type in http//localhost:8181 in the address bar (location). What do you see?

Use curl to connect to localhost on port 8181. What do you see?

Expand using link to the right to see a suggested solution/answer.

You should see

Hello HTTP

Connect to the above server using a browser

Try reloading the page above. It doesnät work. Why?

Expand using link to the right to see a suggested solution/answer.

Since the netcat (listneing mode) exits after communication is done there is no longer a server listening on port 8181.

Make netcat (listening mode) listen forever

You can use a while loop in bash to make netcat start up again once it has finished communicating with a client. If you do this you can reload the page in a browser.

Expand using link to the right to see a suggested solution/answer.

You should see

$ while (true); do ./wserver.sh | nc -l -p 8181; done

Download an image using curl

You can use curl to download other stuff than HTML pages. Let's try with this image http://wiki.juneday.se/example-data/progund.png:

Expand using link to the right to see a suggested solution/answer.

$ curl --output progund.png http://wiki.juneday.se/example-data/progund.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10135  100 10135    0     0  10135      0  0:00:01 --:--:--  0:00:01  224k

or

$ curl -Oq http://wiki.juneday.se/example-data/progund.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10135  100 10135    0     0  10135      0  0:00:01 --:--:--  0:00:01  224k

Note that also curl may come in different versions. If you want to investigate the various options, you can ask curl what they mean:

$ curl --help |grep output
 -f, --fail          Fail silently (no output at all) on HTTP errors (H)
 -i, --include       Include protocol headers in the output (H/F)
 -N, --no-buffer     Disable buffering of the output stream
 -o, --output FILE   Write to FILE instead of stdout
 -O, --remote-name   Write output to a file named as the remote file
 -R, --remote-time   Set the remote file's time on the local output
 -s, --silent        Silent mode (don't output anything)
     --trace-ascii FILE  Like --trace, but without hex output
     --trace-time    Add time stamps to trace/verbose output
 -w, --write-out FORMAT  Use output FORMAT after completion

$ curl --help |grep '\-O'
 -O, --remote-name   Write output to a file named as the remote file

Providing data in JSON format

Let's say we want to provide a client (a program) with a list of products. One way would be to open up our database and let anyone connect. Not very secure is it? Ok, we know, you could require user and password, but still this is not a good solution. The usual way to transfer information such as a list of products is to use XML or JSON. Instead of writing a server reading data from a database and producing JSON we can fake it all. This is oftan a useful strategy if you want to focus on the client insetad of the server. It is not far fetched to imagine a case where a development team is divided into two smaller teams. One developing the server and the other team developing the client.

Download (info on downloading) the script produce-json-static.sh. Make sure the script has execute permission (chmod a+x produce-json-static.sh). You can now execute the script:

$ ./produce-json-static.sh
HTTP/1.0 200 OK
Connection: close
Date: Sat Nov 18 23:17:22 CET 2017
Server: netcat special deal
Content-Type: application/json
Cache-Control: max-age=60

[
  {
    "name" : "Johanneshof Reinisch Pinot Noir",
    "price" : "143.0",
    "alcohol" : "13.0"
  },
  {
    "name" : "Dobogó Tokaji Furmint",
    "price" : "187.0",
    "alcohol" : "13.5"
  }
]

You should now start up netcat in listening mode and use the script's output to be sent to the client. The netcat program should restart for ever and ever ....

Expand using link to the right to see a suggested solution/answer.

 while (true) ; do ./produce-json-static.sh | nc -l -p 8181 ; done

Providing data from a database

This time we're going to use We have written a small script that outputs (to stdout) products in a database in JSON format.

Note: we do not require you to understand the JSON format nor the bash script producing JSON. We want you to have seen a typical way of transferring data between a client and a server.

Download the script produce-json.sh and the database bolaget.db. Make sure the script has execute permission (chmod a+x produce-json.sh). You can now execute the script:

$ ./produce-json.sh

The script limits the number of products to 20. If you want to output them all:

$ ./produce-json.sh --all

If you want to use another limit (e g 3)

$ ./produce-json.sh 3

Ok, finally we can start to work. You should now start up netcat in listening mode and use the script's output to be sent to the client. The netcat program should restart for ever and ever ....

Note: this bash script is painfully slow. We do not in any way recommend writing critical webservers using bash together with netcat. We use it here since we believe it gives you a good insight in how data are transferred.

Expand using link to the right to see a suggested solution/answer.

 while (true) ; do ./produce-json.sh 2 | nc -l -p 8181 ; done

Access the JSON data using curl

Use curl (in another terminal) to get the JSON data.

Expand using link to the right to see a suggested solution/answer.

$ curl localhost:8181
[
  {
    "name" : "Johanneshof Reinisch Pinot Noir",
    "price" : "143.0",
    "alcohol" : "13.0"
  },
  {
    "name" : "Dobogó Tokaji Furmint",
    "price" : "187.0",
    "alcohol" : "13.5"
  }
]

Links

Solutions (source code)

TODO: Verify that these are the correct files

Where to go next

TODO: decide next page in the Programming_with_Java book

« Previous • Book TOC • Next »

HTTP - Exercises

Contents

Exercises on HTTP

Trouble shooting

Exercises

Find header of www.sunet.se

Visit a simple HTML page

Use a command line program to fetch the same simple page

Netcat in listening mode

Connect to your server

Send data between the two

Write a script printing HTTP header and HTML content

Start netcat in listning mode and use wserver.sh

Connect to the above server using curl

Connect to the above server using a browser

Connect to the above server using a browser

Make netcat (listening mode) listen forever

Download an image using curl

Providing data in JSON format

Providing data from a database

Access the JSON data using curl

Links

Solutions (source code)

Where to go next

Navigation menu

Search