View source for HTTP - Exercises

=Exercises on HTTP=
These exercises are tested on Ubuntu, Fedora, MacOS and cygwin using netcat as installed using Juneday's installation scripts ([[Software_used|Software Used]]). Please note that there might be small differences on how to use these commands on cygwin and mac os (as well as differences between different versions of bash on different mac os versions). Unfortunately there are a couple of programs called netcat/nc.

We try to keep the list below updated:
* MacOS:
** HomeBrew: netcat is started using <code>nc</code> (alt <code>/usr/local/bin/nc</code>)
** MacPorts: netcat is started using <code>nc6</code> (alt <code>/opt/local/bin/nc6</code>)
** Without the above the netcat exercises below are, as far as we know, not possible to do
* Cygwin: <code>nc6</code>
* Debian, Ubuntu, Fedora, RedHat: <code>nc</code>


A good way to figure out what to do if something below doesn't work on your computer, is to as the manual (e.g. <code>man nc</code>) or ask the command itself:
<source lang="Bash">
nc -help
OpenBSD netcat (Debian patchlevel 1.105-7ubuntu1)
This is nc from the netcat-openbsd package. An alternative nc is available
in the netcat-traditional package.
usage: nc [-46bCDdhjklnrStUuvZz] [-I length] [-i interval] [-O length]
	  [-P proxy_username] [-p source_port] [-q seconds] [-s source]
	  [-T toskeyword] [-V rtable] [-w timeout] [-X proxy_protocol]
	  [-x proxy_address[:port]] [destination] [port]
	Command Summary:
		-4		Use IPv4
		-6		Use IPv6
		-b		Allow broadcast
		-C		Send CRLF as line-ending
		-D		Enable the debug socket option
		-d		Detach from stdin
		-h		This help text
		-I length	TCP receive buffer length
		-i secs		Delay interval for lines sent, ports scanned
		-j		Use jumbo frame
		-k		Keep inbound sockets open for multiple connects
		-l		Listen mode, for inbound connects
		-n		Suppress name/port resolutions
		-O length	TCP send buffer length
		-P proxyuser	Username for proxy authentication
		-p port		Specify local port for remote connects
        	-q secs		quit after EOF on stdin and delay of secs
		-r		Randomize remote ports
		-S		Enable the TCP MD5 signature option
		-s addr		Local source address
		-T toskeyword	Set IP Type of Service
		-t		Answer TELNET negotiation
		-U		Use UNIX domain socket
		-u		UDP mode
		-V rtable	Specify alternate routing table
		-v		Verbose
		-w secs		Timeout for connects and final net reads
		-X proto	Proxy protocol: "4", "5" (SOCKS) or "connect"
		-x addr[:port]	Specify proxy address and port
		-Z		DCCP mode
		-z		Zero-I/O mode [used for scanning]
	Port numbers can be individual or ranges: lo-hi [inclusive]
</source>
==Trouble shooting==
If you run into trouble while running the various scripts on this page, you can look at our [[Bash FAQ]] for some common problems and their solutions.

=Exercises=
==Find header of www.sunet.se== 
Use a command line program ([[curl]], [[wget]], [[telnet]], [[nc|nc (netcat)]], ..) to find the HTTP headers sent by <code>www.sunet.se</code>
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
[[Curl]]:
<source lang="bash">
$curl --head www.sunet.se
HTTP/1.1 301 Moved Permanently
Date: Sat, 18 Nov 2017 12:21:45 GMT
Server: Apache/2.4.7 (Ubuntu)
Location: https://www.sunet.se/
Content-Type: text/html; charset=iso-8859-1
</source>
[[Wget]]:
Wget is a bit tougher to use to find the headers. Here's our suggested way to do find the headers:
<source lang="bash">
$ wget --server-response www.sunet.se 
--2017-11-18 13:25:18--  http://www.sunet.se/
Resolving www.sunet.se (www.sunet.se)... 192.36.171.231, 2001:6b0:8:2::233, 2001:6b0:8:2::232
Connecting to www.sunet.se (www.sunet.se)|192.36.171.231|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 301 Moved Permanently
  Date: Sat, 18 Nov 2017 12:25:18 GMT
  Server: Apache/2.4.7 (Ubuntu)
  Location: https://www.sunet.se/
  Content-Length: 306
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html; charset=iso-8859-1
Location: https://www.sunet.se/ [following]
--2017-11-18 13:25:18--  https://www.sunet.se/
Connecting to www.sunet.se (www.sunet.se)|192.36.171.231|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Sat, 18 Nov 2017 12:25:19 GMT
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 11627
  Content-Type: text/html; charset=UTF-8
  Age: 0
  X-Cache: MISS
  X-Cache-Hits: 0
  Connection: keep-alive
  Accept-Ranges: bytes
Length: 11627 (11K) [text/html]
Saving to: ‘index.html’

index.html          100%[===================>]  11,35K  --.-KB/s    in 0,001s  

2017-11-18 13:25:19 (7,80 MB/s) - ‘index.html’ saved [64246]
</source>
[[telnet]]:
You use telnet interactively so to make things clearer we've highlighted the stuff you should write:
<source lang="bash" highlight="1,5-6">
$ telnet www.sunet.se 80
Trying 192.36.171.231...
Connected to www.sunet.se.
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.1 301 Moved Permanently
Date: Sat, 18 Nov 2017 12:28:42 GMT
Server: Apache/2.4.7 (Ubuntu)
Location: https:///
Connection: close
Content-Type: text/html; charset=iso-8859-1

Connection closed by foreign host.
</source>
[[nc|netcat or nc]]:
You use netcat interactively so to make things clearer we've highlighted the stuff you should write:
<source lang="bash" highlight="1-2">
$ nc www.sunet.se 80
HEAD / HTTP/1.0

HTTP/1.1 301 Moved Permanently
Date: Sat, 18 Nov 2017 12:35:23 GMT
Server: Apache/2.4.7 (Ubuntu)
Location: https:///
Connection: close
Content-Type: text/html; charset=iso-8859-1
</source>
you may need to end the session by pressing <code>Ctrl-d</code>.

[[lwp-request|lwp-request]] -m HEAD (or HEAD):
<source lang="bash">
$ HEAD http://www.sunet.se/
200 OK
Connection: close
Date: Sat, 18 Nov 2017 12:39:02 GMT
Age: 0
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Client-Date: Sat, 18 Nov 2017 12:39:02 GMT
Client-Peer: 192.36.171.231:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=NL/ST=Noord-Holland/L=Amsterdam/O=TERENA/CN=TERENA SSL CA 3
Client-SSL-Cert-Subject: /C=SE/ST=Stockholm/L=Stockholm/O=SUNET/CN=*.sunet.se
Client-SSL-Cipher: ECDHE-RSA-AES256-GCM-SHA384
Client-SSL-Socket-Class: IO::Socket::SSL
X-Cache: MISS
X-Cache-Hits: 0
</source>
</div></div>

==Visit a simple HTML page== 

Point your browser to [http://wiki.juneday.se/example-data/simple.html http://wiki.juneday.se/example-data/simple.html]. View the source of the page.
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
On chrome and firefox you press <code>Ctrl-u</code> 
</div></div>

==Use a command line program to fetch the same simple page== 

Use a command line program to download the same web page ([http://wiki.juneday.se/example-data/simple.html http://wiki.juneday.se/example-data/simple.html]) as you viewed with a browser.

<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
[[Wget]]:
<source lang="bash">
$ wget http://wiki.juneday.se/example-data/simple.html
--2017-11-18 16:28:55--  http://wiki.juneday.se/example-data/simple.html
Resolving wiki.juneday.se (wiki.juneday.se)... 129.16.69.98
Connecting to wiki.juneday.se (wiki.juneday.se)|129.16.69.98|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 83 [text/html]
Saving to: ‘simple.html’

simple.html         100%[===================>]      83  --.-KB/s    in 0s      

2017-11-18 16:28:55 (3,76 MB/s) - ‘simple.html’ saved [83/83]
</source>
[[Curl]]:
<source lang="bash">
$curl -o simple.html http://wiki.juneday.se/example-data/simple.html 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    83  100    83    0     0     83      0  0:00:01 --:--:--  0:00:01  1566
</source>
</div></div>

Compare the content of the source you viewed in the previous exercise with the content of the file <code>simple.html</code>. Are they similar?
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
The contents should be the same:
<source lang="html">
<html>
<head><title>simple</title></head>
<body>
<p>Hello HTTP</p>
</body>
</html>
</source>
</div></div>

==Netcat in listening mode== 

You can start up netcat in something called listening mode, which makes netcat listen for incoming connections (netcat acts like a server). A webserver also listens to incoming connections. The web server waits until some client (browser, curl ..) connects and then the two can begin to communicate (using http). You should start netcat (listening mode using port <code>8181</code>) in one terminal. In a way this is faking be a proper web server.

'''''Note:''' check the text above about netcat/nc to make sure you invoke netcat/nc using the correct name''. 

<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
<source lang="bash">
$ nc -l -p 8181
</source>
'''Note:''' netcat/nc need to be started using different names on the platoforms Juneday supports. Here's a list:
* MacOS:
** HomeBrew: netcat is started using <code>nc</code> (alt <code>/usr/local/bin/nc</code>)
** MacPorts: netcat is started using <code>nc6</code> (alt <code>/opt/local/bin/nc6</code>)
** Without the above the netcat exercises below are, as far as we know, not possible to do
* Cygwin: <code>nc6</code>
* Debian, Ubuntu, Fedora, RedHat: <code>nc</code>
 

</div></div>

==Connect to your server== 

You should now start up netcat in normal mode on port <code>8181</code> in another terminal, while keeping the listening netcat running.
''Note: You should have two terminals with one netcat process running in each terminal.''
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
<source lang="bash">
$ nc localhost 8181
</source>
</div></div>

==Send data between the two==

Type in some data in one of the terminals and see what happends.

<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
Typing data in the listening netcat:
<source lang="bash">
Hi there
</source>
the following can be seen in the client netcat:
<source lang="bash">
Hi there
</source>
If we type something in the client:
<source lang="bash">
I am a client
</source>
we should be able to see the same text in the listening netcat:
<source lang="bash">
I am a client
</source>
</div></div>
Using netcat we can send data between processes. This is what is done with a normal webserver. Can we use netcat as a webserver? Checkc out the following exercises.

==Write a script printing HTTP header and HTML content== 
Write a script that prints out a header, typically like this
<source lang="text">
HTTP/1.0 200 OK
Connection: close
Date: sat nov 18 21:15:21 CET 2017
Server: netcat special deal
Content-Type: text/html; charset=utf-8
Cache-Control: max-age=60


</source>
and some content, typically:
<source lang="html">
<html>
<head><title>Some web page title</title></head>
<body>
<p>Hello HTTP</p>
</body>
</html>
</source>
Invoke the program to make sure it works.
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
Store the following text in a text file, typically called <code>wserver.sh</code>
<source lang="bash">
#!/bin/bash



#
# Function to output header
#
header()
{
    echo "HTTP/1.0 200 OK"
    echo "Connection: close"
    echo "Date: $(date)"
    echo "Server: netcat special deal"
    echo "Content-Type: text/html; charset=utf-8"
    echo "Cache-Control: max-age=60"
    echo ""
    echo ""
}

#
# Function to outout content 
#
content()
{
    echo "<html>"
    echo "<head><title>Some web page title</title></head>"
    echo "<body>"
    echo "<p>Hello HTTP</p>"
    echo "</body>"
    echo "</html>"
}

CONTENT=$(content)
CLENGTH=$(echo $CONTENT | wc -c)
LENGTH=$(( $CLENGTH + 1  ))
header
echo ${CONTENT}
# close with EOF
exec 1>&- 
</source>
Make sure to make the script executable: <code>chmod a+x wserver.sh</code> and then invoke the script:
<source lang="bash">
$ ./wserver.sh
HTTP/1.0 200 OK
Connection: close
Date: lör nov 18 21:24:48 CET 2017
Server: netcat special deal
Content-Type: text/html; charset=utf-8
Cache-Control: max-age=60


<html>
<head><title>Some web page title</title></head>
<body>
<p>Hello HTTP</p>
</body>
</html>
</source>
The source above can be found here: [https://raw.githubusercontent.com/progund/web-misc/master/exercises/wserver.sh wserver.sh] or here [https://github.com/progund/web-misc/blob/master/exercises/wserver.sh wserver.sh] if you want to view in your browser.
</div></div>

==Start netcat in listning mode and use wserver.sh== 
We can now start netcat in listening mode on port 8181 and use <code>wserver.sh</code> to "answer" (output) text when someone connects. To start netcat in listnening mode we do:
<source lang="bash">
$ nc -l -p 8181
</source>
but then we need to type in the HTTP response and the HTML content our selves. Let's use pipes instead:
<source lang="bash">
$ ./wserver.sh | nc -l -p 8181
</source>

==Connect to the above server using curl== 
Use curl to connect to <code>localhost</code> on port <code>8181</code>. What do you see?
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
<source lang="bash">
$ curl localhost:8181

<html>
<head><title>Some web page title</title></head>
<body>
<p>Hello HTTP</p>
</body>
</html>
</source>
What we see is the content from <code>wserver.sh</code>. Cool isn't it :)
</div></div>


==Connect to the above server using a browser== 
Start netcat in listening mode with <code>wserver.sh</code> as you did before. Now, instead of connecting using curl, you should use a browser. Type in <code>http//localhost:8181</code> in the address bar (location). What do you see?

Use curl to connect to <code>localhost</code> on port <code>8181</code>. What do you see?
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
You should see
<source lang="html">
Hello HTTP
</source>
</div></div>

==Connect to the above server using a browser== 
Try reloading the page above. It doesnät work. Why?
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
Since the netcat (listneing mode) exits after communication is done there is no longer a server listening on port <code>8181</code>. 
</div></div>

==Make netcat (listening mode) listen forever==
You can use a while loop in bash to make netcat start up again once it has finished communicating with a client. If you do this you can reload the page in a browser.

<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
You should see
<source lang="bash">
$ while (true); do ./wserver.sh | nc -l -p 8181; done
</source>
</div></div>

==Download an image using curl== 
You can use curl to download other stuff than HTML pages. Let's try with this image <code>http://wiki.juneday.se/example-data/progund.png</code>:
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
<source lang="bash">
$ curl --output progund.png http://wiki.juneday.se/example-data/progund.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10135  100 10135    0     0  10135      0  0:00:01 --:--:--  0:00:01  224k
</source>
or
<source lang="bash">
$ curl -Oq http://wiki.juneday.se/example-data/progund.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10135  100 10135    0     0  10135      0  0:00:01 --:--:--  0:00:01  224k
</source>
Note that also curl may come in different versions. If you want to investigate the various options, you can ask curl what they mean:
<pre>
$ curl --help |grep output
 -f, --fail          Fail silently (no output at all) on HTTP errors (H)
 -i, --include       Include protocol headers in the output (H/F)
 -N, --no-buffer     Disable buffering of the output stream
 -o, --output FILE   Write to FILE instead of stdout
 -O, --remote-name   Write output to a file named as the remote file
 -R, --remote-time   Set the remote file's time on the local output
 -s, --silent        Silent mode (don't output anything)
     --trace-ascii FILE  Like --trace, but without hex output
     --trace-time    Add time stamps to trace/verbose output
 -w, --write-out FORMAT  Use output FORMAT after completion

$ curl --help |grep '\-O'
 -O, --remote-name   Write output to a file named as the remote file
</pre>
</div></div>

==Providing data in JSON format== 
Let's say we want to provide a client (a program) with a list of products. One way would be to open up our database and let anyone connect. Not very secure is it? Ok, we know, you could require user and password, but still this is not a good solution. The usual way to transfer information such as a list of products is to use [[XML]] or [[JSON]].  Instead of writing a server reading data from a database and producing JSON we can fake it all. This is oftan a useful strategy if you want to focus on the client insetad of the server. It is not far fetched to imagine a case where a development team is divided into two smaller teams. One developing the server and the other team developing the client. 

Download ([[Download_files|info on downloading]]) the script [https://raw.githubusercontent.com/progund/web-misc/master/exercises/produce-json-static.sh produce-json-static.sh]. Make sure the script has execute permission (<code>chmod a+x produce-json-static.sh</code>). You can now execute the script:
<source lang="text">
$ ./produce-json-static.sh
HTTP/1.0 200 OK
Connection: close
Date: Sat Nov 18 23:17:22 CET 2017
Server: netcat special deal
Content-Type: application/json
Cache-Control: max-age=60

[
  {
    "name" : "Johanneshof Reinisch Pinot Noir",
    "price" : "143.0",
    "alcohol" : "13.0"
  },
  {
    "name" : "Dobogó Tokaji Furmint",
    "price" : "187.0",
    "alcohol" : "13.5"
  }
]
</source>

You should now start up netcat in listening mode and use the script's output to be sent to the client. The netcat program should restart for ever and ever ....
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
<source lang="bash">
 while (true) ; do ./produce-json-static.sh | nc -l -p 8181 ; done
</source>
</div></div>

==Providing data from a database== 
This time we're going to use 
We have written a small script that outputs (to stdout) products in a database in JSON format.

''Note: we do not require you to understand the JSON format nor the bash script producing JSON. We want you to have seen a typical way of transferring data between a client and a server.''

[[Download_files|Download]] the script [https://raw.githubusercontent.com/progund/web-misc/master/exercises/produce-json.sh produce-json.sh] and the database [https://github.com/progund/web-misc/raw/master/exercises/bolaget.db bolaget.db]. Make sure the script has execute permission (<code>chmod a+x produce-json.sh</code>). You can now execute the script:
<source lang="bash">
$ ./produce-json.sh 
</source>
The script limits the number of products to 20. If you want to output them all:
<source lang="bash">
$ ./produce-json.sh --all 
</source>
If you want to use another limit (e g 3)
<source lang="bash">
$ ./produce-json.sh 3 
</source>
Ok, finally we can start to work. You should now start up netcat in listening mode and use the script's output to be sent to the client. The netcat program should restart for ever and ever ....

'''''Note:''' this bash script is painfully slow. We do not in any way recommend writing critical webservers using bash together with netcat. We use it here since we believe it gives you a good insight in how data are transferred.''

<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
<source lang="bash">
 while (true) ; do ./produce-json.sh 2 | nc -l -p 8181 ; done
</source>
</div></div>

==Access the JSON data using curl==
Use curl (in another terminal) to get the JSON data.

<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a suggested solution/answer.
<div class="mw-collapsible-content">
<source lang="bash">
$ curl localhost:8181
[
  {
    "name" : "Johanneshof Reinisch Pinot Noir",
    "price" : "143.0",
    "alcohol" : "13.0"
  },
  {
    "name" : "Dobogó Tokaji Furmint",
    "price" : "187.0",
    "alcohol" : "13.5"
  }
]
</source>
</div></div>

=Links=
==Solutions (source code)==
TODO: Verify that these are the correct files
* https://raw.githubusercontent.com/progund/web-misc/master/exercises/produce-json-static.sh
* https://github.com/progund/web-misc/blob/master/exercises/produce-json.sh
* https://github.com/progund/web-misc/raw/master/exercises/bolaget.db
* https://raw.githubusercontent.com/progund/web-misc/master/exercises/wserver.sh
<noinclude>

==Where to go next==
TODO: decide next page in the Programming_with_Java book

{{Nav links|prev=HTTP|TOC=Programming_with_Java#Chapters|next=ToBeDecided}}

</noinclude>