Difference between revisions of "ITIC:Network tools and commands - Exercises"

From Juneday education
Jump to: navigation, search
(Intro IT - Network tools exercises: Some new exercises)
(Connecting to other computers - HTTP (web))
(One intermediate revision by the same user not shown)
Line 95: Line 95:
 
* Use <code>whois</code> to get an idea who's behind the site <code>p.st</code> (feel free to visit it on the web)
 
* Use <code>whois</code> to get an idea who's behind the site <code>p.st</code> (feel free to visit it on the web)
 
** Hint: run dig and pipe it to grep for lines with "nameserver"
 
** Hint: run dig and pipe it to grep for lines with "nameserver"
 
+
* Use <code>whois</code> to get an idea who's behind the site <code>p.st</code> using a reverse lookup of the domain's IP
 
<div class="mw-collapsible mw-collapsed">
 
<div class="mw-collapsible mw-collapsed">
 
Expand using link to the right to see suggested solutions
 
Expand using link to the right to see suggested solutions
Line 177: Line 177:
 
$ # https://www.bahnhof.se/press/press-releases/2014/04/01/ny-soktjanst-hos-bahnhof-avslojar-svenskarnas-privatliv
 
$ # https://www.bahnhof.se/press/press-releases/2014/04/01/ny-soktjanst-hos-bahnhof-avslojar-svenskarnas-privatliv
  
 +
$ # for reverse lookup, first find out the IP of p.st
 +
$ host p.st
 +
p.st has address 94.254.0.29
 +
 +
$ # Reverse lookup, using -H (hide headers) and -B (show email etc)
 +
$ whois -H -B 94.254.0.29
 +
% This is the RIPE Database query service.
 +
% The objects are in RPSL format.
 +
%
 +
% The RIPE Database is subject to Terms and Conditions.
 +
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
 +
 +
% Information related to '94.254.0.0 - 94.254.0.255'
 +
 +
% Abuse contact for '94.254.0.0 - 94.254.0.255' is 'abuse@bahnhof.net'
 +
 +
inetnum:        94.254.0.0 - 94.254.0.255
 +
netname:        RID-0000020487
 +
descr:          RID-0000020487
 +
country:        SE
 +
admin-c:        BD856-RIPE
 +
tech-c:        BD856-RIPE
 +
status:        ASSIGNED PA
 +
notify:        ripe-kr@bahnhof.net
 +
mnt-by:        BAHNHOF-NCC
 +
created:        2012-09-03T11:15:53Z
 +
last-modified:  2012-09-03T11:15:53Z
 +
source:        RIPE
 +
 +
role:          Bahnhof DBM
 +
address:        Bahnhof AB
 +
address:        Isafjordsgatan 32B
 +
address:        164 40 Kista
 +
address:        Sweden
 +
e-mail:        ripe@bahnhof.se
 +
admin-c:        BD856-RIPE
 +
tech-c:        BD856-RIPE
 +
nic-hdl:        BD856-RIPE
 +
mnt-by:        BAHNHOF-NCC
 +
created:        2004-03-01T23:41:37Z
 +
last-modified:  2012-08-16T09:14:55Z
 +
source:        RIPE
 +
 +
% Information related to '94.254.0.0/18AS8473'
 +
 +
route:          94.254.0.0/18
 +
descr:          Bahnhof Internet, Sweden
 +
origin:        AS8473
 +
mnt-by:        BAHNHOF-NCC
 +
created:        2010-02-28T23:20:03Z
 +
last-modified:  2010-02-28T23:20:03Z
 +
source:        RIPE
 +
 +
% This query was served by the RIPE Database Query Service version 1.94.1 (WAGYU)
 +
 +
$ # Shorter output if you filter out interesting lines:
 +
$ whois -H -B 94.254.0.29 | egrep -i 'abuse|descr|notify|email|address|role|mnt-by'
 +
% Abuse contact for '94.254.0.0 - 94.254.0.255' is 'abuse@bahnhof.net'
 +
descr:          RID-0000020487
 +
notify:        ripe-kr@bahnhof.net
 +
mnt-by:        BAHNHOF-NCC
 +
role:          Bahnhof DBM
 +
address:        Bahnhof AB
 +
address:        Isafjordsgatan 32B
 +
address:        164 40 Kista
 +
address:        Sweden
 +
mnt-by:        BAHNHOF-NCC
 +
descr:          Bahnhof Internet, Sweden
 +
mnt-by:        BAHNHOF-NCC
 +
</source>
 +
</div>
 +
</div>
 +
==Information about your computer’s network==
 +
* Find out your computer's IP address(es)
 +
** Use <code>ifconfig</code> (on Windows: <code>ipconfig</code>)
 +
** Use <code>ip</code> (GNU/Linux)
 +
** Use <code>hostname</code> with the appropriate flag
 +
* Find out your computer's hostname
 +
** Use <code>hostname</code>
 +
** Use <code>echo</code> and the <code>$HOSTNAME</code> shell variable
 +
** Use <code>uname</code> with the appropriate flag, to print the "nodename"
 +
** Look at the prompt - after the <code>@</code> sign and before the <code>:</code>
 +
* Use <code>netstat</code> to find any active TCP-connection to a website (port 80 or 443) using numeric addresses
 +
** Then use <code>host</code> to find out the domain name of the IPs you are connected to
 +
** Then use <code>lsof</code> with the appropriate flags and arguments to find out what program(s) on your computer has the connection(s) to one or more IPs
 +
* Use a command (or application) to find out what WLAN SSID (network name) you are connected to:
 +
** On Ubuntu you can use <code>iwconfig</code> with an argument of your wireless network card, grep for SSID in the result
 +
** On macOS you can try something like <code>networksetup -getairportnetwork en1</code> or <code>/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -I</code>
 +
** On Windows you can try something like <code>Netsh WLAN show interfaces</code>
 +
* Follow the instructions [https://www.labnol.org/software/find-wi-fi-network-password/28949/ here] (www.labnol.org) to see if you can get your wifi password using only the command line (don't do this where anyone but yourself can see the result)
 +
* Use <code>ip link</code> (GNU/Linux) to find out the names of your network cards
 +
* Find out what DNS your computer is using
 +
** Ubuntu/GNU/Linux: Use <code>nmcli</code> with appropriate arguments and grep for DNS
 +
** macOS: Try <code>scutil --dns</code>
 +
** Windows: Try <code>ipconfig /all</code> and grep for DNS (e.g. <code>ipconfig /all | findstr "DNS"</code>)
 +
** Windows: Also try <code>nslookup</code> and look at the first lines of text
 +
* Find out your default gateway
 +
** GNU/Linux: Use <code>route</code>, <code>ip</code> or <code>netstat</code> with the appropriate flags
 +
** macOS: Try <code>route get default | grep gateway</code>
 +
** Windows: Try <code>ipconfig | findstr /i "Gateway"</code>
 +
 +
<div class="mw-collapsible mw-collapsed">
 +
Expand using link to the right to see suggested solutions
 +
<div class="mw-collapsible-content">
 +
<source lang="Bash">
 +
$ ifconfig
 +
lo        Link encap:Local Loopback 
 +
          inet addr:127.0.0.1  Mask:255.0.0.0
 +
          inet6 addr: ::1/128 Scope:Host
 +
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
 +
          RX packets:1042133 errors:0 dropped:0 overruns:0 frame:0
 +
          TX packets:1042133 errors:0 dropped:0 overruns:0 carrier:0
 +
          collisions:0 txqueuelen:1
 +
          RX bytes:110036554 (110.0 MB)  TX bytes:110036554 (110.0 MB)
 +
 +
wlp58s0  Link encap:Ethernet  HWaddr 9c:b6:d0:f2:e1:f9 
 +
          inet addr:10.0.116.35  Bcast:10.0.255.255  Mask:255.255.0.0
 +
          inet6 addr: 2001:6b0:2:2801:36bd:fb4f:417b:b503/64 Scope:Global
 +
          inet6 addr: fe80::de1f:acad:148:d9b4/64 Scope:Link
 +
          inet6 addr: 2001:6b0:2:2801:7c4a:f173:1eac:e775/64 Scope:Global
 +
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
 +
          RX packets:11436971 errors:0 dropped:0 overruns:0 frame:0
 +
          TX packets:7864743 errors:0 dropped:0 overruns:0 carrier:0
 +
          collisions:0 txqueuelen:1000
 +
          RX bytes:6755283977 (6.7 GB)  TX bytes:2080639936 (2.0 GB)
 +
 +
$ ip address
 +
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
 +
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 +
    inet 127.0.0.1/8 scope host lo
 +
      valid_lft forever preferred_lft forever
 +
    inet6 ::1/128 scope host
 +
      valid_lft forever preferred_lft forever
 +
2: wlp58s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
 +
    link/ether 9c:b6:d0:f2:e1:f9 brd ff:ff:ff:ff:ff:ff
 +
    inet 10.0.116.35/16 brd 10.0.255.255 scope global dynamic wlp58s0
 +
      valid_lft 43146sec preferred_lft 43146sec
 +
    inet6 2001:6b0:2:2801:7015:323b:6e9e:826c/64 scope global temporary dynamic
 +
      valid_lft 604748sec preferred_lft 85748sec
 +
    inet6 2001:6b0:2:2801:36bd:fb4f:417b:b503/64 scope global mngtmpaddr noprefixroute dynamic
 +
      valid_lft 2591997sec preferred_lft 604797sec
 +
    inet6 fe80::de1f:acad:148:d9b4/64 scope link
 +
      valid_lft forever preferred_lft forever
 +
 +
$ hostname -I
 +
10.0.116.35 192.168.56.1 2001:6b0:2:2801:36bd:fb4f:417b:b503
 +
 +
$ hostname
 +
newdelli
 +
 +
$ echo "$HOSTNAME"
 +
newdelli
 +
 +
$ uname -n
 +
newdelli
 +
 +
$ # Example checking http(s) connections:
 +
$ netstat -t4wn | egrep ':80|:443'
 +
tcp        0      0 10.0.116.35:41322      140.82.114.25:443      ESTABLISHED
 +
tcp        0      0 10.0.116.35:48982      52.20.247.134:443      ESTABLISHED
 +
tcp        0      0 10.0.116.35:50480      142.93.180.140:443      ESTABLISHED
 +
tcp        0      0 10.0.116.35:59784      130.241.39.132:443      ESTABLISHED
 +
 +
$ host 140.82.114.25
 +
25.114.82.140.in-addr.arpa domain name pointer lb-140-82-114-25-iad.github.com.
 +
$ host 52.20.247.134
 +
134.247.20.52.in-addr.arpa domain name pointer ec2-52-20-247-134.compute-1.amazonaws.com.
 +
$ host 142.93.180.140
 +
140.180.93.142.in-addr.arpa domain name pointer do-39.lastpass.com.
 +
$ host 130.241.39.132
 +
132.39.241.130.in-addr.arpa domain name pointer owa.gu.se.
 +
 +
$ # checking what application is connected to amazonaws on local port 48982
 +
$ lsof -i :48982
 +
COMMAND    PID  USER  FD  TYPE  DEVICE SIZE/OFF NODE NAME
 +
signal-de 15685 rikard  46u  IPv4 16548135      0t0  TCP 10.0.116.35:48982->ec2-52-20-247-134.compute-1.amazonaws.com:https (ESTABLISHED)
 +
 +
$ # It was Signal instant messenger
 +
 +
$ # Ubuntu: getting the SSID of your current wifi connection
 +
$ iwconfig wlp58s0 | grep SSID
 +
wlp58s0  IEEE 802.11abgn  ESSID:"eduroam" 
 +
 +
$ iwconfig wlp58s0 | grep SSID | cut -d '"' -f2
 +
eduroam
 +
 +
$ # Names of my network cards
 +
$ ip link
 +
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
 +
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 +
2: wlp58s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000
 +
    link/ether 9c:b6:d0:f2:e1:f9 brd ff:ff:ff:ff:ff:ff
 +
9: vboxnet0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
 +
    link/ether 0a:00:27:00:00:00 brd ff:ff:ff:ff:ff:ff
 +
 +
$ ip link | grep ^[0-9] | cut -d ':' -f2
 +
lo
 +
wlp58s0
 +
vboxnet0
 +
 +
$ # What DNS is configured?
 +
$ nmcli dev show | grep IP4.DNS
 +
IP4.DNS[1]:                            10.0.0.1
 +
IP4.DNS[2]:                            10.0.0.2
 +
 +
$ grep nameserver /etc/resolv.conf
 +
nameserver 127.0.1.1
 +
 +
$ grep -Po ".*nameserver '\K[^']+" /var/log/kern.log | tail -2
 +
10.0.0.1
 +
10.0.0.2
 +
 +
$ # Default gateway?
 +
$ route | grep default
 +
default        10.0.0.1        0.0.0.0        UG    600    0        0 wlp58s0
 +
 +
$ ip r | grep default
 +
default via 10.0.0.1 dev wlp58s0  proto static  metric 600
 +
 +
$ netstat -r | grep default
 +
default        10.0.0.1        0.0.0.0        UG        0 0          0 wlp58s0
 +
</source>
 +
</div>
 +
</div>
 +
==Connecting to other computers - HTTP (web)==
 +
* Get the HTTP headers from http://ait.gu.se
 +
** Look for a header revealing what web server they are using
 +
** Look for the "Via:" header (used by proxies for e.g. caching) - read about [https://varnish-cache.org/intro/index.html#intro varnish here] - and make a note of what varnish is and what its use is
 +
* Use <code>curl</code> to download the course curriculum PDF from <code>http://kursplaner.gu.se/pdf/kurs/sv/TIG015</code> and make <code>curl</code> save the file as <code>TIG015.pdf</code>
 +
* Download the same file using <code>wget</code>
 +
** How did wget handle the fact that you already had a file with the same name?
 +
 +
<div class="mw-collapsible mw-collapsed">
 +
Expand using link to the right to see suggested solutions
 +
<div class="mw-collapsible-content">
 +
<source lang="Bash">
 +
$ # HTTP headers from ait.gu.se
 +
$ HEAD http://ait.gu.se | egrep 'Via:|Server:'        # HEAD is an alias for lwp-request -m HEAD
 +
Via: 1.1 varnish (Varnish/5.2)
 +
Server: nginx/1.12.2
 +
 +
$ # Download using curl and wget
 +
$ curl -s 'http://kursplaner.gu.se/pdf/kurs/sv/TIG015' -o TIG015.pdf
 +
 +
$ wget -q 'http://kursplaner.gu.se/pdf/kurs/sv/TIG015' -O TIG015.pdf # will overwrite the file
  
 +
$ # The single quotes above were not necessary, but it's a good habit to ALWAYS quote URLs
 
</source>
 
</source>
 
</div>
 
</div>

Revision as of 14:09, 14 August 2019

Contents

Intro IT - Network tools exercises

We will add more exercises here, but for now, look below for exercises included from another page on this wiki.

Checking if a host is up

One of the more basic tools for networking is ping. Ping sends a small ICMP package to a destination IP number (a computer on the network), and reports response times and lost packages etc, which makes it suitable for diagnosing network problems. Often its use is very basic and trivial, like to check if a computer is "up" (running) and connected to the network.

There are several flags that you can use to control the way ping does its job. Your task is to figure out the flags for ping on your OS (ping is installed by default on most systems).

  • How do you tell ping not to send packages continuously (the default behavior), but to send a specified number of pings and then terminate? Write down the command line for sending 5 pings to 130.241.135.117 (the computer running this wiki - at least at the time of writing this, 2019-08-14)
  • How do you tell ping to wait a specified interval between each ping? Write down the command line for sending 5 pings to the same computer as above, but wait two seconds between each probe.
  • How do you tell ping to only report numeric addresses (not lookup host names) in its output? Use wiki.juneday.se instead of the IP number as the argument.
  • How do you tell ping to only print a summary when finished (not report each ping probe)? Write down the command line for sending 15 pings to the same computer as above, but only report the summary and not each ping.

Expand using link to the right to see suggested solutions

Note, suggested solutions are for GNU/Linux. On macOS or Windows, the syntax may vary

  • How do you tell ping not to send packages continuously (the default behavior), but to send a specified number of pings and then terminate? Write down the command line for sending 5 pings to 130.241.135.117 (the computer running this wiki - at least at the time of writing this, 2019-08-14)
$ ping -c 5 130.241.135.117
PING 130.241.135.117 (130.241.135.117) 56(84) bytes of data.
64 bytes from 130.241.135.117: icmp_seq=1 ttl=56 time=6.45 ms
64 bytes from 130.241.135.117: icmp_seq=2 ttl=56 time=4.21 ms
64 bytes from 130.241.135.117: icmp_seq=3 ttl=56 time=3.10 ms
64 bytes from 130.241.135.117: icmp_seq=4 ttl=56 time=3.73 ms
64 bytes from 130.241.135.117: icmp_seq=5 ttl=56 time=3.80 ms

--- 130.241.135.117 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 3.100/4.260/6.453/1.153 ms
  • How do you tell ping to wait a specified interval between each ping? Write down the command line for sending 5 pings to the same computer as above, but wait two seconds between each probe.
$ ping -c 5 -i 2 130.241.135.117
PING 130.241.135.117 (130.241.135.117) 56(84) bytes of data.
64 bytes from 130.241.135.117: icmp_seq=1 ttl=56 time=3.41 ms
64 bytes from 130.241.135.117: icmp_seq=2 ttl=56 time=17.2 ms
64 bytes from 130.241.135.117: icmp_seq=3 ttl=56 time=3.81 ms
64 bytes from 130.241.135.117: icmp_seq=4 ttl=56 time=7.31 ms
64 bytes from 130.241.135.117: icmp_seq=5 ttl=56 time=5.98 ms

--- 130.241.135.117 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 8005ms
rtt min/avg/max/mdev = 3.411/7.550/17.229/5.046 ms
  • How do you tell ping to only report numeric addresses (not lookup host names) in its output? Use wiki.juneday.se instead of the IP number as the argument.
$ ping -c 5 wiki.juneday.se # lookup hostnames (default behavior)
PING wiki.juneday.se (130.241.135.117) 56(84) bytes of data.
64 bytes from juneday.ait.gu.se (130.241.135.117): icmp_seq=1 ttl=56 time=9.87 ms
64 bytes from juneday.ait.gu.se (130.241.135.117): icmp_seq=2 ttl=56 time=5.91 ms
64 bytes from juneday.ait.gu.se (130.241.135.117): icmp_seq=3 ttl=56 time=45.3 ms
64 bytes from juneday.ait.gu.se (130.241.135.117): icmp_seq=4 ttl=56 time=136 ms
64 bytes from juneday.ait.gu.se (130.241.135.117): icmp_seq=5 ttl=56 time=159 ms

--- wiki.juneday.se ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 5.911/71.414/159.573/64.409 ms

$ ping -c 5 -n wiki.juneday.se # only numeric addresses
PING wiki.juneday.se (130.241.135.117) 56(84) bytes of data.
64 bytes from 130.241.135.117: icmp_seq=1 ttl=56 time=2.96 ms
64 bytes from 130.241.135.117: icmp_seq=2 ttl=56 time=3.48 ms
64 bytes from 130.241.135.117: icmp_seq=3 ttl=56 time=5.90 ms
64 bytes from 130.241.135.117: icmp_seq=4 ttl=56 time=8.10 ms
64 bytes from 130.241.135.117: icmp_seq=5 ttl=56 time=4.86 ms

--- wiki.juneday.se ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 2.969/5.064/8.103/1.839 ms
  • How do you tell ping to only print a summary when finished (not report each ping probe)? Write down the command line for sending 15 pings to the same computer as above, but only report the summary and not each ping
$ ping -c 15 -q wiki.juneday.se
PING wiki.juneday.se (130.241.135.117) 56(84) bytes of data.

--- wiki.juneday.se ping statistics ---
15 packets transmitted, 15 received, 0% packet loss, time 14018ms
rtt min/avg/max/mdev = 3.071/5.238/9.514/1.968 ms

DNS tools - looking up names and numbers

  • Use host (if you have it on your OS, you may need to install it) to find out the IPv4 IP number(s) of ftp.sunet.se
  • Use host to look up the names from the IP number(s) you got from the previous task (the numbers may not translate back to ftp.sunet.se - the machine may have more than one name!)
  • Use host to find out the DNS used for
    • juneday.se
    • umu.se
  • Use nslookup to find out the IP number of juneday.se
    • Using the default DNS on your system (no extra argument)
    • Using the DNS for juneday.se
  • Use dig (if you have it, may need to install it) to find out the mail server(s) of gu.se
    • Use the flag/argument for making the output short
  • Use host and nslookup to find the mail server(s) for chalmers.se
  • Use whois to get an idea who's behind the site p.st (feel free to visit it on the web)
    • Hint: run dig and pipe it to grep for lines with "nameserver"
  • Use whois to get an idea who's behind the site p.st using a reverse lookup of the domain's IP

Expand using link to the right to see suggested solutions

$ # names and numbers:
$ host ftp.sunet.se
ftp.sunet.se is an alias for sunet.ftp.acc.umu.se.
sunet.ftp.acc.umu.se has address 194.71.11.173
sunet.ftp.acc.umu.se has address 194.71.11.165
sunet.ftp.acc.umu.se has IPv6 address 2001:6b0:19::173
sunet.ftp.acc.umu.se has IPv6 address 2001:6b0:19::165
sunet.ftp.acc.umu.se mail is handled by 0 mail.acc.umu.se.

$ host 194.71.11.173
173.11.71.194.in-addr.arpa domain name pointer napoleon.ftp.acc.umu.se.
rikard@newdelli:~$ host 194.71.11.165
165.11.71.194.in-addr.arpa domain name pointer hammurabi.ftp.acc.umu.se.

$ host -t CNAME ftp.sunet.se
ftp.sunet.se is an alias for sunet.ftp.acc.umu.se

$ # Finding DNS for domains:
$ host -t NS juneday.se
juneday.se name server ns1.loopia.se.
juneday.se name server ns2.loopia.se.

$ host -t NS umu.se
umu.se name server ns2.umu.se.
umu.se name server ns1.umu.se.
umu.se name server sunic.sunet.se.
umu.se name server dns.norrnod.se.

$ # Using nslookup and the default DNS
$ nslookup juneday.se
Server:		127.0.1.1
Address:	127.0.1.1#53

Non-authoritative answer:
Name:	juneday.se
Address: 89.18.105.40

$ # Using nslookup and ns1.loopia.se:
$ nslookup juneday.se ns1.loopia.se
Server:		ns1.loopia.se
Address:	2a02:250:ffff::20#53

Name:	juneday.se
Address: 89.18.105.40

$ # Using dig to find out mail server(s) for gu.se (short report)
$ dig gu.se mx +short
10 v-mailfilter03.sunet.se.
11 e-mailfilter03.sunet.se.
11 e-mailfilter04.sunet.se.

$ # Using host to find out mail server(s) for gu.se
$ host -t mx chalmers.se
chalmers.se mail is handled by 10 e-mailfilter04.sunet.se.
chalmers.se mail is handled by 10 e-mailfilter03.sunet.se.
chalmers.se mail is handled by 10 v-mailfilter03.sunet.se.

$ # Using nslookup to find out mail server(s) for gu.se
$ nslookup -query=mx chalmers.se
Server:		127.0.1.1
Address:	127.0.1.1#53

Non-authoritative answer:
chalmers.se	mail exchanger = 10 e-mailfilter03.sunet.se.
chalmers.se	mail exchanger = 10 v-mailfilter03.sunet.se.
chalmers.se	mail exchanger = 10 e-mailfilter04.sunet.se.

Authoritative answers can be found from:

$ # Who could be behind p.st?
$ whois p.st | grep -i nameserver
nameserver: ns.bahnhof.net
nameserver: ns2.bahnhof.net
$ # Aha, it's Bahnhof
$ # This site was an april fool's joke: 
$ # https://www.bahnhof.se/press/press-releases/2014/04/01/ny-soktjanst-hos-bahnhof-avslojar-svenskarnas-privatliv

$ # for reverse lookup, first find out the IP of p.st
$ host p.st
p.st has address 94.254.0.29

$ # Reverse lookup, using -H (hide headers) and -B (show email etc)
$ whois -H -B 94.254.0.29
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Information related to '94.254.0.0 - 94.254.0.255'

% Abuse contact for '94.254.0.0 - 94.254.0.255' is 'abuse@bahnhof.net'

inetnum:        94.254.0.0 - 94.254.0.255
netname:        RID-0000020487
descr:          RID-0000020487
country:        SE
admin-c:        BD856-RIPE
tech-c:         BD856-RIPE
status:         ASSIGNED PA
notify:         ripe-kr@bahnhof.net
mnt-by:         BAHNHOF-NCC
created:        2012-09-03T11:15:53Z
last-modified:  2012-09-03T11:15:53Z
source:         RIPE

role:           Bahnhof DBM
address:        Bahnhof AB
address:        Isafjordsgatan 32B
address:        164 40 Kista
address:        Sweden
e-mail:         ripe@bahnhof.se
admin-c:        BD856-RIPE
tech-c:         BD856-RIPE
nic-hdl:        BD856-RIPE
mnt-by:         BAHNHOF-NCC
created:        2004-03-01T23:41:37Z
last-modified:  2012-08-16T09:14:55Z
source:         RIPE

% Information related to '94.254.0.0/18AS8473'

route:          94.254.0.0/18
descr:          Bahnhof Internet, Sweden
origin:         AS8473
mnt-by:         BAHNHOF-NCC
created:        2010-02-28T23:20:03Z
last-modified:  2010-02-28T23:20:03Z
source:         RIPE

% This query was served by the RIPE Database Query Service version 1.94.1 (WAGYU)

$ # Shorter output if you filter out interesting lines:
$ whois -H -B 94.254.0.29 | egrep -i 'abuse|descr|notify|email|address|role|mnt-by'
% Abuse contact for '94.254.0.0 - 94.254.0.255' is 'abuse@bahnhof.net'
descr:          RID-0000020487
notify:         ripe-kr@bahnhof.net
mnt-by:         BAHNHOF-NCC
role:           Bahnhof DBM
address:        Bahnhof AB
address:        Isafjordsgatan 32B
address:        164 40 Kista
address:        Sweden
mnt-by:         BAHNHOF-NCC
descr:          Bahnhof Internet, Sweden
mnt-by:         BAHNHOF-NCC

Information about your computer’s network

  • Find out your computer's IP address(es)
    • Use ifconfig (on Windows: ipconfig)
    • Use ip (GNU/Linux)
    • Use hostname with the appropriate flag
  • Find out your computer's hostname
    • Use hostname
    • Use echo and the $HOSTNAME shell variable
    • Use uname with the appropriate flag, to print the "nodename"
    • Look at the prompt - after the @ sign and before the :
  • Use netstat to find any active TCP-connection to a website (port 80 or 443) using numeric addresses
    • Then use host to find out the domain name of the IPs you are connected to
    • Then use lsof with the appropriate flags and arguments to find out what program(s) on your computer has the connection(s) to one or more IPs
  • Use a command (or application) to find out what WLAN SSID (network name) you are connected to:
    • On Ubuntu you can use iwconfig with an argument of your wireless network card, grep for SSID in the result
    • On macOS you can try something like networksetup -getairportnetwork en1 or /System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -I
    • On Windows you can try something like Netsh WLAN show interfaces
  • Follow the instructions here (www.labnol.org) to see if you can get your wifi password using only the command line (don't do this where anyone but yourself can see the result)
  • Use ip link (GNU/Linux) to find out the names of your network cards
  • Find out what DNS your computer is using
    • Ubuntu/GNU/Linux: Use nmcli with appropriate arguments and grep for DNS
    • macOS: Try scutil --dns
    • Windows: Try ipconfig /all and grep for DNS (e.g. ipconfig /all | findstr "DNS")
    • Windows: Also try nslookup and look at the first lines of text
  • Find out your default gateway
    • GNU/Linux: Use route, ip or netstat with the appropriate flags
    • macOS: Try route get default | grep gateway
    • Windows: Try ipconfig | findstr /i "Gateway"

Expand using link to the right to see suggested solutions

$ ifconfig 
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:1042133 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1042133 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:110036554 (110.0 MB)  TX bytes:110036554 (110.0 MB)

wlp58s0   Link encap:Ethernet  HWaddr 9c:b6:d0:f2:e1:f9  
          inet addr:10.0.116.35  Bcast:10.0.255.255  Mask:255.255.0.0
          inet6 addr: 2001:6b0:2:2801:36bd:fb4f:417b:b503/64 Scope:Global
          inet6 addr: fe80::de1f:acad:148:d9b4/64 Scope:Link
          inet6 addr: 2001:6b0:2:2801:7c4a:f173:1eac:e775/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11436971 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7864743 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:6755283977 (6.7 GB)  TX bytes:2080639936 (2.0 GB)

$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: wlp58s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 9c:b6:d0:f2:e1:f9 brd ff:ff:ff:ff:ff:ff
    inet 10.0.116.35/16 brd 10.0.255.255 scope global dynamic wlp58s0
       valid_lft 43146sec preferred_lft 43146sec
    inet6 2001:6b0:2:2801:7015:323b:6e9e:826c/64 scope global temporary dynamic 
       valid_lft 604748sec preferred_lft 85748sec
    inet6 2001:6b0:2:2801:36bd:fb4f:417b:b503/64 scope global mngtmpaddr noprefixroute dynamic 
       valid_lft 2591997sec preferred_lft 604797sec
    inet6 fe80::de1f:acad:148:d9b4/64 scope link 
       valid_lft forever preferred_lft forever

$ hostname -I
10.0.116.35 192.168.56.1 2001:6b0:2:2801:36bd:fb4f:417b:b503

$ hostname
newdelli

$ echo "$HOSTNAME"
newdelli

$ uname -n
newdelli

$ # Example checking http(s) connections:
$ netstat -t4wn | egrep ':80|:443'
tcp        0      0 10.0.116.35:41322       140.82.114.25:443       ESTABLISHED
tcp        0      0 10.0.116.35:48982       52.20.247.134:443       ESTABLISHED
tcp        0      0 10.0.116.35:50480       142.93.180.140:443      ESTABLISHED
tcp        0      0 10.0.116.35:59784       130.241.39.132:443      ESTABLISHED

$ host 140.82.114.25
25.114.82.140.in-addr.arpa domain name pointer lb-140-82-114-25-iad.github.com.
$ host 52.20.247.134
134.247.20.52.in-addr.arpa domain name pointer ec2-52-20-247-134.compute-1.amazonaws.com.
$ host 142.93.180.140
140.180.93.142.in-addr.arpa domain name pointer do-39.lastpass.com.
$ host 130.241.39.132
132.39.241.130.in-addr.arpa domain name pointer owa.gu.se.

$ # checking what application is connected to amazonaws on local port 48982
$ lsof -i :48982
COMMAND     PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
signal-de 15685 rikard   46u  IPv4 16548135      0t0  TCP 10.0.116.35:48982->ec2-52-20-247-134.compute-1.amazonaws.com:https (ESTABLISHED)

$ # It was Signal instant messenger

$ # Ubuntu: getting the SSID of your current wifi connection
$ iwconfig wlp58s0 | grep SSID
wlp58s0   IEEE 802.11abgn  ESSID:"eduroam"  

$ iwconfig wlp58s0 | grep SSID | cut -d '"' -f2
eduroam

$ # Names of my network cards
$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: wlp58s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000
    link/ether 9c:b6:d0:f2:e1:f9 brd ff:ff:ff:ff:ff:ff
9: vboxnet0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether 0a:00:27:00:00:00 brd ff:ff:ff:ff:ff:ff

$ ip link | grep ^[0-9] | cut -d ':' -f2
 lo
 wlp58s0
 vboxnet0

$ # What DNS is configured?
$ nmcli dev show | grep IP4.DNS
IP4.DNS[1]:                             10.0.0.1
IP4.DNS[2]:                             10.0.0.2

$ grep nameserver /etc/resolv.conf
nameserver 127.0.1.1

$ grep -Po ".*nameserver '\K[^']+" /var/log/kern.log | tail -2
10.0.0.1
10.0.0.2

$ # Default gateway?
$ route | grep default
default         10.0.0.1        0.0.0.0         UG    600    0        0 wlp58s0

$ ip r | grep default
default via 10.0.0.1 dev wlp58s0  proto static  metric 600

$ netstat -r | grep default
default         10.0.0.1        0.0.0.0         UG        0 0          0 wlp58s0

Connecting to other computers - HTTP (web)

  • Get the HTTP headers from http://ait.gu.se
    • Look for a header revealing what web server they are using
    • Look for the "Via:" header (used by proxies for e.g. caching) - read about varnish here - and make a note of what varnish is and what its use is
  • Use curl to download the course curriculum PDF from http://kursplaner.gu.se/pdf/kurs/sv/TIG015 and make curl save the file as TIG015.pdf
  • Download the same file using wget
    • How did wget handle the fact that you already had a file with the same name?

Expand using link to the right to see suggested solutions

$ # HTTP headers from ait.gu.se
$ HEAD http://ait.gu.se | egrep 'Via:|Server:'        # HEAD is an alias for lwp-request -m HEAD
Via: 1.1 varnish (Varnish/5.2)
Server: nginx/1.12.2

$ # Download using curl and wget
$ curl -s 'http://kursplaner.gu.se/pdf/kurs/sv/TIG015' -o TIG015.pdf

$ wget -q 'http://kursplaner.gu.se/pdf/kurs/sv/TIG015' -O TIG015.pdf # will overwrite the file

$ # The single quotes above were not necessary, but it's a good habit to ALWAYS quote URLs

Links (Intro IT and computing)

Where to go next

Next page is ITIC:Using_a_text_editor.

« PreviousBook TOCNext »

For now, inclusion of MoreBash:Exercises_-_Network_Tools below:

Work in progress

This chapter is a work in progress. Remove this section when the page is production-ready.

Note to hesa: Not sure this should be about Scripts primarilly - your slides talk about the tools, not using them in scripts...

Totally agree. Let's rename the page (skip the scripts part)

Prerequisite knowledge

These exercises assume that you have basic knowledge of Bash, computers and networks. We have books covering some of these basics, if you need to refresh them.

Make sure you have seen the lectures from the above chapters and made all the exercises, unless you have equivalent prior knownledge.

Introduction

The purpose of these exercises is to get you familiar with the some network commands and tools available in bash. See the previous chapter PDFs and video lectures for an introduction to network tools.

Some of the tools are: General networking:

  • telnet
  • netcat (nc)

Connectivity between servers:

  • ssh/scp
  • rsync

Diagnostics

  • ping
  • traceroute
  • netstat
  • nmap
  • tcpdump
  • wireshark
  • iftop

HTTP and Web stuff

  • lwprequest
  • wget
  • curl

DNS stuff

  • whois
  • host
  • dig
  • nslookup

Exercises on IP and domains

Look up the IP of www.gnu.org using dig

Use dig to find the IP addresss of the server www.gnu.org. What dns was used?

Expand using link to the right to see a hint.

$ dig www.gnu.org

; <<>> DiG 9.10.4-P5-RedHat-9.10.4-4.P5.fc25 <<>> www.gnu.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20673
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.gnu.org.			IN	A

;; ANSWER SECTION:
www.gnu.org.		93	IN	CNAME	wildebeest.gnu.org.
wildebeest.gnu.org.	93	IN	A	208.118.235.148

;; Query time: 10 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Feb 03 11:04:52 CET 2017
;; MSG SIZE  rcvd: 81

From this we can see that the IP (via wildebeest.gnu.org) is 208.118.235.148.

The DNS used was 8.8.8.8.

Look up the IP of www.gnu.org using host

Use host to find the IP addresss of the server www.gnu.org. What dns was used?

Expand using link to the right to see a hint.

$ host www.gnu.org
www.gnu.org is an alias for wildebeest.gnu.org.
wildebeest.gnu.org has address 208.118.235.148
wildebeest.gnu.org has IPv6 address 2001:4830:134:3::a

From this we can see that the IP (via wildebeest.gnu.org) is 208.118.235.148.

The DNS used was 8.8.8.8.

Look up the IP of www.gnu.org using nslookup

Use nslookup to find the IP addresss of the server www.gnu.org. What dns was used?

Expand using link to the right to see a hint.

$ nslookup www.gnu.org
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
www.gnu.org	canonical name = wildebeest.gnu.org.
Name:	wildebeest.gnu.org
Address: 208.118.235.148

From this we can see that the IP (via wildebeest.gnu.org) is 208.118.235.148.

The DNS used was 8.8.8.8.

Look up the IP using another DNS using dig

Use dig to find the IP addresss of the server www.gnu.org. Use the DNS 8.8.4.4 instead.

Expand using link to the right to see a hint.

$ dig @8.8.4.4 www.gnu.org

; <<>> DiG 9.10.4-P5-RedHat-9.10.4-4.P5.fc25 <<>> @8.8.4.4 www.gnu.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28503
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.gnu.org.			IN	A

;; ANSWER SECTION:
www.gnu.org.		13	IN	CNAME	wildebeest.gnu.org.
wildebeest.gnu.org.	13	IN	A	208.118.235.148

;; Query time: 9 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Fri Feb 03 11:11:02 CET 2017
;; MSG SIZE  rcvd: 81

From this we can see that the IP (via wildebeest.gnu.org) is 208.118.235.148. The same as when using 8.8.8.8.

Silently look up the IP using another DNS using dig

Use dig to find the IP addresss of the server www.gnu.org. Use the DNS 8.8.4.4 instead. Make dig print out less information (hint: search for short in the manual).

Expand using link to the right to see a hint.

$ dig @8.8.4.4 +short www.gnu.org
wildebeest.gnu.org.
208.118.235.148

From this we can still see that the IP (via wildebeest.gnu.org) is 208.118.235.148. The amount of text is less (shorter).

Look up the domain of the IP address 208.118.235.148

Use dig to find the domain of IP addresss 208.118.235.148. Make dig print out less information (hint: search for "reverse lookups" in the manual).

Expand using link to the right to see a hint.

$ dig +short -x  208.118.235.148
wildebeest.gnu.org.

The domain of the reversed looked up IP address is 208.118.235.148.

Look up the domain of the IP address of the domain www.gnu.org

Use dig to

  1. find the IP addresss www.gnu.org
  2. and reverse lookup that IP

To do this you need to use the output of the first dig execution as argument (not stdin) to the next call to dig.

If we do this manually we type:

$ dig +short www.gnu.org
wildebeest.gnu.org.
208.118.235.148

and copy/paste the IP (208.118.235.148) to dig;

$ dig +short -x 208.118.235.148
wildebeest.gnu.org.

Your task now is to automate this in one command.

Expand using link to the right to see a hint.

$ dig +short -x $(dig +short www.gnu.org | egrep -e "^[0-9\.]{4}") wildebeest.gnu.org

To get an idea of how we came up with this odd command line we will guide you through our stupid thoughts.

We start with finding the IP of www.gnu.org

$ dig +short www.gnu.org
wildebeest.gnu.org.
208.118.235.148

This is too much information so we use egrep to keep only the lines containing an IP address. An IP address is four numbers separated by a dot. So we create a regular expression matching IP addresses: ^[0-9\.]{4}. This expression says:

  • the line shall start with (this is what ^ means)
  • followed by four occurances of the digits 0 to 9 (0-9) or a dot (\.)

Using this we get:

$ dig +short www.gnu.org | egrep -e "^[0-9\.]{4}"
208.118.235.148

Nice! This is what we want as argument (again, not stdin so we shall not pipe) to a new dig execution. We could store the output of the command above in a variable and use the variable: Note: this regular expression is not perfect since it will match non IP addresses but it will do fine for this exercise. A better regular expression would be ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$.

$ IP=$(dig +short www.gnu.org | egrep -e "^[0-9\.]{4}")
$ dig +short -x $IP
wildebeest.gnu.org.

The above is hopefully ok. But we think it's a bit cumbersome so we suggest skipping the variable:

$ dig +short -x $(dig +short www.gnu.org | egrep -e "^[0-9\.]{4}")
wildebeest.gnu.org.

... and for the keen student we would like to point out that we actually could use a pipe:

$ dig +short www.gnu.org | egrep -e "^[0-9\.]{4}" | xargs dig +short -x 
wildebeest.gnu.org.

In the last command line we pipe the output of dig +short www.gnu.org | egrep -e "^[0-9\.]{4}" to xargs. xargs reads from stdin and starts dig +short -x using the text (read from stdin) as argument to dig. So the second dig is started (by xargs) like this dig +short -x 208.118.235.148

Write a small script that does the above

The script shall:

  • take the domain as an argument
  • exit with 1 and a printout to stderr if no argument was given
  • exit with the status code of dig

Expand using link to the right to see a hint.

Create a file called rcheck_domain.sh

#!/bin/bash

DOMAIN=$1

if [ "$DOMAIN" != "" ]
then
       dig +short -x $(dig +short $DOMAIN | grep "^[0-9]*\.") ;
       exit $?
else
    echo "Missing domain" 1>&2
    exit 1
fi

You can find complete source code to the suggested solutions below in the . directory in this zip file or in the git repository.

Write a bash function that does the above

The function shall:

  • take the domain as an argument
  • exit with 1 and a printout to stderr if no argument was given
  • exit with the status code of dig

Expand using link to the right to see a hint.

$ rcheck_domain() { DOMAIN=$1; if [ "$DOMAIN" = "" ] ; then  echo "Missing domain" 1>&2; exit 1 ; fi ; dig +short -x $(dig +short $1 | grep "^[0-9]*\.") ; }

You can now use the function above like this:

$ rcheck_domain www.gnu.org
wildebeest.gnu.org.
$ rcheck_domain www.sunet.se
webc.sunet.se.
$ rcheck_domain www.funet.fi
www.funet.fi.

If you want to be able to use it in the future you put the function in your ~/.bashrc file.

Exercises on checking network

Use ping to check if 8.8.8.8 is up

Expand using link to the right to see a hint.

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=44 time=23.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=44 time=25.3 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=44 time=23.5 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=44 time=24.4 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=44 time=24.7 ms
64 bytes from 8.8.8.8: icmp_seq=6 ttl=44 time=23.4 ms
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5009ms
rtt min/avg/max/mdev = 23.485/24.202/25.341/0.706 ms

Press control-c to interrupt the program

Ping the host again, but at most 3 times

Expand using link to the right to see a hint.

$ ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=44 time=25.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=44 time=23.8 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=44 time=24.7 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 23.851/24.845/25.935/0.872 ms

What was the return value of the above?

If you're on a proper network the host 8.8.8.8 shall be "pingable" so you should get a 0 back. With proper network we mean a network that does not block ping.

What can such return value be used for?

Expand using link to the right to see a hint.

$ ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=44 time=25.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=44 time=23.8 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=44 time=24.7 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 23.851/24.845/25.935/0.872 ms
$ echo $?
0

The return value can be used as a check to see if a host is up and take action depending on that.

Exercises on web pages

Use curl to get the html page of www.gnu.org

Expand using link to the right to see a hint.

$ curl www.gnu.org
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE html PUBLI
C "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">

<head>
<!-- start of server/head-include-1.html -->
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<link rev="made" href="mailto:webmasters@gnu.org" />
<link rel="icon" type="image/png" href="/graphics/gnu-head-mini.png" />
<meta name="ICBM" content="42.355469,-71.058627" />
<meta name="DC.title" content="gnu.org" />
<link rel="stylesheet" href="/combo.css" media="screen" />
<link rel="stylesheet" href="/mini.css" media="handheld" />
<link rel="stylesheet" href="/layout.min.css" media="screen" />
<link rel="stylesheet" href="/print.min.css" media="print" />
<!-- end of server/head-include-1.html -->

<!-- end of server/header.html -->

<!-- Parent-Version: 1.79 -->

<title>The GNU Operating System and the Free Software Movement</title>

<meta http-equiv="Keywords" content="GNU, FSF, Free Software Foundation, Linux, Emacs, GCC, Unix, 
Free Software, Libre Software, Operating System, GNU Kernel, GNU Hurd" />
<meta http-equiv="Description" content="Since 1983, developing the free Unix style operating syste
m GNU, so that computer users can have the freedom to share and improve the software they use." />
<link rel="alternate" title="Planet GNU" href="http://planet.gnu.org/rss20.xml" type="application/
rss+xml" />
....


Use curl to get the html page of www.gnu.org and store it on file

Use curl to get the html page of www.gnu.org and store it on a file called www-gnu-org.html. What is the return value?

Expand using link to the right to see a hint.

$ curl www.gnu.org  -o www-gnu-org.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 26560    0 26560    0     0  64738      0 --:--:-- --:--:-- --:--:-- 64780

On success curl returns 0.


Use curl to get a non-existing html page

Use curl to try to get www.sunet.se/this-page-does-not-exists.html. What exit code is returned?

Expand using link to the right to see a hint.

$ curl www.sunet.se/this-page-does-not-exists.html
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.sunet.se/this-page-does-not-exists.html">here</a>.</p>
<hr>
<address>Apache/2.4.7 (Ubuntu) Server at www.sunet.se Port 80</address>
</body></html>
$ echo $?
0

curl succeeds and returns 0.


Use curl to get a html page from a non-existing server

Use curl to try to get http://madeupurl.thatreallydoesn.otexist.com/this-page-does-not-exists.html. What exit code is returned?

Expand using link to the right to see a hint.

$ curl http://madeupurl.thatreallydoesn.otexist.com/this-page-does-not-exists.html
curl: (6) Could not resolve host: madeupurl.thatreallydoesn.otexist.com
$ echo $?
6

curl fails looking up the server and and returns 6 as exit code.

Use w3m to get the text of an html page

Use w3m to get the web page of www.nytimes.com.

Expand using link to the right to see a hint.

Check out the -dump option to w3m.

Expand using link to the right to see a suggested solution.

$ w3m -dump www.nytimes.com

Count the number of times London is mentioned

Use w3m to get the web page of www.nytimes.com and check how many times London is mentioned.

Expand using link to the right to see a hint.

w3m outputs the web content (with the html removed). So you should be able to pipe the output to grep. You can use the -c to grep to count the occuranes.

Expand using link to the right to see a suggested solution.

$ w3m -dump www.nytimes.com | grep -c London
Received cookie: nyt-a=3405f3e1be1967838ead183860360c4934bd00a943d8ea9632efc3050e5b22ab
Received cookie: nyt-a=3405f3e1be1967838ead183860360c4934bd00a943d8ea9632efc3050e5b22ab
0

Write a bash function that searches a web page for a search string

The function shall take two arguments:

  1. the site
  2. the search expression

If any of the arguments is missing the function shall output an error message to stderr.

Expand using link to the right to see a suggested solution.

$ wgrep() { SITE=$1; REG_EXP=$2; if [ "$REG_EXP" = "" ] ; then echo "Missing argument(s)" 1>&1 ; else  w3m -dump $SITE | grep -c $REG_EXP ; fi ; }

You can now use the function to check the number occurances of words on a web site. Examples below:

$ wgrep www.nytimes.com Europe
Received cookie: nyt-a=3405f3e1be1967838ead183860360c4934bd00a943d8ea9632efc3050e5b22ab
Received cookie: nyt-a=3405f3e1be1967838ead183860360c4934bd00a943d8ea9632efc3050e5b22ab
0

Remove the printouts to stderr from w3m in the function above

Expand using link to the right to see a suggested solution.

$ wgrep() { SITE=$1; REG_EXP=$2; if [ "$REG_EXP" = "" ] ; then echo "Missing argument(s)" 1>&1 ; else  w3m -dump $SITE 2>/dev/null | grep -c $REG_EXP ; fi ; }

You can now use the function to check the number occurances of words on a web site. Examples below:

$ $ wgrep www.nytimes.com Europe
0
$ wgrep www.nytimes.com Super
5
$ wgrep www.dailymirror.com London
11

Write a function that uses the function above to check several sites

Write a function that checks, using the function above, the following web sites for a search string:

  • www.nytimes.com
  • www.mirror.co.uk
  • www.daily-sun.com
  • www.washingtonpost.com
  • www.chicagotribune.com
  • www.theguardian.com/us
  • timesofindia.indiatimes.com
  • www.dailymail.co.uk

The search string shall be given as argument to the function.

Expand using link to the right to see a suggested solution.

To start of with we can write a function the can echo the sites.

$ dwgrep() { for site in www.nytimes.com www.mirror.co.uk www.daily-sun.com www.washingtonpost.com www.chicagotribune.com www.theguardian.com/us timesofindia.indiatimes.com www.dailymail.co.uk ; do echo $site; done ; } 
[hesa@bartok bash-network-tools]$ dwgrep 
www.nytimes.com
www.mirror.co.uk
www.daily-sun.com
www.washingtonpost.com
www.chicagotribune.com
www.theguardian.com/us
timesofindia.indiatimes.com
www.dailymail.co.uk

Ok, it seems to work. Let's check the search word (given as argument):

$ dwgrep() { REG_EXP=$1 ; if [ "$REG_EXP" = "" ] ; then echo "Missing argument(s)" 1>&1 ; else for site in www.nytimes.com www.mirror.co.uk www.daily-sun.com www.washingtonpost.com www.chicagotribune.com www.theguardian.com/us timesofindia.indiatimes.com www.dailymail.co.uk ; do echo $site $REG_EXP ; done ; fi ; } 
$ dwgrep 
Missing argument(s)
$ dwgrep London
www.nytimes.com London
www.mirror.co.uk London
www.daily-sun.com London
www.washingtonpost.com London
www.chicagotribune.com London
www.theguardian.com/us London
timesofindia.indiatimes.com London
www.dailymail.co.uk London

Ok, let's invoke the function we wrote earlier

$ dwgrep() { REG_EXP=$1 ; if [ "$REG_EXP" = "" ] ; then echo "Missing argument(s)" 1>&1 ; else for site in www.nytimes.com www.mirror.co.uk www.daily-sun.com www.washingtonpost.com www.chicagotribune.com www.theguardian.com/us timesofindia.indiatimes.com www.dailymail.co.uk ; do echo -n "$site: " ; wgrep $site $REG_EXP ; done ; fi ; } 
$ dwgrep London
www.nytimes.com: 0
www.mirror.co.uk: 11
www.daily-sun.com: 0
www.washingtonpost.com: 0
www.chicagotribune.com: 1
www.theguardian.com/us: 0
timesofindia.indiatimes.com: 0
www.dailymail.co.uk: 19

Not a lot of code and lots of work done :)

Netcat and telnet to the rescue

Use telnet to get a webpage

Use telnet to open a connection to www.apache.org (default for web and http is port 80).


Expand using link to the right to see a suggested solution.

$ telnet www.apache.org 80 
Trying 88.198.26.2...
Connected to www.apache.org.
Escape character is '^]'


HTTP/1.1 200 OK Date: Mon, 06 Feb 2017 08:42:55 GMT Server: Apache/2.4.7 (Ubuntu) Last-Modified: Mon, 06 Feb 2017 08:10:28 GMT ETag: "d572-547d82a205060" Accept-Ranges: bytes Content-Length: 54642 Vary: Accept-Encoding Cache-Control: max-age=3600 Expires: Mon, 06 Feb 2017 09:42:55 GMT Connection: close Content-Type: text/html

<!DOCTYPE html> <html lang="en"> <head>

 <meta charset="utf-8">
 <meta http-equiv="X-UA-Compatible" content="IE=edge">
 <meta name="viewport" content="width=device-width, initial-scale=1">
 <meta name="description" content="Home page of The Apache Software Foundation">
 

......

Use netcat to get a webpage

Use netcat to open a connection to www.apache.org (default for web and http is port 80).


Expand using link to the right to see a suggested solution.

$ nc www.apache.org 80 
GET / HTTP/1.0

HTTP/1.1 200 OK
Date: Mon, 06 Feb 2017 08:39:15 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Mon, 06 Feb 2017 08:10:28 GMT
ETag: "d572-547d82a205060"
Accept-Ranges: bytes
Content-Length: 54642
Vary: Accept-Encoding
Cache-Control: max-age=3600
Expires: Mon, 06 Feb 2017 09:39:15 GMT
Connection: close
Content-Type: text/html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <meta name="description" content="Home page of The Apache Software Foundation">
  
........

Use telnet to send an email

We will not do this an exercise where we expect you to learn and remember. Think of this more as a way to get an understanding of a protocol and how a client talks to a server.

Connect to an email server:

$ telnet mail.youremailprovider.com 25
Trying 89.18.105.40...
Connected to mail.youremailprovider.com.
Escape character is '^]'.


Say hi:

EHLO myown
250-mail.youremailprovider.com Hello myown [81.170.163.11]
250-SIZE 52428800
250-8BITMIME
250-PIPELINING
250-AUTH CRAM-MD5
250-STARTTLS
250 HELP


Specify from who the email is:

MAIL FROM: myemail@myemailserver.com
250 OK

Specify to who the email is:

RCPT TO: hesa@youremailprovider.com
250 Accepted

Write som content:

DATA
354 Enter message, ending with "." on a line by itself
Hi there
.
250 OK id=1cagHb-00020O-Ab

The mail server mail.youremailprovider.com is of course a faked one. When writing this exercise Henrik used his own mail server.

Use netcat to launch a webserver

In one terminal, start netcat with the following arguments -l -p 8080.

$ nc -l -p 8080

This will open up a listening netcat, waiting for someone to connect on port 8080.

In another terminal, start netcat with the following arguments -p 8080.

$ nc localhost -p 8080

This will open up a connection to 8080, which is where your listening netcat "awaits your call".

Your netcat sessions are now "connected" over the local network so if you type something in one of the terminals you should be able to see the same text in the other. And vice verse.


Write a small script that outputs a valid web page to stdout

The script shall output something like this:

HTTP/1.1 200 OK
Connection: close
Date: Mon Feb  6 10:22:50 CET 2017
Server: netcat special deal
Content-Length: 136
Content-Type: text/html; charset=utf-8
Cache-Control: max-age=60



<!DOCTYPE html>
<html>

<head>
<title>Page Title</title>
</head>

<body>
Current date is: Mon Feb  6 10:22:50 CET 2017
</body>

</html>

Expand using link to the right to see a suggested solution.

Here's a sample script:

#!/bin/bash


content()
{
    echo "<!DOCTYPE html>"
    echo "<html>"
    echo ""
    echo "<head>"
    echo "<title>Page Title</title>"
    echo "</head>"
    echo ""
    echo "<body>"
    echo "Current date is: $(date)"
    echo "</body>"
    echo ""
    echo "</html>"
}

header()
{
    echo "HTTP/1.1 200 OK"
    echo "Connection: close"
    echo "Date: $(date)"
    echo "Server: netcat special deal"
    echo "Content-Length: $LENGTH"
    echo "Content-Type: text/html; charset=utf-8"
    echo "Cache-Control: max-age=60"
    echo ""
    echo ""
    echo ""
}

LENGTH=$(content | wc -c)

header
content


Here's a sample script you can use.

You can find complete source code to the suggested solutions below in the webserver-nc directory in this zip file or in the git repository.

Use the script together with netcat to make a web server

The scripts prints to stdout. Use a pipe to redirect the output of your script to be stdin for netcat. Add the same listning flags as above to netcat.

Once started you should be able to go to the URL localhost:8080 with a browser. Try reloading the page. Explain what happens when reloading.

Expand using link to the right to see a suggested solution.

$ ./webserver.sh | nc -l -p 8080

The page loads fine once. After this the netcat sessions is done and no reloading can be done.



Make the server start all over again in a loop

Start the above command in a loop (use while (true); do ......; done in bash). Once started you should be able to go to the URL localhost:8080 with a browser. Try reloading the page. Explain what happens when reloading.

Expand using link to the right to see a suggested solution.

$ while (true); do  ./webserver.sh | nc -l -p 8080 ;  done

The page loads fine once. After the page has been lodaded and netcat exits a netcat new session is started and reloading can be done.

Referer

Check the output from your netcat sessions. It most likely looks something like this

GET / HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
DNT: 1
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: en-US,en;q=0.8,sv;q=0.6

GET /favicon.ico HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36
Accept: image/webp,image/*,*/*;q=0.8
DNT: 1
Referer: http://localhost:8080/
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: en-US,en;q=0.8,sv;q=0.6

The important line in this printout right now is

Referer: http://localhost:8080/


Now, access this page (the same as before) by clicking here: localhost:8080. The output in the terminal where you're reunning your netcat/webserver should look something like this:

GET / HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
DNT: 1
Referer: http://virt08.itu.chalmers.se/mediawiki/index.php?title=MoreBash:Exercises_-_Scripts_-_Network_Tools&action=submit
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: en-US,en;q=0.8,sv;q=0.6

GET /favicon.ico HTTP/1.1
Host: localhost:8080
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36
Accept: image/webp,image/*,*/*;q=0.8
DNT: 1
Referer: http://localhost:8080/
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: en-US,en;q=0.8,sv;q=0.6

The important lines in this printout right now is

GET / HTTP/1.1
Referer: http://virt08.itu.chalmers.se/mediawiki/index.php?title=MoreBash:Exercises_-_Scripts_-_Network_Tools

and

GET /favicon.ico HTTP/1.1
Referer: http://localhost:8080/

The first referer comes from the browser (client) which tells the server that it came across this page by clicking on a link on page http://virt08.itu.chalmers.se/mediawiki/index.php?title=MoreBash:Exercises_-_Scripts_-_Network_Tools. The second referer is the browser, without us saying so, check if the server has a favicon. It does this by refering to the web page.

Links

External links

Example scripts

END_INCLUSION of MoreBash:Exercises_-_Network_Tools