MoreBash:Network Tools

From Juneday education
Jump to: navigation, search

Work in progress

This chapter is a work in progress. Remove this section when the page is production-ready.

Introduction

This chapter introduces some network tools and network commands in Bash.

After reading, move on to the exercises (link at the bottom).

Some examples

ping - send ICMP ECHO_REQUEST to network hosts

Used to see if you get an ICMP ECHO_RESPONSE from a host or gateway.

$ ping -c 5 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=121 time=19.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=121 time=13.9 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=121 time=14.1 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=121 time=17.5 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=121 time=14.3 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4003ms
rtt min/avg/max/mdev = 13.964/15.827/19.190/2.140 ms

Ping is often used when investigating network problems, to see what hosts are reachable.

host - DNS lookup utility

host is a simple utility for performing DNS lookups.

$ host 8.8.4.4
4.4.8.8.in-addr.arpa domain name pointer google-public-dns-b.google.com.

$ host www.gu.se
www.gu.se has address 130.241.151.114

nslookup - query Internet name servers interactively

Nslookup is a program to query Internet domain name servers. You can use it interactively or non-interactively.

Here's some examples using it non-interactively:

$ nslookup www.gu.se
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	www.gu.se
Address: 130.241.151.114

$ nslookup 130.241.151.114
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
114.151.241.130.in-addr.arpa	name = www.gu.se.

$ host www.gu.se ns1.net.gu.se
Using domain server:
Name: ns1.net.gu.se
Address: 130.241.150.34#53
Aliases: 

www.gu.se has address 130.241.151.114

Here's using it interactively:

$ nslookup
> Server 8.8.4.4
Default server: 8.8.4.4
Address: 8.8.4.4#53
> www.gu.se
Server:		8.8.4.4
Address:	8.8.4.4#53

Non-authoritative answer:
Name:	www.gu.se
Address: 130.241.151.114
>

You exit using Ctrl-d

dig - DNS lookup utility

With dig, you can query DNS servers for information on domains and servers.

Here's looking up the MX (mail exchange) servers for the domain gu.se:

$ dig gu.se ns

; <<>> DiG 9.10.3-P4-Ubuntu <<>> gu.se ns
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42301
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;gu.se.				IN	NS

;; ANSWER SECTION:
gu.se.			21599	IN	NS	sunic.sunet.se.
gu.se.			21599	IN	NS	ns2.net.gu.se.
gu.se.			21599	IN	NS	ns2.chalmers.se.
gu.se.			21599	IN	NS	ns1.net.gu.se.

;; Query time: 56 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Dec 25 03:49:56 CET 2018
;; MSG SIZE  rcvd: 127

Here's looking up the NS (name servers) for the same domain:

$ dig gu.se mx

; <<>> DiG 9.10.3-P4-Ubuntu <<>> gu.se mx
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25704
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;gu.se.				IN	MX

;; ANSWER SECTION:
gu.se.			722	IN	MX	10 v-mailfilter03.sunet.se.
gu.se.			722	IN	MX	11 e-mailfilter03.sunet.se.
gu.se.			722	IN	MX	11 e-mailfilter04.sunet.se.

;; Query time: 24 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Dec 25 03:52:54 CET 2018
;; MSG SIZE  rcvd: 133

whois - client for the whois directory service

whois is used to query various databases for information on domain names. Let's lookup gu.se:

$ whois gu.se
# Copyright (c) 1997- IIS (The Internet Foundation In Sweden).
# All rights reserved.
# The information obtained through searches, or otherwise, is protected
# by the Swedish Copyright Act (1960:729) and international conventions.
# It is also subject to database protection according to the Swedish
# Copyright Act.
# Any use of this material to target advertising or
# similar activities is forbidden and will be prosecuted.
# If any of the information below is transferred to a third
# party, it must be done in its entirety. This server must
# not be used as a backend for a search engine.
# Result of search for registered domain names under
# the .se top level domain.
# This whois printout is printed with UTF-8 encoding.
#
state:            active
domain:           gu.se
holder:           gtebor0811-00012
admin-c:          -
tech-c:           gtebor0811-00012
tech-c:           gtebor1309-00001
tech-c:           gtebor1309-00002
billing-c:        gtebor0811-00013
created:          1989-01-01
modified:         2016-11-10
expires:          2026-12-31
nserver:          ns1.net.gu.se 130.241.150.34 2001:6b0:d:9600::22
nserver:          ns2.chalmers.se
nserver:          ns2.net.gu.se 130.241.25.13 2001:6b0:d:25::25:13
nserver:          sunic.sunet.se
dnssec:           signed delegation
status:           ok
registrar:        SE Direkt

One use for whois, is to find out when a domain name expires (when the owner must renew the domain name ownership):

$ whois gu.se | grep expires:|awk '{print $2;}'
2026-12-31

$ whois aftonbladet.se | grep expires:|awk '{print $2;}'
2019-08-28

$ whois ituniv.se | grep expires:|awk '{print $2;}'
2019-10-18

$ whois sisa-org.se | grep expires:|awk '{print $2;}'
2019-08-28

ss - netstat replacement

Checks the network traffic:

$ ss -nat
State       Recv-Q Send-Q  Local Address:Port                 Peer Address:Port              
LISTEN      0      5           127.0.0.1:631                             *:*                  
CLOSE-WAIT  1      0        192.168.0.13:45818                129.16.69.98:80                 
CLOSE-WAIT  1      0        192.168.0.13:56936              104.236.216.52:80                 
CLOSE-WAIT  1      0        192.168.0.13:52910                52.216.32.80:443                
CLOSE-WAIT  1      0        192.168.0.13:53934                52.85.246.64:80                 
CLOSE-WAIT  417    0        192.168.0.13:50888                 31.13.72.12:443                
CLOSE-WAIT  433    0        192.168.0.13:50894                 31.13.72.12:443                
CLOSE-WAIT  1      0        192.168.0.13:52912                52.216.32.80:443                
ESTAB       0      0        192.168.0.13:51414              192.30.253.124:443                
CLOSE-WAIT  1      0        192.168.0.13:45816                129.16.69.98:80                 
ESTAB       0      0      192.168.43.105:53952              207.171.162.180:80                 
LISTEN      0      5                 ::1:631                            :::*

The flags -nat stands for nUmeric aLL tCP (write addresses numerically rather than using domain names, list all connections, show only TCP connections).

ip - ifconfig replacement

Shows and manipulates routing, devices, tunnels etc. Example:

$ ip -s -h address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    3.91M      44.6k    0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    3.91M      44.6k    0       0       0       0       
25: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e4:b3:18:0d:70:91 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.13/24 brd 192.168.0.255 scope global dynamic wlan0
       valid_lft 2744sec preferred_lft 2744sec
    inet6 fe80::e6b3:18ff:fe0d:7091/64 scope link 
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    14.7M      19.5k    0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    3.18M      17.1k    0       0       0       0

Flags -s -h stands for "statistics" and "human readable units".

To list all interfaces with their IPv4 adressses:

$ ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
25: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.0.13/24 brd 192.168.0.255 scope global dynamic wlan0
       valid_lft 2111sec preferred_lft 2111sec

Show a named interface (like wlan0 if that is the device name for your wlan card):

$ ip -4 a show wlan0
25: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.0.13/24 brd 192.168.0.255 scope global dynamic wlan0
       valid_lft 2004sec preferred_lft 2004sec

mtr - combines ping and traceroute

Example run - check route to ftp.sunet.se and network quality statistics:

$ mtr -i 1 -r ftp.sunet.se
Start: Sat Feb  4 16:12:44 2017
HOST: dellasoul                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.0.1                0.0%    10    0.7   2.2   0.7  10.4   2.9
  2.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  3.|-- fa-bbr-1-be10-10.net.comh  0.0%    10    7.5  10.8   7.5  24.9   5.0
  4.|-- vrr-core-1-be104.net.comh  0.0%    10   15.6  15.9  13.8  17.9   1.1
  5.|-- mtc-core-1-be2.net.comhem  0.0%    10   16.3  21.5  14.1  72.8  18.1
  6.|-- 213.200.162.32             0.0%    10   15.6  20.8  13.0  53.8  12.2
  7.|-- netnod-ix-ge-a-sth-4470.s  0.0%    10   14.7  17.2  12.4  33.9   5.9
  8.|-- stockholm-fre-r1.sunet.se  0.0%    10   14.0  15.7  11.7  21.0   2.4
  9.|-- uppsala-upa-r1.sunet.se    0.0%    10   16.3  16.9  14.2  20.1   1.6
 10.|-- gavle-sbo-r1.sunet.se      0.0%    10   17.6  17.8  15.9  23.0   1.9
 11.|-- sundsvall-sva-r1.sunet.se  0.0%    10   24.8  20.9  19.1  24.8   1.7
 12.|-- umu2.sunet.se              0.0%    10   22.2  24.8  20.7  32.3   4.2
 13.|-- 130.239.0.29               0.0%    10   21.6  33.7  19.7 126.4  32.9
 14.|-- 130.239.0.86               0.0%    10   23.1  28.6  22.3  68.8  14.3
 15.|-- hammurabi.acc.umu.se       0.0%    10   23.4  24.1  21.4  30.4   2.6

Flags used: -i 1 -r i stands for interval - how many seconds between probes and r stands for "report" mode - run tests and print the report. Otherwise the program runs interactively.

tracepath - similar to traceroute

Prints the route (path) to a host. Example run:

$ tracepath ftp.sunet.se
 1?: [LOCALHOST]                                         pmtu 1500
 1:  192.168.0.1                                           3.862ms 
 1:  192.168.0.1                                           1.153ms 
 2:  no reply
 3:  fa-bbr-1-be10-10.net.comhem.se                       12.223ms 
 4:  vrr-core-1-be104.net.comhem.se                       16.208ms asymm  8 
 5:  nap-core-1-be1.net.comhem.se                         15.040ms asymm  7 
 6:  213.200.162.34                                       17.904ms 
 7:  netnod-ix-ge-b-sth-4470.sunet.se                     18.819ms 
 8:  uppsala-upa-r1.sunet.se                              22.934ms 
 9:  gavle-sbo-r1.sunet.se                                21.504ms 
10:  sundsvall-sva-r1.sunet.se                            21.673ms 
11:  umu2.sunet.se                                        22.872ms 
12:  130.239.0.29                                         24.426ms asymm 13 
13:  130.239.0.86                                         25.424ms asymm 14 
14:  hammurabi.acc.umu.se                                 30.087ms reached
     Resume: pmtu 1500 hops 14 back 15

Compare to traceroute:

$ traceroute ftp.sunet.se
traceroute to ftp.sunet.se (130.239.18.165), 30 hops max, 60 byte packets
 1  192.168.0.1 (192.168.0.1)  2.656 ms  3.077 ms  13.482 ms
 2  * * *
 3  fa-bbr-1-be10-10.net.comhem.se (213.200.164.85)  28.906 ms  28.921 ms  28.913 ms
 4  vrr-core-1-be104.net.comhem.se (213.200.163.217)  35.208 ms  35.219 ms  38.866 ms
 5  mtc-core-1-be2.net.comhem.se (213.200.162.10)  38.886 ms  38.879 ms nap-core-1-be1.net.comhem.se (213.200.162.2)  38.869 ms
 6  213.200.162.34 (213.200.162.34)  38.857 ms 213.200.162.30 (213.200.162.30)  19.846 ms 213.200.162.32 (213.200.162.32)  16.446 ms
 7  netnod-ix-ge-b-sth-4470.sunet.se (195.69.119.19)  16.442 ms  19.647 ms netnod-ix-ge-a-sth-4470.sunet.se (195.245.240.19)  21.605 ms
 8  uppsala-upa-r1.sunet.se (130.242.4.25)  22.670 ms stockholm-fre-r1.sunet.se (130.242.4.94)  22.868 ms uppsala-upa-r1.sunet.se (130.242.4.25)  27.058 ms
 9  uppsala-upa-r1.sunet.se (130.242.4.27)  23.870 ms  26.171 ms  28.471 ms
10  gavle-sbo-r1.sunet.se (130.242.4.18)  30.035 ms  28.747 ms  19.219 ms
11  umu2.sunet.se (130.242.6.147)  26.835 ms  25.329 ms sundsvall-sva-r1.sunet.se (130.242.4.17)  25.165 ms
12  umu2.sunet.se (130.242.6.147)  25.657 ms  24.739 ms 130.239.0.29 (130.239.0.29)  27.136 ms
13  130.239.0.29 (130.239.0.29)  24.999 ms 130.239.0.90 (130.239.0.90)  70.756 ms 130.239.0.29 (130.239.0.29)  25.751 ms
14  130.239.0.86 (130.239.0.86)  65.610 ms 130.239.0.90 (130.239.0.90)  65.834 ms hammurabi.acc.umu.se (130.239.18.165)  24.865 ms

wget


A non-interactive network downloader.

GNU Wget (or just Wget, formerly Geturl, also written as its package name, wget) is a computer program that retrieves content from web servers. It is part of the GNU Project. Its name derives from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP.
- Wikipedia on wget


$ wget --no-check-certificate --content-disposition https://github.com/progund/tig015-weekly/raw/master/beverages-framework/resources/sortiment.xml


For more examples, check out Download files

curl


Description: transfer a URL

cURL is a computer software project providing a library and command-line tool for transferring data using various protocols. The cURL project produces two products, libcurl and cURL. It was first released in 1997. The name originally stood for "see URL".
- Wikipedia on curl


Example:

$ curl -LJO -k https://github.com/progund/tig015-weekly/raw/master/beverages-framework/resources/sortiment.xml


For more examples, check out Download files

nc


Description: ncat - Concatenate and redirect sockets. Sometimes nc, netc, netcat.

Netcat (often abbreviated to nc) is a computer networking utility for reading from and writing to network connections using TCP or UDP. Netcat is designed to be a dependable back-end that can be used directly or easily driven by other programs and scripts. At the same time, it is a feature-rich network debugging and investigation tool, since it can produce almost any kind of connection its user could need and has a number of built-in capabilities.
- Wikipedia on netcat


Example on how to use:

Connect to www.apache.org web server and get the top html page.

$ nc www.apache.org 80
GET / HTTP 1.1                 

HTTP/1.1 200 OK
Date: Tue, 07 Feb 2017 08:34:34 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Tue, 07 Feb 2017 08:10:32 GMT
ETag: "d716-547ec482a0c1c"
Accept-Ranges: bytes
Content-Length: 55062
Vary: Accept-Encoding
Cache-Control: max-age=3600
Expires: Tue, 07 Feb 2017 09:34:34 GMT
Connection: close
Content-Type: text/html

<!DOCTYPE html>
<html lang="en">
<head>
......

Start up a listening server on port 9090.

$ nc -l -p 9090

... and connect to that server using netcat (in another terminal)

$ nc localhost 9090

You can now transfer text between the two.

Printing some information about a host

Here's an inspirational one-liner in bash to get some information about a host. You can look at the command line and figure out how to make a script out of it, so that you can provide the host address as an argument instead.

$ echo 'IP(s)' of ftp.sunet.se: $(host ftp.sunet.se|tr ' ' '\n'|egrep '^[0-9]+'|tr '\n' ' ');echo From this computer, there are $(traceroute ftp.sunet.se|tail -1 | awk '{print $1;}') hops to ftp.sunet.se.;echo The domain sunet.se $(whois sunet.se|grep expires|sed -e 's/\( *\)\([0-9].*\)/ \2/')
IP(s) of ftp.sunet.se: 130.239.18.173 130.239.18.165 2001:6b0:e:2018::173 2001:6b0:e:2018::165 0
From this computer, there are 14 hops to ftp.sunet.se.
The domain sunet.se expires: 2017-12-31

The line IP(s) of ftp.sunet.se: 130.239.18.173 130.239.18.165 2001:6b0:e:2018::173 2001:6b0:e:2018::165 0 comes from the following command:

$ echo 'IP(s)' of ftp.sunet.se: $(host ftp.sunet.se|tr ' ' '\n'|egrep '^[0-9]+'|tr '\n' ' ')

Run host ftp.sunet.se and replace all spaces with newlines. Then grep for lines starting with numbers and replace the newlines with spaces again.

The line From this computer, there are 14 hops to ftp.sunet.se. comes from the following command:

$ echo From this computer, there are $(traceroute ftp.sunet.se|tail -1 | awk '{print $1;}') hops to ftp.sunet.se.

Run traceroute and keep only the last line, which you send to awk which prints only the first token.

The line The domain sunet.se expires: 2017-12-31 comes from the following command:

$ echo The domain sunet.se $(whois sunet.se|grep expires|sed -e 's/\( *\)\([0-9].*\)/ \2/')

Run whois sunet.se and grep for the line with "expires". Send the line to sed, which replaces all the spaces with only one space.

How many videos have Henrik and Rikard published on Vimeo?

$ w3m -dump 'https://vimeo.com/user52531669/collections'|grep -i videos|egrep '^ *[0-9]+ Videos';LC_TIME=en_US date
    665 Videos
Tue Dec 25 02:56:47 CET 2018

How many github repositories with different programming languages exist for progund?

$ echo "There are $(w3m -dump 'https://github.com/progund?language=java'|grep "Overview Repositories"|sed -e 's/Overview Repositories //') repositories with Java";LC_TIME=en_US date
There are 75 repositories with Java
Thu Feb 15 11:16:25 CET 2018
$ echo "There are $(w3m -dump 'https://github.com/progund?language=shell'|grep "Overview Repositories"|sed -e 's/Overview Repositories //') repositories with Shell scripts";LC_TIME=en_US date
There are 75 repositories with Shell scripts
Thu Feb 15 11:17:34 CET 2018

How many Java files etc exist in some github repo?

Doesn't work anymore, since github layout changed - fix this as an exercise! Search github and figure out how to create the url etc.

Below isn't correct. Please, students, fix this! ;-)

Let's see Progund's java-web repository:

$ w3m -dump 'https://github.com/progund/java-web/search?l=XML&q=.java&utf8=%E2%9C%93'|awk '/Languages/,/Search.all.of/'|grep Java|sed -e 's/\(.* \)\([0-9].*\)/\2/'
75 Java

How many PDFs exists on this Wiki?

Let's use w3m to dump the text from this wiki's page with a list of PDFs:

$ w3m -dump 'http://wiki.juneday.se/mediawiki/index.php?title=Special:MIMESearch/application/pdf&limit=500&offset=0&mime=application%2Fpdf' | grep '(download)' | tail -1 | awk '{print $1;}'
239.

We use grep to filter rows with (download), and we keep only the last line, which we take the first column from.

Slides and videos

Links

Bash - tools

External links

Source code

Chapter links

previous (Exercises - Tools) | Next (exercises)