Never confuse education with intelligence, you can have a PhD and still be an idiot.
- Richard Feynman -



ITIC:Network tools and commands

From Juneday education
Revision as of 13:27, 13 August 2019 by Rikard (Talk | contribs) (Removed TODO)

Jump to: navigation, search

Below, we have included a page from another course material on this wiki. The material is overlapping with the course material for the Introduction to IT and computing, but we have videos and lecture slides that are new and produced exclusively for this course material.

The tools we want you to focus on (but feel free to learn about more tools!) are:

  • ping - see if remote computer responds
  • traceroute/mtr - see hops to remote computer
  • host/nslookup/dig - translate between names and IP numbers
  • whois - get information about domain names
  • ip/ifconfig/hostname -I - check your own IP number etc
  • route/ip r/netstat -nr - see what your default gateway is
  • netstat/ss - check your computer's network activity
  • lastlog - see who has logged in remotely to your computer
  • curl/wget - download files from the web
  • lwp-request/GET/HEAD - send HTTP requests to web servers
  • ssh - login on remote computer
  • scp/rsync - download/upload securely from/to remote computer
  • /etc/hosts - file with hostnames and IP addresses (local DNS)
  • /etc/resolv.conf - file with DNS settings

The best way to learn how to use these tools is to use them! Try them out and do the exercises. For those of you who don't run GNU/Linux, find the equivalent tools for Windows or macOS. If there are the same tools on your OS as the GNU/Linux (and UNIX) tools we show here, find out how the syntax or flags differ. Use a search engine to find out the facts, and write them down as documentation for future reference.

Trouble shooting example

Knowing these basic network tools helps enormously when diagnosing network problems. Knowing where on the network the problem is will save you a lot of time and problems. Let's say you can't get out on the web at home. Is it the web page that is down? Can you reach any web pages? What about using an IP number? Can you reach, e.g. http://130.241.135.117 (or wiki as per 2019-08-13)?

If you can reach http://130.241.135.117 but not http://wiki.juneday.se, what do you think the problem is? It's DNS related, of course. Check your DNS settings. Can you ping your DNS? Is it down? Can you use another DNS (an easy-to-remember DNS IP is 8.8.8.8 - Google's DNS): 

$ host wiki.juneday.se 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases: 

wiki.juneday.se has address 130.241.135.117

In the example above, I used 8.8.8.8 to lookup the IP for wiki.juneday.se. If I can do that, but not use the default DNS (the one my computer is configured to use), I know that the problem is with my DNS settings.

If I can't reach any DNS, then how far from my computer can I get? Can I ping my gateway? (the computer between my local network and the Internet is called gateway, as you might remember.)

Let's first find out what my gateway IP is: 

$ route | grep default
default         10.0.0.1        0.0.0.0         UG    600    0        0 wlp58s0

Can I ping the gateway's internal network card? That is, the network card it uses for the private local network. 

$ ping -c 5 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.

--- 10.0.0.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4001ms

The above either means I can't reach the gateway my computer has configured, or that the gateway doesn't allow ping (for some reason some network technicians think that disallowing ping is an effective security measure - we are not so sure).

If I can ping the gateway's internal network card, how far out on the network can I get?

Here's a successfull traceroute all the way to Google's DNS: 

$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max
  1   10.0.0.101  2,593ms  1,840ms  3,485ms 
  2   129.16.4.177  59,812ms  55,974ms  27,809ms 
  3   129.16.2.185  2,300ms  3,418ms  2,089ms 
  4   130.242.6.10  2,300ms  3,724ms  2,056ms 
  5   130.242.4.176  2,315ms  2,202ms  2,172ms 
  6   130.242.4.44  3,596ms  2,463ms  2,181ms 
  7   130.242.4.49  3,954ms  3,901ms  3,989ms 
  8   130.242.4.73  6,824ms  5,666ms  5,764ms 
  9   130.242.4.71  7,660ms  6,028ms  6,264ms 
 10   109.105.102.122  7,046ms  6,558ms  6,760ms 
 11   109.105.97.57  10,666ms  10,703ms  10,805ms 
 12   193.42.155.46  10,847ms  10,764ms  14,291ms 
 13   108.170.253.49  20,602ms  32,088ms  20,699ms 
 14   216.239.43.123  20,418ms  20,671ms  21,109ms 
 15   8.8.8.8  19,262ms  20,767ms  19,333ms

Here's the same thing using mtr instead of traceroute: 

$ mtr -nr 8.8.8.8
Start: Tue Aug 13 08:38:09 2019
HOST: newdelli                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.0.0.101                 0.0%    10    9.6   4.0   2.0   9.6   2.4
  2.|-- 129.16.4.177               0.0%    10    2.5   9.3   2.1  42.9  13.1
  3.|-- 129.16.2.185               0.0%    10    2.5   2.5   2.3   3.8   0.3
  4.|-- 130.242.6.10               0.0%    10    3.6   4.5   2.2  18.4   4.9
  5.|-- 130.242.4.176              0.0%    10    2.5  10.9   2.4  50.4  16.3
  6.|-- 130.242.4.44               0.0%    10    2.5   3.8   2.3   7.3   1.4
  7.|-- 130.242.4.49               0.0%    10    4.1   5.2   4.1  11.3   2.1
  8.|-- 130.242.4.73               0.0%    10    7.3   8.1   5.9  18.9   4.0
  9.|-- 130.242.4.71               0.0%    10    7.9   7.2   6.1  10.9   1.4
 10.|-- 109.105.102.122            0.0%    10    6.8   8.4   6.6  14.9   2.9
 11.|-- 109.105.97.57              0.0%    10   11.9  12.7  10.8  25.9   4.6
 12.|-- 193.42.155.46              0.0%    10   33.2  14.1  10.9  33.2   6.8
 13.|-- 108.170.253.33             0.0%    10   19.6  20.0  19.6  21.2   0.0
 14.|-- 209.85.251.131             0.0%    10   21.2  20.2  19.6  21.2   0.5
 15.|-- 8.8.8.8                    0.0%    10   20.0  20.6  20.0  23.0   0.8

As you see, knowing a few basic network tools can help you diagnose network problems at your home, school or work. Please take the time to watch the videos and do the exercises, and maybe you will be the network hero next time your colleagues can't get on Facebook or something similarly critical.

Some help for finding Windows and macOS equivalent tools

Here's a few sites to get you started:

Links(Intro IT and computing - network tools)

Videos and video slides

Swedish videos

English videos

Summary lecture slides

Workshop slides

  • TODO

Where to go next

The next page is ITIC:Network_tools_and_commands_-_Exercises.

« Previous • Book TOC • Next »

Included material

For now, inclusion of MoreBash:Network_Tools below:

Work in progress

This chapter is a work in progress. Remove this section when the page is production-ready.

Introduction

This chapter introduces some network tools and network commands in Bash.

After reading, move on to the exercises (link at the bottom).

Some examples

ping - send ICMP ECHO_REQUEST to network hosts

Used to see if you get an ICMP ECHO_RESPONSE from a host or gateway. 

$ ping -c 5 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=121 time=19.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=121 time=13.9 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=121 time=14.1 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=121 time=17.5 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=121 time=14.3 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4003ms
rtt min/avg/max/mdev = 13.964/15.827/19.190/2.140 ms

Ping is often used when investigating network problems, to see what hosts are reachable.

host - DNS lookup utility

host is a simple utility for performing DNS lookups. 

$ host 8.8.4.4
4.4.8.8.in-addr.arpa domain name pointer google-public-dns-b.google.com.

$ host www.gu.se
www.gu.se has address 130.241.151.114

nslookup - query Internet name servers interactively

Nslookup is a program to query Internet domain name servers. You can use it interactively or non-interactively.

Here's some examples using it non-interactively: 

$ nslookup www.gu.se
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	www.gu.se
Address: 130.241.151.114

$ nslookup 130.241.151.114
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
114.151.241.130.in-addr.arpa	name = www.gu.se.

$ host www.gu.se ns1.net.gu.se
Using domain server:
Name: ns1.net.gu.se
Address: 130.241.150.34#53
Aliases: 

www.gu.se has address 130.241.151.114

Here's using it interactively: 

$ nslookup
> Server 8.8.4.4
Default server: 8.8.4.4
Address: 8.8.4.4#53
> www.gu.se
Server:		8.8.4.4
Address:	8.8.4.4#53

Non-authoritative answer:
Name:	www.gu.se
Address: 130.241.151.114
>

You exit using Ctrl-d

dig - DNS lookup utility

With dig, you can query DNS servers for information on domains and servers.

Here's looking up the MX (mail exchange) servers for the domain gu.se: 

$ dig gu.se ns

; <<>> DiG 9.10.3-P4-Ubuntu <<>> gu.se ns
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42301
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;gu.se.				IN	NS

;; ANSWER SECTION:
gu.se.			21599	IN	NS	sunic.sunet.se.
gu.se.			21599	IN	NS	ns2.net.gu.se.
gu.se.			21599	IN	NS	ns2.chalmers.se.
gu.se.			21599	IN	NS	ns1.net.gu.se.

;; Query time: 56 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Dec 25 03:49:56 CET 2018
;; MSG SIZE  rcvd: 127

Here's looking up the NS (name servers) for the same domain: 

$ dig gu.se mx

; <<>> DiG 9.10.3-P4-Ubuntu <<>> gu.se mx
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25704
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;gu.se.				IN	MX

;; ANSWER SECTION:
gu.se.			722	IN	MX	10 v-mailfilter03.sunet.se.
gu.se.			722	IN	MX	11 e-mailfilter03.sunet.se.
gu.se.			722	IN	MX	11 e-mailfilter04.sunet.se.

;; Query time: 24 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Dec 25 03:52:54 CET 2018
;; MSG SIZE  rcvd: 133

whois - client for the whois directory service

whois is used to query various databases for information on domain names. Let's lookup gu.se: 

$ whois gu.se
# Copyright (c) 1997- IIS (The Internet Foundation In Sweden).
# All rights reserved.
# The information obtained through searches, or otherwise, is protected
# by the Swedish Copyright Act (1960:729) and international conventions.
# It is also subject to database protection according to the Swedish
# Copyright Act.
# Any use of this material to target advertising or
# similar activities is forbidden and will be prosecuted.
# If any of the information below is transferred to a third
# party, it must be done in its entirety. This server must
# not be used as a backend for a search engine.
# Result of search for registered domain names under
# the .se top level domain.
# This whois printout is printed with UTF-8 encoding.
#
state:            active
domain:           gu.se
holder:           gtebor0811-00012
admin-c:          -
tech-c:           gtebor0811-00012
tech-c:           gtebor1309-00001
tech-c:           gtebor1309-00002
billing-c:        gtebor0811-00013
created:          1989-01-01
modified:         2016-11-10
expires:          2026-12-31
nserver:          ns1.net.gu.se 130.241.150.34 2001:6b0:d:9600::22
nserver:          ns2.chalmers.se
nserver:          ns2.net.gu.se 130.241.25.13 2001:6b0:d:25::25:13
nserver:          sunic.sunet.se
dnssec:           signed delegation
status:           ok
registrar:        SE Direkt

One use for whois, is to find out when a domain name expires (when the owner must renew the domain name ownership): 

$ whois gu.se | grep expires:|awk '{print $2;}'
2026-12-31

$ whois aftonbladet.se | grep expires:|awk '{print $2;}'
2019-08-28

$ whois ituniv.se | grep expires:|awk '{print $2;}'
2019-10-18

$ whois sisa-org.se | grep expires:|awk '{print $2;}'
2019-08-28

ss - netstat replacement

Checks the network traffic: 

$ ss -nat
State       Recv-Q Send-Q  Local Address:Port                 Peer Address:Port              
LISTEN      0      5           127.0.0.1:631                             *:*                  
CLOSE-WAIT  1      0        192.168.0.13:45818                129.16.69.98:80                 
CLOSE-WAIT  1      0        192.168.0.13:56936              104.236.216.52:80                 
CLOSE-WAIT  1      0        192.168.0.13:52910                52.216.32.80:443                
CLOSE-WAIT  1      0        192.168.0.13:53934                52.85.246.64:80                 
CLOSE-WAIT  417    0        192.168.0.13:50888                 31.13.72.12:443                
CLOSE-WAIT  433    0        192.168.0.13:50894                 31.13.72.12:443                
CLOSE-WAIT  1      0        192.168.0.13:52912                52.216.32.80:443                
ESTAB       0      0        192.168.0.13:51414              192.30.253.124:443                
CLOSE-WAIT  1      0        192.168.0.13:45816                129.16.69.98:80                 
ESTAB       0      0      192.168.43.105:53952              207.171.162.180:80                 
LISTEN      0      5                 ::1:631                            :::*

The flags -nat stands for Numeric All Tcp (write addresses numerically rather than using domain names, list all connections, show only TCP connections).

ip - ifconfig replacement

Shows and manipulates routing, devices, tunnels etc. Example: 

$ ip -s -h address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    3.91M      44.6k    0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    3.91M      44.6k    0       0       0       0       
25: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e4:b3:18:0d:70:91 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.13/24 brd 192.168.0.255 scope global dynamic wlan0
       valid_lft 2744sec preferred_lft 2744sec
    inet6 fe80::e6b3:18ff:fe0d:7091/64 scope link 
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    14.7M      19.5k    0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    3.18M      17.1k    0       0       0       0

Flags -s -h stands for "statistics" and "human readable units".

To list all interfaces with their IPv4 adressses: 

$ ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
25: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.0.13/24 brd 192.168.0.255 scope global dynamic wlan0
       valid_lft 2111sec preferred_lft 2111sec

Show a named interface (like wlan0 if that is the device name for your wlan card): 

$ ip -4 a show wlan0
25: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.0.13/24 brd 192.168.0.255 scope global dynamic wlan0
       valid_lft 2004sec preferred_lft 2004sec

mtr - combines ping and traceroute

Example run - check route to ftp.sunet.se and network quality statistics: 

$ mtr -i 1 -r ftp.sunet.se
Start: Sat Feb  4 16:12:44 2017
HOST: dellasoul                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.0.1                0.0%    10    0.7   2.2   0.7  10.4   2.9
  2.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  3.|-- fa-bbr-1-be10-10.net.comh  0.0%    10    7.5  10.8   7.5  24.9   5.0
  4.|-- vrr-core-1-be104.net.comh  0.0%    10   15.6  15.9  13.8  17.9   1.1
  5.|-- mtc-core-1-be2.net.comhem  0.0%    10   16.3  21.5  14.1  72.8  18.1
  6.|-- 213.200.162.32             0.0%    10   15.6  20.8  13.0  53.8  12.2
  7.|-- netnod-ix-ge-a-sth-4470.s  0.0%    10   14.7  17.2  12.4  33.9   5.9
  8.|-- stockholm-fre-r1.sunet.se  0.0%    10   14.0  15.7  11.7  21.0   2.4
  9.|-- uppsala-upa-r1.sunet.se    0.0%    10   16.3  16.9  14.2  20.1   1.6
 10.|-- gavle-sbo-r1.sunet.se      0.0%    10   17.6  17.8  15.9  23.0   1.9
 11.|-- sundsvall-sva-r1.sunet.se  0.0%    10   24.8  20.9  19.1  24.8   1.7
 12.|-- umu2.sunet.se              0.0%    10   22.2  24.8  20.7  32.3   4.2
 13.|-- 130.239.0.29               0.0%    10   21.6  33.7  19.7 126.4  32.9
 14.|-- 130.239.0.86               0.0%    10   23.1  28.6  22.3  68.8  14.3
 15.|-- hammurabi.acc.umu.se       0.0%    10   23.4  24.1  21.4  30.4   2.6

Flags used: -i 1 -r i stands for interval - how many seconds between probes and r stands for "report" mode - run tests and print the report. Otherwise the program runs interactively.

tracepath - similar to traceroute

Prints the route (path) to a host. Example run: 

$ tracepath ftp.sunet.se
 1?: [LOCALHOST]                                         pmtu 1500
 1:  192.168.0.1                                           3.862ms 
 1:  192.168.0.1                                           1.153ms 
 2:  no reply
 3:  fa-bbr-1-be10-10.net.comhem.se                       12.223ms 
 4:  vrr-core-1-be104.net.comhem.se                       16.208ms asymm  8 
 5:  nap-core-1-be1.net.comhem.se                         15.040ms asymm  7 
 6:  213.200.162.34                                       17.904ms 
 7:  netnod-ix-ge-b-sth-4470.sunet.se                     18.819ms 
 8:  uppsala-upa-r1.sunet.se                              22.934ms 
 9:  gavle-sbo-r1.sunet.se                                21.504ms 
10:  sundsvall-sva-r1.sunet.se                            21.673ms 
11:  umu2.sunet.se                                        22.872ms 
12:  130.239.0.29                                         24.426ms asymm 13 
13:  130.239.0.86                                         25.424ms asymm 14 
14:  hammurabi.acc.umu.se                                 30.087ms reached
     Resume: pmtu 1500 hops 14 back 15

Compare to traceroute: 

$ traceroute ftp.sunet.se
traceroute to ftp.sunet.se (130.239.18.165), 30 hops max, 60 byte packets
 1  192.168.0.1 (192.168.0.1)  2.656 ms  3.077 ms  13.482 ms
 2  * * *
 3  fa-bbr-1-be10-10.net.comhem.se (213.200.164.85)  28.906 ms  28.921 ms  28.913 ms
 4  vrr-core-1-be104.net.comhem.se (213.200.163.217)  35.208 ms  35.219 ms  38.866 ms
 5  mtc-core-1-be2.net.comhem.se (213.200.162.10)  38.886 ms  38.879 ms nap-core-1-be1.net.comhem.se (213.200.162.2)  38.869 ms
 6  213.200.162.34 (213.200.162.34)  38.857 ms 213.200.162.30 (213.200.162.30)  19.846 ms 213.200.162.32 (213.200.162.32)  16.446 ms
 7  netnod-ix-ge-b-sth-4470.sunet.se (195.69.119.19)  16.442 ms  19.647 ms netnod-ix-ge-a-sth-4470.sunet.se (195.245.240.19)  21.605 ms
 8  uppsala-upa-r1.sunet.se (130.242.4.25)  22.670 ms stockholm-fre-r1.sunet.se (130.242.4.94)  22.868 ms uppsala-upa-r1.sunet.se (130.242.4.25)  27.058 ms
 9  uppsala-upa-r1.sunet.se (130.242.4.27)  23.870 ms  26.171 ms  28.471 ms
10  gavle-sbo-r1.sunet.se (130.242.4.18)  30.035 ms  28.747 ms  19.219 ms
11  umu2.sunet.se (130.242.6.147)  26.835 ms  25.329 ms sundsvall-sva-r1.sunet.se (130.242.4.17)  25.165 ms
12  umu2.sunet.se (130.242.6.147)  25.657 ms  24.739 ms 130.239.0.29 (130.239.0.29)  27.136 ms
13  130.239.0.29 (130.239.0.29)  24.999 ms 130.239.0.90 (130.239.0.90)  70.756 ms 130.239.0.29 (130.239.0.29)  25.751 ms
14  130.239.0.86 (130.239.0.86)  65.610 ms 130.239.0.90 (130.239.0.90)  65.834 ms hammurabi.acc.umu.se (130.239.18.165)  24.865 ms

wget


A non-interactive network downloader.

GNU Wget (or just Wget, formerly Geturl, also written as its package name, wget) is a computer program that retrieves content from web servers. It is part of the GNU Project. Its name derives from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP.
- Wikipedia on wget 


$ wget --no-check-certificate --content-disposition https://github.com/progund/tig015-weekly/raw/master/beverages-framework/resources/sortiment.xml


For more examples, check out Download files

curl


Description: transfer a URL

cURL is a computer software project providing a library and command-line tool for transferring data using various protocols. The cURL project produces two products, libcurl and cURL. It was first released in 1997. The name originally stood for "see URL".
- Wikipedia on curl


Example: 

$ curl -LJO -k https://github.com/progund/tig015-weekly/raw/master/beverages-framework/resources/sortiment.xml


For more examples, check out Download files

nc


Description: nc — arbitrary TCP and UDP connections and listens

Sometimes installed as nc, netc, netcat, ncat.

From the man file:

The nc (or netcat) utility is used for just about anything under the sun involving TCP, UDP, or UNIX-domain sockets. It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scan‐ ning, and deal with both IPv4 and IPv6. Unlike telnet(1), nc scripts nicely, and separates error messages onto standard error instead of send‐

ing them to standard output, as telnet(1) does with some.


Example on how to use:

Connect to www.apache.org web server and get the top html page.

Using -C flag to get the correct line-endings: 

$ nc -C www.apache.org 80
GET / HTTP/1.0
Host: www

HTTP/1.1 200 OK
Date: Wed, 03 Jul 2019 07:08:16 GMT
Server: Apache/2.4.18 (Ubuntu)
Last-Modified: Wed, 03 Jul 2019 06:10:21 GMT
ETag: "136f2-58cc0b759e22b"
Accept-Ranges: bytes
Content-Length: 79602
Vary: Accept-Encoding
Cache-Control: max-age=3600
Expires: Wed, 03 Jul 2019 08:08:16 GMT
Connection: close
Content-Type: text/html

<!DOCTYPE html>
<html lang="en">
<head>
...etc, etc...

Your input is hightlighted. The rest is the response from the web server. Please note that you need two line-breaks after the Host: header.

Using echo and pipes (to get the correct line-endings): 

$ echo -e 'GET / HTTP/1.0\r\nHost: www\r\nConnection: close\r\n\r\n'|nc www.apache.org 80
HTTP/1.1 200 OK
Date: Tue, 07 Feb 2017 08:34:34 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Tue, 07 Feb 2017 08:10:32 GMT
ETag: "d716-547ec482a0c1c"
Accept-Ranges: bytes
Content-Length: 55062
Vary: Accept-Encoding
Cache-Control: max-age=3600
Expires: Tue, 07 Feb 2017 09:34:34 GMT
Connection: close
Content-Type: text/html

<!DOCTYPE html>
<html lang="en">
<head>
......

Start up a listening server on port 9090. 

$ nc -l -p 9090

... and connect to that server using netcat (in another terminal) 

$ nc localhost 9090

You can now transfer text between the two.

Printing some information about a host

Here's an inspirational one-liner in bash to get some information about a host. You can look at the command line and figure out how to make a script out of it, so that you can provide the host address as an argument instead. 

$ echo 'IP(s)' of ftp.sunet.se: $(host ftp.sunet.se|tr ' ' '\n'|egrep '^[0-9]+'|tr '\n' ' ');echo From this computer, there are $(traceroute ftp.sunet.se|tail -1 | awk '{print $1;}') hops to ftp.sunet.se.;echo The domain sunet.se $(whois sunet.se|grep expires|sed -e 's/\( *\)\([0-9].*\)/ \2/')
IP(s) of ftp.sunet.se: 130.239.18.173 130.239.18.165 2001:6b0:e:2018::173 2001:6b0:e:2018::165 0
From this computer, there are 14 hops to ftp.sunet.se.
The domain sunet.se expires: 2017-12-31

The line IP(s) of ftp.sunet.se: 130.239.18.173 130.239.18.165 2001:6b0:e:2018::173 2001:6b0:e:2018::165 0 comes from the following command: 

$ echo 'IP(s)' of ftp.sunet.se: $(host ftp.sunet.se|tr ' ' '\n'|egrep '^[0-9]+'|tr '\n' ' ')

Run host ftp.sunet.se and replace all spaces with newlines. Then grep for lines starting with numbers and replace the newlines with spaces again.

The line From this computer, there are 14 hops to ftp.sunet.se. comes from the following command: 

$ echo From this computer, there are $(traceroute ftp.sunet.se|tail -1 | awk '{print $1;}') hops to ftp.sunet.se.

Run traceroute and keep only the last line, which you send to awk which prints only the first token.

The line The domain sunet.se expires: 2017-12-31 comes from the following command: 

$ echo The domain sunet.se $(whois sunet.se|grep expires|sed -e 's/\( *\)\([0-9].*\)/ \2/')

Run whois sunet.se and grep for the line with "expires". Send the line to sed, which replaces all the spaces with only one space.

How many videos have Henrik and Rikard published on Vimeo?

$ w3m -dump 'https://vimeo.com/user52531669/collections'|grep -i videos|egrep '^ *[0-9]+ Videos';LC_TIME=en_US date
    678 Videos
Wed Jul  3 08:40:08 CEST 2019

How many github repositories with different programming languages exist for progund?

$ echo "There are $(w3m -dump 'https://github.com/progund?language=java'|grep "results for repositories"|cut -d ' ' -f1) repositories with Java";LC_TIME=en_US date
There are 66 repositories with Java
Wed Jul  3 08:30:30 CEST 2019

$ echo "There are $(w3m -dump 'https://github.com/progund?language=shell'|grep "results for repositories"|cut -d ' ' -f1) repositories with Shell scripts";LC_TIME=en_US date
There are 12 repositories with Shell scripts
Wed Jul  3 08:33:17 CEST 2019

How many Java files etc exist in some github repo?

Let's see Progund's java-web repository: 

$ w3m -dump 'https://github.com/progund/java-web/search?q=.java&utf8=%E2%9C%93'|grep '.*• [0-9].*'
  • 85 Java
  • 22 Shell
  • 17 XML
  • 4 PLpgSQL
  • 4 Text
  • 3 JSON
  • 3 Markdown

How many PDFs exists on this Wiki?

Let's use w3m to dump the text from this wiki's page with a list of PDFs: 

$ w3m -dump 'http://wiki.juneday.se/mediawiki/index.php?title=Special:MIMESearch/application/pdf&limit=500&offset=0&mime=application%2Fpdf' | grep '(download)' | tail -1 | awk '{print $1;}'
336.

We use grep to filter rows with (download), and we keep only the last line, which we take the first column from.

Slides and videos

Links

Bash - tools

External links

Source code

END_INCLUSION of MoreBash:Network_Tools

Retrieved from "http://wiki.juneday.se/mediawiki/index.php?title=ITIC:Network_tools_and_commands&oldid=13515"