Chap 2. Foundational Knowledge in 25 Pages or Less

The Command-Line Interface

Command Line - Terminal - Shell

terminal (terminal simulator) : a program that provides a textual user interface : - reads characters from the keyboard & displays them on the screen : - handle escape code (aka escape sequence), e.g. Ctrl + C : e.g. : - OS’s defaults: MacOS Terminal, GNOME Terminal, Command Prompt (Windows) : - 3rd-party: iTerm2, Windows Terminal : - New generation that use GPU: Alacritty, Kitty, Warp

shell : a program that run inside the terminal: : - acts as a command interpreter : - handles input/outputs (IO) via streams (stdin, stdout, stderr) : - provides: variables (e.g. $SHELL), built-in commands (e.g. ls, cd) : e.g. : - sh: Bourne shell : - bash: Bourne again shell : - Modern, human-friendly shells: zsh, fish : - Windows: MS-DOS, PowerShell

Why Do I Need the Command Line?

• Speed (with enough practice)
• The CLI tools can do things GUI tool can’t do.

Getting Started with the Command Line

Shell’s Streams

When performing I/O, a system call refer to an open file using a file descriptor - a (usually small) nonnegative integer.

Open files are regular files or pipes, FIFOs, sockets, terminals, devices.

Each process has its own set of file descriptors:

• Including 3 standard file descriptions:

  File descriptor	Purpose	stdio stream
  0	Standard Input	stdin
  1	Standard Output	stdout
  2	Standard Error	stderr

• These 3 standard file descriptor are opened on the process’s behalf (by the shell), before the process is started.

  

Protocols: A High-Level Overview

protocol (communication protocol) : When multiple parties communicate, a protocol is an agreement on how each party will act

Computers are built around conformance to protocols to ensure interoperability.

The internet are built on common, shared & open protocols:

• Internet Protocol suite (TCP/IP)
• OSI model

Protocol Layers

• TCP/IP protocol: 4 layers
• OSI model: 7 layers

When devices communicate over a network, the data is passed down the layers

• from an application e.g. a web browser
• eventually onto a physical medium e.g. wired Ethernet or radio signals for WiFi.

The receiver then passes the data up through the layers to the corresponding application on the receiving side.

Two Protocols Plus Another

TCP (Transmission Control Protocol) : connection-oriented : applications receive packets in an orderly sequence

UDP (User Datagram Protocol) : connection-less : applications receive packets in an un-orderly sequence : - ensuring that packets have arrived : - asking for retransmission when packets have not arrived

ICMP (Internet Control Message Protocol) : Used by ping command : - The ping command sends Echo Request, (ICMP message type 8) : - The device under test responds with Echo Reply, (ICMP message type 0)

Basic Internet Protocols

DNS

Domain name system

DNS (domain name system) : What? A naming system for computers, resources… in the Internet : Analogy? Phone book of the internet : - Name -> Phone Number : - Domain Name (aka hostname) -> IP Address : How? Delegates the responsibility of assigning domain names & mapping those names to Internet resources by designating authoritative name servers for each domain

The Internet maintains two principal namespaces:

• the domain name space (aka domain name hierarchy)
• the IP address spaces (IPv4, IPv6)

The Domain Name System

• maintains the domain name space
• provides translation services between the domain name space and the address spaces

Domain name space, DNS zones

The domain name space consists of a tree data structure:

• Each node (or leaf) in the tree has a label and zero or more resource records (RR), which hold information associated with the domain name.

• The tree sub-divides into zones:

  • A DNS zone may consist of as many domains and subdomains as the zone manager chooses.

    • Administrative responsibility for any zone may be divided by creating additional zones.

      Authority over the new zone is said to be delegated to a designated name server. The parent zone ceases to be authoritative for the new zone.

  • The tree begins at the root zone (a single dot .)

  • The second level of zones:

    • Top-level domain (TLDs):
      • Generic TLDs (gTLDs): com, org, net, edu…
      • Country-code TLDs (ccTLDs): us, uk, vn…

DNS servers can load zone data from:

• zone data file (aka master files)

• other DNS servers via a mechanism called a zone transfer

Domain name

The domain name itself consists of the label, concatenated with the label of its parent nodes on the right, separated by a dot (.).

• e.g. example.com

  • example: DNS node’s label
  • .: separator
  • com: parent node’s name

• The right-most label conveys the top-level domain (TLD)

  e.g. the domain name example.com belongs to the TLD com.

• The hierarchy of domains descends from right to left; each label to the left specifies a subdivision, or subdomain of the domain to the right.

  e.g.

  • the label example is a subdomain of the com domain
  • the label www is a subdomain of example.com domain

DNS limitations

• The domain name space (hierarchy) may have up to 127 levels.
• A label may contain 63 characters.
• The full domain name may contain 253 characters (other 2 octets are for the length of the domain name)
• The character may be in a subset of ASCII character set - LDH rule:
  • Letter: a-z, A-Z (The domain names are case-insensitive interpreted)
  • Digit: 0-9
  • Hyphen: -

Domain registration

Each TLD is delegated to a registrar to managed that DNS zone.

Domain (within a TLD) is registered within the desired TLD, according to the rules ot the registrar (of that TLD).

When a domain name is registered, control of the sub-domain within that domain is delegated to the registrant (the person/organization who registered the domain with the register of that TLD)

• The registrant needs to provide at least 2 authoritative name server, which response to DNS query for sub-domain of that domain name.

DNS Resource Records

Data in DNS is stored in units of resource records (RR).

Resource records come in different classes and types.

• The classes were intended to allow DNS to function as the naming service for different kinds of networks.

  In practice DNS is used only on the internet and TCP/IP networks, so just one class, IN for internet, is used.

• The types of resource records in the IN class specify both the format and application of the data stored

  • Some of the most common resource record types in the IN class

    RR Type	Description	Function
    A	IPv4 Address	Maps a domain name to a single IPv4 address
    AAAA	IPv6 Address	Maps a domain name to a single IPv6 address
    CNAME	Canonical Name (Alias)	Maps a domain name (the alias) to another domain name (the canonical name)
    MX	Mail Exchanger	Names a mail exchanger (mail server) for an email destination
    NS	Name Server	Names a name server (or DNS server) for a zone
    PTR	Pointer	Maps an IP address back to a domain name
    SOA	Start of Authority	Provides parameters for a zone

  • Overview of all active DNS record types

    

Resource Records are in master file format which including 5 fields [NAME] [TTL] [CLASS] TYPE RDATA.

Hostname resolution

Resolution

For a host - with a valid IP address - to communicate on a network, the host needs to translate the friendly name (domain name) to an IP address.

• First, it examines a locally file called hosts

  e.g.

  • Linux/Mac: /etc/hosts

    ##
    # Host Database
    #
    # localhost is used to configure the loopback interface
    # when the system is booting.  Do not change this entry.
    ##
    127.0.0.1       localhost
    255.255.255.255 broadcasthost
    ::1             localhost
    

  • Windows: %SystemRoot%\System32\Drivers\etc\hosts

• If the destination computer is not defined in the hosts file, the DNS is queried next (by DNS resolvers).

DNS resolution (by DNS resolvers)

Domain name resolvers determine the domain name servers responsible for the domain name in question by a sequence of queries starting with the right-most (top-level) domain label.

• The resolution process starts with a query to one of the root servers.

• In typical operation, the root servers do not answer directly, but respond with a referral to more authoritative servers,

  e.g., a query for “www.wikipedia.org” is referred to the org servers.

• The resolver now queries the servers referred to, and iteratively repeats this process until it receives an authoritative answer.

The diagram illustrates this process for the host that is named by the fully qualified domain name “www.wikipedia.org”.

DNS cache servers - TTL

To improve efficiency, reduce DNS traffic across the Internet, and increase performance in end-user applications, the Domain Name System supports DNS cache servers which store DNS query results for a period of time determined in the configuration (time-to-live - TTL) of the domain name record in question.

The DNS cache servers is also known as recursive server

• Typically, such caching DNS servers also implement the recursive algorithm necessary to resolve a given name starting with the DNS root through to the authoritative name servers of the queried domain.
• Typically, Internet service providers (ISPs) provide recursive and caching name servers for their customers.
• In addition,
  • many home networking routers implement DNS caches and recursion to improve efficiency in the local network.
  • you can maintain your own DNS resolvers, e.g. Bind9, CoreDNS, PiHole, Adguard Home
  • you can use a DNS services (public DNS resolvers), e.g. Google Public DNS (8.8.8.8), CloudFlare DNS (1.1.1.1)

Client DNS lookup

Users generally do not communicate directly with a DNS resolver. Instead DNS resolution takes place transparently in applications such as web browsers, e-mail clients, and other Internet applications.

• When an application makes a request that requires a domain name lookup, such programs send a resolution request to the DNS resolver in the local operating system, which in turn handles the communications required.

Split-horizon DNS

A split-horizon DNS (aka split DNS, split-view DNS) provides different set of DNS records, selected by the source of the DNS request.

Start of Authority and time-to-live

Each DNS zone contains a Start of Authority (SOA) record that defines domain metadata.

• The serial number and the refresh, retry, and expiration intervals are all related to zone transfers

• The negative-caching TTL specifies to other DNS servers how long they can cache negative responses from this zone’s authoritative DNS servers.

HTTP

HTTP (HyperText Transfer Protocol) : the language of the web : used for transferring web pages & remote programmatic access between services

HTTP is a stateless protocol

• When the client - e.g. your computer - makes a request to a web server that speaks HTTP, the server does not remember one request to the next; each request is new

A HTTP message consists of:

• control data
• headers
• content (aka body, payload)
• trailers

For more information, see

• RFC 2616: HTTP/1.1 - HTTP Message
• RFC 9110: HTTP Semantics - Message Abstraction
• RFC 9113: HTTP/2 - HTTP Control Data

HTTPS

HTTP uses TCP connections as the transport layer.

• HTTP is the top layer in a “protocol stack” of “HTTP over TCP over IP”.

• When HTTP wants to transmit a message, it streams the contents of the message data, in order, through an open TCP connection.

  • TCP takes the stream of data, chops up the data stream into chunks called TCP segments, and transports the segments across the Internet inside envelopes called IP packets.
  • Each TCP segment is carried by an IP packet from one IP address to another IP address.

HTTPS, a secure variant of HTTP, inserts a cryptographic encryption layer (called TLS or SSL) between HTTP and TCP.

• In additional to the TCP connection (of the transport layer), HTTPS also has a TLS connection to encrypt the data.

Other protocols

File Transfer Protocol(s) (FTP)

• FTP, SFTP (Secure FTP)
• FTPS (FTP over SSL)
• SCP (Secure Copy Protocol)

Secure Shell (SSH)

A DevSecOps operation will use SSH to configure servers remotely.

Simple Network Management Protocol (SNMP)

The monitoring infrastructure that can appear within a DevSecOps team might use SNMP.

Data Security: Confidentiality, Integrity, & Availability (CIA)

Confidentiality : ensures that only specific, authorized individuals can view, modify, or share this guarded information. : applies to both data in transit (traversing a network) & data at rest (in storage)

Integrity : ensures that data is maintained in a known-good state

Availability : ensures that business-related data is available to the organization, partners, or end-users whenever & wherever

Development Overview for Scripting

Shell Commands: Built-in & External

built-in commands : built-in the shell : executed directly (by the shell), without invoking another process : e.g. pwd, cd

external commands : exist regardless of the shell : invoked in a separate environment (another process) that cannot affect the shell’s execution environment. : e.g. ls, cp

$ type -a cd
cd is a shell builtin
cd is /usr/bin/cd

For more information, see

• GNU Bash manual
  • Bash Reference Manual
    • Bash Reference Manual - Shell Builtin Commands
• The Z Shell Manual
  • zsh - Shell Builtin Commands

Basic Bash Constructs: Variables, Data, and Data Types

Making Decisions with Conditionals

if statement

In Bash, if statement “test” if a command succeeds

• e.g.

  if [ -z "$BASH_VERSION" ]; then
    echo "Bash is required to interpret this script."
  fi
  

  • [ EXPRESSION ] Exit with the status determined by EXPRESSION.
  • -z "$BASH_VERSION" Test if the length of STRING is zero

name="Bob"
if [[ $name == "" ]]; then
  echo "Hello, world"
else
  echo "Hello, $name"
fi

• Syntax

  If	If - else	If - elif - else
  if TEST_COMMANDS; then CONSEQUENT_COMMANDS; fi	if TEST_COMMANDS; then CONSEQUENT_COMMANDS; else MORE_CONSEQUENTS; fi	if TEST_COMMANDS; then CONSEQUENT_COMMANDS; [elif MORE_TEST_COMMANDS; then MORE_CONSEQUENTS;] else ALTERNATE_CONSEQUENTS; fi
  1 - If TEST_COMMANDS succeeds, CONSEQUENT_COMMANDS list is executed
  		(When elifs are present) 2 - If TEST_COMMAND fails, each elif list is executed in turn, if MORE_TEST_COMMANDS succeeds, the corresponding MORE_CONSEQUENTS is executed and the command completes.
  	(When else is present) 3a - If the final command - in TEST_COMMANDS - fails, ALTERNATE_CONSEQUENTS is executed.	(When else is present) 3b - If the final command - in the final elif clause - fails, ALTERNATE_CONSEQUENTS is executed.
  e.g. if true; then # Do something fi	e.g. if false; then # Unreachable code else # Do something fi	e.g. OS="$(uname)" if [[ "${OS}" == "Linux" ]]; then echo "this script is running on Linux" elif [[ "${OS}" == "Darwin" ]]; then echo "his script is running on MacOS" else echo "unsupported OS: only support on macOS and Linux" fi

• Use case

  In other programming language, if works a little different then in Bash.

  • In Go, if specify the conditional execution of two branches according to the value of a boolean expression.

  • In Bash, if specify the conditional execution of two branches according to the exit status² of the test commands.

    The test commands can be:

    • A single or a list of commands³
    • A built-in or an external command⁴
    • [ (test command): a little bit tricky/unpredictable
    • [[ (extended test command): works like other programming language

[ and test command

• The [ - left bracket - is a builtin command, synonym for test - builtin command.

• [ or test:

  • considers its arguments as comparison expressions or file tests
  • returns an exit status corresponding to the result of the comparison/test:
    • if the comparison/test is true, [ succeeds (returns 0)
    • if the comparison/test is false, [ fails (returns non-zero status)

[[ - extended test command

• The [[ … ]] is extended test command, which performs comparisons in a manner more familiar to programmers from other languages.

• Using the [[ … ]] test construct, rather than [ … ] can prevent many logic errors in scripts.

  • No word splitting or filename expansion takes place between [[ and ]]
  • The &&, ||, <, and > operators work within a [[ ]] test, despite giving an error within a [ ] construct.
  • Arithmetic evaluation of octal / hexadecimal constants takes place automatically within a [[ ]] construct.

Conditional expressions

• Bash conditional expressions

  	Category	Conditional Expressions	Meaning	Note
  Unary	Test a file	-e file	True if file [e]xists
  		-d file	True if file exists and is a [d]irectory.
  		-f file	True if file exists and is a regular [f]ile.
  		-r file	True if file exists and is [r]eadable.
  		-w file	True if file exists and is [w]ritable
  		-x file	True if file exists and is e[x]ecutable.
  	Test a string	-n string or string	True if the length of string is [n]on-zero.
  		-z string	True if the length of string is [z]ero.
  Binary	Compare strings	string1 == string2	True if the strings are equal.
  		string1 != string2	True if the strings are not equal.
  		string1 < string2	True if string1 sorts before string2 lexicographically	test uses ASCII ordering
  		string1 > string2	True if string1 sorts after string2 lexicographically	[[… ]] use the current locale
  	Compare numbers	arg1 OP arg2	true if arg1 is OP to arg2, respectively.

  For the full list of conditional expressions, see Bash Reference Manual - Bash Conditional Expressions

• Bash conditional expression supports 6 arithmetic operator

  -eq	-ne	-lt	-le	-gt	-ge
  equal to	not equal to	less than	less than or equal to	greater than	greater than or equal to

Looping

A loop is a block of code that iterates a list of commands as long as the loop control condition is true.

Bash supports 3 types of loops: for, while, until.

for

for loop run a set of commands for each item in a list.

• Syntax

  for <name> in <words>
  do
    <commands>
  done
  

• Example

  for item in ~/effective-shell/* # Loop through all items in a folder
  do
    echo "Found: $item"           # Print the item's full path to the screen
  done
  

  Output

  Found: /home/dwmkerr/effective-shell/data
  Found: /home/dwmkerr/effective-shell/docs
  Found: /home/dwmkerr/effective-shell/logs
  Found: /home/dwmkerr/effective-shell/pictures
  Found: /home/dwmkerr/effective-shell/programs
  Found: /home/dwmkerr/effective-shell/quotes
  Found: /home/dwmkerr/effective-shell/scripts
  Found: /home/dwmkerr/effective-shell/templates
  Found: /home/dwmkerr/effective-shell/text
  Found: /home/dwmkerr/effective-shell/websites
  

See, Effective Shell - For loop

while

A while loop iterates through a block of code only when a condition remains true.

If the condition isn’t true to begin with, the while loop will never execute the code within.

See, Effective Shell - While loop

until

The until loop is like the while loop except that the code will only be executed when a condition remains false.

If the condition is true to begin with, then the code within the until loop will never execute

See, Effective Shell - Until loop

Arrays

• Arrays are variables that can store multiple values.

• An array is created by using the equals symbol and putting the array values in parenthesis.

  days=("Monday" "Tuesday" "Wednesday" "Thursday" "Friday" "Saturday" "Sunday")
  

• Once you have defined your array you can retrieve an element at a given index by using the square bracket notation shown below:

  echo "The first day is: ${days[0]}" # Monday
  echo "The last day is: ${days[6]}"  # Sunday
  

See Effective Shell - Arrays

Summary