FANDOM


Basic

# shutdown -h now   # shutdown right now

# # shutdown after 5 minutes
# shutdown -h +5 "Server is going down for upgrade. Please save your work."

# # restart after 5 minutes
# shutdown -r +5 "Server will restart in 5 minutes. Please save your work."

# shutdown -c   # cancel restart

# reboot -p   # shutdown right now

# reboot   # restart

Kernel

File/Directory Description Remarks
/proc/sys/
/proc/sys/fs
/proc/sys/kernel
/proc/sys/vm
/proc/sys/dev
/proc/sys/net
Parameter Description Remarks
fs.file-max the maximum number of file-handles that the Linux kernel will allocate
fs.file-nr the number of allocated file handles, the number of free(allocated but unused) file handles, and the maximum number of file handles
fs.nr_open the maximum number of file-handles a process can allocate default: 1048576 (1024*1024)
net.ipv6.conf.all.disable_ipv6 0|1
net.ipv6.conf.default.disable_ipv6 0|1
net.ipv6.conf.lo.disable_ipv6 0|1
File Item Description Remarks
/proc/stat ctxt the total number of context switches across all CPUs
btime the time at which the system booted, in seconds since the Unix epoch
processes the number of processes and threads created, which includes (but is not limited to) those created by calls to the fork() and clone() system calls
procs_running the number of processes currently running on CPUs
procs_blocked the number of processes currently blocked, waiting for I/O to complete

sysctl

Options
Option Description Remarks
-p, -f, --load Read values from file

Environment Variables

Variable Description Remarks
IFS determines how bash recognizes fields, or word boundaries, when it interprets character strings Internal Field Separator
LC_ALL Determines the values for all locale categories. The value of the LC_ALL environment variable has precedence over any of the other environment variables starting with LC_ and the LANG environment variable.
LANG Determines the locale category for native language, local customs and coded character set in the absence of the LC_ALL and other LC_* environment variables.
TZ Timezone information
http_proxy ftp, wget, curl, ssh, apt-get, yum
https_proxy
ftp_proxy

Locale

User Management

User Management Files

  • Entry format
File Entry Format Remarks
/etc/passwd login_name : password : uid : gid : comment : home_dir : login_shell
/etc/shadow login_name : password : lastchanged : minimum : maximum : warn : inactive : expire : flag
/etc/group groupname : password : gid : user-list
  • Password field of shadow file
Value Description Remarks
$id$salt$hashed the printable form of a password hash as produced by crypt
empty string No password
! the account is password locked, user will be unable to log in via password authentication but other methods (e.g. ssh key) may be still allowed
* the account is locked, user will be unable to log in via password authentication but other methods (e.g. ssh key) may be still allowed
  • Algorithm($id) identifier of shadowed password
Value Algorithm Remarks
$1$ MD5
$5$ SHA-256
$6$ SHA-512

System Groups

Group Description Remark
adm Group adm is used for system monitoring tasks. Members of this group can read many log files in /var/log, and can use xconsole. Historically, /var/log was /usr/adm (and later /var/adm), thus the name of the group.
admin The admin group is used to grant sudo access on ubuntu 11.10 and earlier. It's still found on 12.04 for backwards compatibility.

Memory Management

Disk and Filesystem Management

Fundamentals

  • Logical volume management
    • Physical Volume Group (PVG) = ∑ Physical Volume
    • Physical Volume (PV) = ∑ Physical Extent
    • Logical Extentn = Physical Extentm
    • Logical Volume (LV) = ∑ Logical Extent (in a single PVG)
Filesystem Description Mount Point Remarks
proc Kernel and process information virtual filesystem /proc /proc/sys
sys Kernel and system information virtual filesystem /sys
etc Host-specific system configuration /etc

procfs (proc Filesystem)

  • a special filesystem in Unix-like operating systems that presents information about processes and other system information in a hierarchical file-like structure, providing a more convenient and standardized method for dynamically accessing process data held in the kernel than traditional tracing methods or direct access to kernel memory
  • CentOS / /proc/sys/ : provides information about the system and allows the system administrator to immediately enable and disable kernel features.
File/Directory Description Remarks
/proc/stat/ Overall/various statistics about the system, such as the number of page faults since the system was booted
/proc/sys/
/proc/cpuinfo a collection of CPU and system architecture dependent items processor, cpu_family, model_name
/proc/meminfo statistics about memory usage on the system. MemTotal, MemFree, MemAvailable, Buffers, Cached
/proc/{pid}/
/proc/{pid}/cmdline a file containing full command-line for this process
/proc/{pid}/cwd a symbolic link to the current working directory of this process
/proc/{pid}/exe a symbolic link to the actual executable file for this process
/porc/{pid}/environ a file containing environment variables used by this process
/proc/{pid}/status a file containing basic information for this process including its run state and memory usage
/proc/{pid}/limits
/proc/{pid}/fd/ a directory containing symbolic links for the all the open file descriptors by this process
/proc/{pid}/task/ a directory containing hard links for the all the tasks that have been started by this process

sysfs (sys Filesystem)

  • a pseudo file system provided by the Linux kernel that exports information about various kernel subsystems, hardware devices, and associated device drivers from the kernel's device model to user space through virtual files.

etc Filesystem

File/Directory Description Remarks
/etc/sysctl.conf
/etc/sysctl.d/
/etc/environment Contains variables specifying the basic environment for all processes.
/etc/default/locale
/etc/issue contains a message or system identification to be printed before the login prompt of a telnet session
/etc/motd displayed by login after a successful login but just before it executes the login shell Message Of The Day
/etc/network/interfaces Network interface configuration for ifup and ifdown
/etc/resolv.conf A set of routines in the C library that provide access to the Internet Domain Name System (DNS)
/etc/nsswitch.conf The Name Service Switch (NSS) configuration file used by the GNU C Library to determine the sources from which to obtain name-service information in a range of categories, and in what order. Name Service Switch
/etc/passwd User account information
/etc/shadow Secure user account information
/etc/group Group account information
/etc/gshadow Secure group account information
/etc/login.defs Shadow password suite configuration
/etc/default/useradd Default values for account creation
/etc/shells Contains list of available shells
/etc/ntp.conf Network Time Protocol (NTP) daemon configuration file. Official NTP Documentation
/etc/default/ntp NTPD_OPTS='-g'
/etc/default/telegraf Contains environment variables for Telegraf service of Systemd

var Filesystem

Category File/Directory Description Remarks
/var/run/reboot-required.pkgs
dpkg /var/lib/dpkg/info/ contains scripts (pre-install, post-install, pre-remove, post-rmove) of debian/APT packages
/var/log/dpkg.log dpkg.log.1, dpkg.log.2.gz
apt /var/log/apt/history.log
telegraf /var/log/telegraf/telegraf.log

Commands

command description remarks
mount mount a filesystem
tune2fs adjust tunable filesystem parameters on ext2/ext3/ext4 filesystems
dumpe2fs dump ext2/ext3/ext4 filesystem information

Filesystem Types

Filesystem Description Remarks
SquashFS a compressed read-only filesystem for Linux
Ext4 Filesystem

Readings

  • tmpfs
    • a common name for a temporary file storage facility on many Unix-like operating systems

Service Management

Init system Released when/with Remarks
SysV init 1983
Upstart 2006, Ubuntu 6.10
systemd 2011, Fedora 15

init

service
Command Syntax Remarks
List all services sudo service --status-all
Start a service sudo service script start service sshd stop
Check a status of a service sudo service script status service sshd status
Stop a service sudo service script stop service sshd stop
Restart a service sudo service script restart service sshd restart
List all scripts ls /etc/init.d/

Upstart

systemd

References
Manpage Description Remarks
systemd.conf System and session service manager configuration
systemd.unit Unit configuration
systemd.service Service unit configuration
systemd.exec Execution environment configuration
  • If systemd encounters an unknown option, it will write a warning log message but continue loading the unit. If an option or section name is prefixed with X-, it is ignored completely by systemd.
  • Along with a unit file foo.service, a "drop-in" directory foo.service.d/ may exist. All files with the suffix ".conf" from this directory will be parsed after the unit file itself is parsed. This is useful to alter or add configuration settings for a unit, without having to modify unit files. Drop-in files must contain appropriate section headers. For instantiated units, this logic will first look for the instance ".d/" subdirectory (e.g. "foo@bar.service.d/") and read its ".conf" files, followed by the template ".d/" subdirectory (e.g. "foo@.service.d/") and the ".conf" files there. Moreover for units names containing dashes ("-"), the set of directories generated by truncating the unit name after all dashes is searched too. Specifically, for a unit name foo-bar-baz.service not only the regular drop-in directory foo-bar-baz.service.d/ is searched but also both foo-bar-.service.d/ and foo-.service.d/. This is useful for defining common drop-ins for a set of related units, whose names begin with a common prefix.
  • In addition to /etc/systemd/system, the drop-in ".d/" directories for system services can be placed in /usr/lib/systemd/system or /run/systemd/system directories. Drop-in files in /etc take precedence over those in /run which in turn take precedence over those in /usr/lib. Drop-in files under any of these directories take precedence over unit files wherever located. Multiple drop-in files with different names are applied in lexicographic order, regardless of which of the directories they reside in.
    • /etc/systemd/system/foo.service.d > /run/systemd/system/foo.service.d > /usr/lib/systemd/system/foo.service.d
Options
Section Option Description Remarks
Shared WorkingDirectory Sets the working directory for executed processes
User, Group Set the UNIX user or group that the processes are executed as, respectively
SupplementaryGroups Sets the supplementary Unix groups the processes are executed as.
EnvironmentFile Reads the environment variables from a text file
LimitNOFILE Set soft and hard limits on number of file descriptors ulimit -n
Service ExecStart Commands with their arguments that are executed when this service is started
Restart Configures whether the service shall be restarted when the service process exits, is killed, or a timeout is reached. Revival process
Manage DefaultLimitNOFILE
State
  • systemctl --state=help
Category Enum Description
Load States loaded
not-found
error
merged
masked
Active States active
reloading
inactive
failed
activating
deactivating
Specifiers
Specifier Title Description
%i Instance name For instantiated units this is the string between the first "@" character and the type suffix. Empty for non-instantiated units.
%I Unescaped instance name Same as %i, but with escaping undone.
Directory Sandboxing
Location For System for users Environment Variable
RuntimeDirectory= /run $XDG_RUNTIME_DIR $RUNTIME_DIRECTORY
StateDirectory= /var/lib $XDG_STATE_HOME $STATE_DIRECTORY
CacheDirectory= /var/cache $XDG_CACHE_HOME $CACHE_DIRECTORY
LogsDirectory= /var/log $XDG_CONFIG_HOME/log $LOGS_DIRECTORY
ConfigurationDirectory= /etc $XDG_CONFIG_HOME $CONFIGURATION_DIRECTORY
Directories
Directory Description Remarks
/lib/systemd
/lib/systemd/system
/usr/lib/systemd
/usr/lib/systemd/system
/usr/lib/systemd/user
/etc/systemd
/etc/systemd/system
/etc/systemd/system/multi-user.target.wants
/etc/systemd/user
Readings
systemctl
  • Control the systemd system and service manager
Command Description Flags Remarks
systemctl list-units List units that systemd currently has in memory
systemctl list-unit-files List unit files installed on the system, in combination with their enablement state
systemctl enable Enable one or more units or unit instances
systemctl disable Disables one or more units
systemctl start Start (activate) one or more units specified on the command line
systemctl stop Stop (deactivate) one or more units specified on the command line
systemctl restart Stop and then start one or more units specified on the command line
systemctl reload Asks all units listed on the command line to reload their configuration
systemctl reload-or-restart Reload one or more units if they support it. If not, stop and then start them instead.
systemctl mask Mask one or more units, as specified on the command line, making it impossible to start them.
systemctl unmask
systemctl status Show terse runtime status information about one or more units, followed by most recent log data from the journal
systemctl show Show properties of one or more units, jobs, or the manager itself
systemctl cat Show backing files of one or more units
systemctl is-enabled Checks whether any of the specified unit files are enabled (as with enable)
systemctl is-active Check whether any of the specified units are active (i.e. running)
systemctl list-dependencies --all, --reverse
systemctl edit /etc/systemd/system/service_name.d/override.conf
Tips and Tricks
Check whether a service support reload or not and how it is supported

Check CanReload and ExecReload properties using systemctl show command.

$ systemctl show telegraf.service | grep -i reload
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
CanReload=yes
NeedDaemonReload=no

Misc

  • inetd : a super-server daemon on many Unix systems that provides Internet services
  • xinetd : a secure replacement for inetd

Process Management

Signal Number Description Remarks
SIGHUP 1 Usually means that the controlling pseudo or virtual terminal has been closed
SIGKILL 9 Sent to a process to cause it to terminate immediately

Network Management

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

LAN

Monitoring & Diagnosis

Process

CPU

Memory

Disk IO

Security

Virtualization

Time Synchronization

Programs

Command Description Remarks
ntpd an operating system daemon that synchronizes the system clock to remote NTP time servers or local reference clocks.
ntpstat Show network time synchronisation status.
ntpq Standard NTP query program
ntpdate Sets the local date and time by polling the Network Time Protocol (NTP) server(s). -q : Query only - don't set the clock.
sntp a Simple Network Time Protocol (SNTP) client that can be used to query a Network Time Protocol (NTP) server and display the time offset of the system clock relative to the server clock.

NTP Daemon(nptd)

Readings

Tips and Tricks

Configuring two ntpd servers as upstream and downstream
"restrict" applies to both servers and clients, so a configuration that might be intended to block requests from certain clients could also end up blocking replies from your own upstream servers.
  • Upstream server config at 172.16.1.25
  1. # /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
  2.  
  3. driftfile /var/lib/ntp/ntp.drift
  4.  
  5. #statsdir /var/log/ntpstats/
  6. statistics loopstats peerstats clockstats
  7. filegen loopstats file loopstats type day enable
  8. filegen peerstats file peerstats type day enable
  9. filegen clockstats file clockstats type day enable
  10.  
  11. pool 0.ubuntu.pool.ntp.org iburst
  12. pool 1.ubuntu.pool.ntp.org iburst
  13. pool 2.ubuntu.pool.ntp.org iburst
  14. pool 3.ubuntu.pool.ntp.org iburst
  15. pool ntp.ubuntu.com
  16.  
  17. # Note that "restrict" applies to both servers and clients, so a configuration
  18. # that might be intended to block requests from certain clients could also end
  19. # up blocking replies from your own upstream servers.
  20. discard average 3 minimum 1 monitor 1000
  21. restrict -4 default ignore
  22. restrict -6 default ignore
  23.  
  24. restrict 127.0.0.1
  25. restrict ::1
  26.  
  27. restrict source notrap nomodify noquery
  28.  
  29. restrict 172.16.31.0 mask 255.255.255.224 nomodify notrap nopeer noquery limited
  30.  
  31. server time.service.networklayer.com prefer
  • Downstream server config at 172.16.31.15
  1. # /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
  2.  
  3. driftfile /var/lib/ntp/ntp.drift
  4.  
  5. #statsdir /var/log/ntpstats/
  6. statistics loopstats peerstats clockstats
  7. filegen loopstats file loopstats type day enable
  8. filegen peerstats file peerstats type day enable
  9. filegen clockstats file clockstats type day enable
  10.  
  11. discard average 3 minimum 1 monitor 1000
  12. restrict -4 default ignore
  13. restrict -6 default ignore
  14.  
  15. restrict 127.0.0.1
  16. restrict ::1
  17.  
  18. server 172.16.1.25 iburst prefer
  19. restrict 172.16.1.25 notrap nopeer limited
Check the status of local nptd

Use ntpstat or ntpq like the followings

$ ntpstat

$ ntpq -np

Shell

Bash

When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The --noprofile option may be used when the shell is started to inhibit this behavior.

When a login shell exits, bash reads and executes commands from the files ~/.bash_logout and /etc/bash.bash_logout, if the files exists.

When an interactive shell that is not a login shell is started, bash reads and executes commands from ~/.bashrc, if that file exists. This may be inhibited by using the --norc option. The --rcfile file option will force bash to read and execute commands from file instead of ~/.bashrc.

Bash programming general

Special Parameters

Parameter Description Remarks
$@ Expands to the positional parameters, starting from one When the expansion occurs within double quotes, each parameter expands to a separate word. That is, "$@" is equivalent to "$1" "$2" ….
$* Expands to the positional parameters, starting from one.
$# Expands to the number of positional parameters in decimal
$0 Expands to the name of the shell or shell script. ./scripts/ganache-cli-start.sh
$? Expands to the exit status of the most recently executed foreground pipeline.
$$ Expands to the process ID of the shell.
$! Expands to the process ID of the job most recently placed into the background.
!$ Designates the last argument of the preceding command.
!! Refer to the previous command. (synonym for '!-1') sudo !!

Shell expansions

Expansion Syntax Description Remarks
Brace Expansion {str1,str2,str3, ...}, {x..y[..incr]} $ mkdir /opt/lego/{bin,conf,lib,docs,log,tmp}
Tilde Expansion ~, ~dir(/dir)*, ~/user(/dir)*
Shell Parameter Expansion ${parameter:-word}, ${parameter:=word}, ${parameter:?word}, ${parameter:+word}, ${parameter##word}, ${parameter%%word}, ...
Command Substitution $(command), `command` `echo $CODE` | base64`
$(echo $CODE | base64)
Arithmetic Expansion $(( expression ))
Filename Expansion Pattern syntax in file name expansion is different from general regex patterns.
History Expansion !!, !n, !-n(!-1, !-2, !-3 ...), !!:$, !$, !0
Shell Parameter Expansion
Expression Meaning Remarks
${parameter:-word} If parameter is unset or null, the expansion of word is substituted. Otherwise, the value of parameter is substituted.
${parameter/pattern/string} Parameter is expanded and the longest match of pattern against its value is replaced with string.

If pattern begins with ‘/’, all matches of pattern are replaced with string. Normally only the first match is replaced. If pattern begins with ‘#’, it must match at the beginning of the expanded value of parameter. If pattern begins with ‘%’, it must match at the end of the expanded value of parameter. If string is null, matches of pattern are deleted and the / following pattern may be omitted

${parameter#pattern} If the pattern matches the beginning of the expanded value of parameter, then the result of the expansion is the expanded value of parameter with the shortest matching pattern deleted.
${parameter%pattern} If the pattern matches a trailing portion of the expanded value of parameter, then the result of the expansion is the value of parameter with the shortest matching deleted.
${parameter:offset(:length)} Expands to up to length characters of the value of parameter starting at the character specified by offset.
${#parameter} The length in characters of the expanded value of parameter is substituted.
${#@}, ${#*} the number of positional parameters
${#arr[@]}, ${#arr[*]} the number of elements in the array
${!arr[@]}, ${!arr[*]} Expands to the list of array indices (keys) assigned in array named arr.
Filename Expansion
Basic Globbing
Wildcard Title Description Remark
* Any string matches any number of any characters including none
? Any single character matches any single character
[c1c2c3...] Any one of the enclosed characters matches one character given in the bracket
[s-e] matches one character from the (locale-dependent) range given in the bracket
[!c1c2c3...] matches one character that is not given in the bracket
[!s-e] matches one character that is not from the range given in the bracket
Extended Globbing on Bash
Indirect Expansion

Operators and expressions

String

Arithmetic

Array

declare -a fruits=(banana apple orange mango)

# ${arr[@]} : list of array
for f in ${fruits[@]}; do
  echo $f
done

# ${#arr[@]} : length of array
for (( i = 0; i < ${#fruits[@]}; i++ )); do
  echo ${fruits[$i]}
done

# ${!arr[@]} : list of indices in array
for i in ${!fruits[@]}; do
  echo ${fruits[$i]}
done

Redirections

Value Name <stdio.h> remarks
0 standard input stdin
1 standard output stdout
2 standard error stderr
find ... > found.txt              # write down only output
find ... > out_and_err.txt 2>&1   # write down both output and error
find ... 2> /dev/null             # hide only error
find ... > /dev/null 2>&1         # hide both output and error

If statement

For statement

Function

getopts

getopt

Keyboard Shortcuts

Category Shortcut Description Remarks
Working With Processes Ctrl+C Interrupt (kill) the current foreground process running in in the terminal. SIGINT
Ctrl+Z Suspend the current foreground process running in bash. SIGTSTP
Cutting and Pasting Ctrl+U Cut the part of the line before the cursor, adding it to the clipboard.
Working With Command History Ctrl+R Recall the last command matching the characters you provide.
Ctrl+O Run a command you found with Ctrl+R.
Ctrl+G Leave history searching mode without running a command.

X11

GNOME

SELinux

Commands

command description remarks
sestatus SELinux status tool
setenforce modify the mode SELinux is running in
seinfo allows the user to query the components of a SELinux policy e.g.) seinfo -t, seinfo -r, seinfo -u -x
semanage SELinux Policy Management tool semanage boolean, semanage user, semanage login, semanage module, semanage port, semanage interface, semanage node, semanage fcontext
matchpathcon get the default SELinux security context for the specified path from the file contexts configuration
restorecon restore file(s) default SELinux security contexts
chcat change file SELinux security category
task command-line remarks
List all permissive types $ sudo semanage permissive -l
List all modules $ sudo semodule -l
List all module binaries of default policy $ ls /usr/share/linux/default
View log file $ sudo tail -f -n 300 /var/log/audit/audit.log
List all SELinux users $ sudo semanage user -l
List all login mappings $ sudo semanage login -l
List all roles $ seinfo -r
List all types $ seinfo -t
List types that are accessible for a role $ seinfo -rdbadm_r -x
List all security categories $ chcat -L

Files and Directories

file/directory description remarks
/etc/selinux/config
/etc/selinux/semanage.conf
/etc/selinux/default/seusers
/etc/selinux/default/logins/
/etc/selinux/default/contexts/default_contexts used by SELinux-aware applications that need to set a security context for user processes (generally the login applications) default_contexts man page
/etc/selinux/default/contexts/users/ overrides rules in /etc/selinux/default/contexts/default_contexts by user
/etc/selinux/default/modules/active/users_extra
/etc/selinux/default/modules/active/users_extra.local
/etc/selinux/default/modules/active/users.local
/etc/selinux/default/modules/active/seusers
/etc/selinux/default/modules/active/users.final

Syntax

Define user

user user1_r roles { role1_r role2_r ... } level ...

Notes

  • SELinux adds type enforcement to standard Linux. This means that both the standard Linux and enhanced SELinux access controls must be satisfied to access an object. So, for example, if we have SELinux write access to a file but we do not have w permission on the file, we cannot write the file.
  • In general, consider domain, domain type, subject type, and process type to be synonymous.
  • Finally, be aware of the differences between the user ID in standard Linux security and the user identifier in a security context. Technically, these are completely orthogonal identifiers, used separately by the standard and security-enhanced access control mechanisms, respectively. Any relationship between these two is strictly provided via the login process according to conventions not directly enforced by the SELinux policy.
  • Remember that a type_transition rule causes a domain transition to be attempted by default, but it does not allow it.
  • SELinux Users can have multiple roles that they can reach, and then in those roles they can reach multiple types.
  • Three users that you will usually see on the system are "user_u", "system_u" and root. The user_u is the default SELinux User for a logged in user on a system. "system_u" is the default User for processes started during the boot up process.
  • The role field on a file is always object_r, and really has no meaning other than as a place holder.
  • RBAC(Roles Based Access Control) is not really used in targeted policy, but becomes more important in Strict and MLS policy.
  • Most of the policy rules in SELinux revolve around what subject types have what access to which object types.
  • LSM provides a set of hooks in the kernel system call logic. These hooks are usually placed after the standard Linux access checks but before the actual resource is accessed by the kernel on behalf of the caller.
  • In standard Linux, if you have a file descriptor, you can use it regardless of the change in file access mode. In SELinux, for objects such as files where access is validated on all attempts to use (for example, every read system call is checked against the policy and not just open calls), access revocation works fine.
  • Using and applying SELinux is all about writing and understanding policies.
  • SELinux dose not change the Linux DAC implementation nor can it override denials made by the Linux DAC permissions. If regular system (without SELinux) prevents a particular access, there is nothing SELinux can do to override this decision. This is because the LSM hooks are triggered after the regular DAC permission checks have been executed, which is a conscious design decision from the LSM project.
  • There are more than 80 classes and over 200 permissions known to SELinux and policy rules need to take into account all these classes and permissions for each interaction between two objects and resources.
  • SELinux has no notion of Linux process ownership and, once running, does not care how the process is called, witch processID it has, and what account the process runs as.
  • The majority of SELinux policy rules(over 99 percent) consists of rules related to the interaction between two types(without mentioning roles, users or sensitivity levels).
  • Multiple Linux users can be assigned to the same SELinux user.
  • When distributing SELinux policy modules, most Linux distributions place the *.pp SELinux policy modules inside /usr/share/selinux, usually withing a subdirectory named after the policy store.

Targeted Policy

  • Unconfined services executed by init end up running in the unconfined_service_t domain.
  • Unconfined services executed by kernel end up running in the kernel_t domain.
  • Unconfined services executed by unconfined Linux users end up running in the unconfined_t domain.

SELinux Reference Policy

Linux Distributions

Ubuntu

Installation

  • Must Know
    • Creating and deleting partitions can be done from within debian-installer as well as from an existing operating system.
    • During hardware detection debian-installer checks if any of the drivers for the hardware devices in your system require firmware to be loaded. If any firmware is requested but unavailable, a dialog will be displayed that allows the missing firmware to be loaded from a removable medium.
  • Boot parameters
priority=medium
 
netcfg/disable_autoconfig=true
  • Procedure
    1. Check available memory / low memory mode
    2. Selecting Localization Options
      • language, country, locales
    3. Choosing a Keyboard
    4. Looking for the Ubuntu Installer ISO Image
      • When installing via the hd-media method
    5. Configuring the Network
    6. Configuring the Clock and Time Zone
      • /usr/share/zoneinfo/
    7. Setting Up Users And Passwords
    8. Partitioning and Mount Point Selection
      • supports LVM, Software RAID, Serial ATA RAID, and Encryption
    9. Installing the Base System
    10. Installing Additional Software
    11. Making Your System Bootable
    12. Finishing the Installation
Installer Components for 16.04
Components Description Remarks
localechooser Allows the user to select localization options for the installation and the installed system: language, country and locales.
console-setup Shows a list of keyboards, from which the user chooses the model which matches his own.
hw-detect Automatically detects most of the system's hardware, including network cards, disk drives, and PCMCIA.
cdrom-detect Looks for and mounts an Ubuntu installation CD.
netcfg Configures the computer's network connections so it can communicate over the internet.
iso-scan Searches for ISO images (.iso files) on hard drives.
choose-mirror Presents a list of Ubuntu archive mirrors.
cdrom-checker Checks integrity of a CD-ROM.
lowmem Tries to detect systems with low memory and then does various tricks to remove unnecessary parts of debian-installer from the memory
anna Installs packages which have been retrieved from the chosen mirror or CD. Anna's Not Nearly APT
user-setup Sets up the root password, and adds a non-root user.
clock-setup Updates the system clock and determines whether the clock is set to UTC or not.
tzsetup Selects the time zone, based on the location selected earlier.
partman Allows the user to partition disks attached to the system, create file systems on the selected partitions, and attach them to the mountpoints. LVM support
lvmcfg Helps the user with the configuration of the LVM (Logical Volume Manager).
mdcfg Allows the user to set up Software RAID.
base-installer Installs the most basic set of packages which would allow the computer to operate under Ubuntu when rebooted.
apt-setup Configures apt, mostly automatically, based on what media the installer is running from.
pkgsel Uses tasksel to select and install additional software.
os-prober
bootloader-installer
shell Allows the user to execute a shell from the menu, or in the second console.
save-logs Provides a way for the user to record information on a floppy disk, network, hard disk, or other media when trouble is encountered.

Repositories

PPAs
PPA Title Description Remarks
ppa:adiscon/v8-stable rsyslog v8-stable Contains the latest RSyslog V8-Stable packages and dependencies.
ppa:jonathonf/backports Backport collection Backports of various (low impact) packages to Trusty and Xenial sqlite3
ppa:wireshark-dev/stable Wireshark stable releases Latest stable Wireshark releases back-ported from Debian package versions.
ppa:vbernat/haproxy-1.8 HAProxy 1.8 Contains packages for HAProxy 1.8.

Commands

  • tcp
    • /proc/sys/net/ipv4/

Service Control

Security

Troubleshooting

Debian

References

Packages

Package Description Remarks
procps /proc file system utilities free, kill, ps, sysctl, top, uptime, vmstat, and watch
net-tools includes the important tools for controlling the network subsystem of the Linux kernel ifconfig, netstat, route, ...

Readings

CentOS

Utilities

journalctl

Fedora

Red Hat Enterprise Linux

UNIX Systems

AIX

AIX Commands

  • oslevel
    • Reports the latest installed level (technology level, maintenance level and service pack) of the system.
  • lsdev
    • Displays devices in the system and their characteristics.
  • prtconf
    • Displays system configuration information.
  • lparstat
    • Reports logical partition (LPAR) related information and statistics.
  • no
    • Manages network tuning parameters.

Solaris

Commands, Utilities and Tools

Common

Command Description Readings
id return user identity http://en.wikipedia.org/wiki/Id_(Unix)
set allows you to change the values of shell options and set the positional parameters, or to display the names and values of shell variables. Bash set builtin
eval
source read and execute ex commands from file POSIX.1-2008/Utilities/ex/source
. evaluates commands in a computer file in the current execution context https://en.wikipedia.org/wiki/Dot_(command)
sudo allows users to run programs with the security privileges of another user (normally the superuser, or root) http://en.wikipedia.org/wiki/Sudo
history
kill sends the specified signal to the specified process or process group https://en.wikipedia.org/wiki/Kill_(command)
date print or set the system date and time

ls

Category Option Description Remarks
Format -h, --human-readable Print human readable sizes with -l and/or -s
Content -i, --inode Print the index number of each file
Sort -t Sort by modification time newest first
-S Sort by file size largest first
-v Natural sort of (version) numbers within text
-r, --reverse Reverse order while sorting

cat

Option Description Remarks
-n, --number Number all output lines
-E, --show-ends Display $ at end of each line

id

set

Option Description Remarks
-v Print shell input lines as they are read.set echo on
-x Print a trace of simple commands

eval

su

sudo

Option Description Remarks
-H, --set-home Request that the security policy set the HOME environment variable to the home directory specified by the target user's password database entry.

history

Readings

kill

killall

date

IO

Command Description Readings
find find files POSIX.1-2008/Utilities/find
grep search a file for a pattern POSIX.1-2008/Utilities/grep
xargs construct argument lists and invoke utility POSIX.1-2008/Utilities/xargs
wc word, line, and byte or character count POSIX.1-2008/Utilities/wc
tee reads standard input and writes it to both standard output and one or more files, effectively duplicating its input. tee
curl command line tool and library for transferring data with URLs Manual
man page
wget non-interactive download of files from the Web GNU Wget Manual
scp secure remote file copy program

find

Option Description Values Remarks
-type c the type of the file d: directory, l: symbolic link, f: regular file, s: socket ...

grep

Category Option Description Remarks
Input -E Interpret PATTERN as an extended regular expression.
-v Invert the sense of matching, to select non-matching lines.
-P PATTERN is a Perl regular expression
Output -m NUM Stop reading a file after NUM matching lines.
Context -A NUM Print NUM lines of trailing context.
-B NUM Print NUM lines of leading context.
-C NUM, -NUM Print NUM lines of both leading and trailing context. grep -10 ssh
Samples
$ #Exclude empty or commented lines(starting with leading #)
$ grep -Ev '(^#|^$)' ...

$ #Another expression to exclude empty or commented lines
$ grep -P '^[^#].*' ...

xarg

sort

tee

curl

Option Description Remarks
-X, --request command Specifies a custom request method to use when communicating with the HTTP server. PUT DELETE
-d, --data data Sends the specified data in a POST request to the HTTP server, in the same way that a browser does when a user has filled in an HTML form and presses the submit button
-H, --header header|@file (HTTP) Extra header to include in the request when sending HTTP to a server. -H "Connection: close"
-L, --location If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code), this option will make curl redo the request on the new place
-o, --output file Write output to file instead of stdout.
-O, --remote-name Write output to a local file named like the remote file we get
-i, --include Include the HTTP response headers in the output.
-v, --verbose Makes curl verbose during the operation.
-s, --slient
-S, --show-error
--interface name Perform an operation using a specified interface by interface name, IP address or host name.
-x, --proxy [protocol://]host[:port] Use the specified proxy.
-U, --proxy-user user:password Specify the user name and password to use for proxy authentication.
--connect-timeout SECONDS Maximum time allowed for connection
-m --max-time
-k
--key
--cert
--cacert
--cert-type
Readings
  • Timeouts : --max-time(-m), --connect-timeout
Security
Typical Usage
$ # Download binary file being aware of redirect
$ curl -OL https://github.com/vi/websocat/releases/download/v1.4.0/websocat_1.4.0_ssl1.0_amd64.deb

$ # Capture the HTTP status code
$ status=`curl -s -o /dev/null -w '%{http_code}' http://www.google.com/`
$ echo $status

scp

Editing

sed

References
Syntax
[addr]X[options]
  • [addr] : optional line address - single line number, a range of lines, a regular expression
  • X : single-letter command - a, c, d, D, i, p, s
Commands
Command Description Remarks
a text Append text after a line
c text Replace (change) lines with text
i text Insert text before a line
p Print the pattern space
P Print the pattern space, up to the first new line
n next
{ cmd ; cmd ... } Group several commands together.
Command Syntax Sample Remark
Substitution 's/pattern/replacement/option' sed 's/\r$//g' README.windows.txt > README.linux.txt
Append Lines 'address a text-to-append' sed -i '2 a export GOROOT=/usr/lib/go-1.8\nPATH=$PATH:$GOROOT/bin\n' .profile
sed -i '$ a \\nexport GOROOT=/usr/lib/go-1.8\nexport PATH=$PATH:$GOROOT/bin\n' ~/.bashrc
'$' as an address means the last line.
'\n' at the beginning of replacement string need additional leading backslash to form '\\n'.
Change Lines 'address c text-to-change'
Delete Lines 'address d' \s*)$/ d' squid.conf
CLI Options
Option Description Remarks
-e script, --expression=script add the script to the commands to be executed
-i[SUFFIX], --in-place[=SUFFIX] edit files in place (makes backup if SUFFIX supplied)
-E, -r, --regexp-extended use ERE in the script
-n, --quiet, --silent suppress automatic printing of pattern space
Character escape

The only difference between basic and extended regular expressions is in the behavior of a few characters: ‘?’, ‘+’, parentheses, braces (‘{}’), and ‘|’. While basic regular expressions require these to be escaped if you want them to behave as special characters, when using extended regular expressions you must escape them if you want them to match a literal character.

Notation Special Character Literal
BRE \?, \+, \(, \), \{, \}, \| ?, +, (, ), {, }, |
ERE ?, +, (, ), {, }, | \?, \+, \(, \), \{, \}, \|
Readings
Examples
$ # Simple substitution
$ timedatectl status | grep 'Network time on' | sed  's/.*\(yes\|no\).*/\1/'
 
$ # Using shell parameter or variable
$ sed -r -i 's/^\s*(vm\.'"${param}"'.*)/#\1/g' /etc/sysctl.conf 
 
$ # Escape single quote using '\x27'
$ sed -iE 's/^NTPD_OPTS.*$/NTPD_OPTS=\x27-g -4 -I eth0\x27/' /etc/default/ntp
 
$ # Multiple edit in a single line
$ echo $output | sed -e 's/.*\[validator_token] \(.*\)/\1/; s/\s//g'
 
$ # Print out the sentence like file names such as song title more human readable
$ lowered='A|An|The|In|On|Under|Of|To|For|Like'
$ ls | sed -E 's/(.*)/\L\1/; s/(\b\w)/\U\1/g; s/\b('$lowered')\b/\L\1/g; s/^(.)/\U\1/'
 
$ # Using # instead of / for substitution command
$ sed -i -r 's#@@peer1@@#\"peer1\": '"$peer_str"',#' ./generated/vm${no}/channelConfig.json
 
$ # Adding a line 
$ sed -i '2 a export GOROOT=/usr/lib/go-1.8' .profile
 
$ # Deleting commented or empty lines
$ sed -iE '/^(#.*|\s*)$/ d' squid.conf
 
$ # Using multiple command in a line using { }
$ cat rippled.cfg | sed -n '/^\[validator_token\]/{n;p}'

awk

  • Desc. : search files for lines (or other units of text) that contain certain patterns.
  • Syntax :
awk [options] [--] 'pattern { action } pattern { action } ...' file
References
Command-Line Options
Option Description Remarks
-v var=val Set the variable var to the value val before execution of the program begins. -v OFS='\t'
Predefined Variables
Variable Type Description Remarks
NR Auto-set The number of input records awk has processed since the beginning of the program’s execution
NF Auto-set The number of fields in the current input record.
Statements
Statement Description Remarks
print produce output with simple, standardized formatting.
printf you can specify the width to use for each item, as well as various formatting choices for numbers
Readings
Examples
$ # Just filtering without any action
$ ls -l | awk '$3 == "root"' 
 
$ # Filtering using ~ operator and regex
$ netstat -antp | awk '$4 ~ /.*:(1580|1590|1600|1943|1953|1963|2080|2090|2100|2443|2453|2463|5505|5515|6506|6516)/'
 
$ # Escape $ when using double quotation
$ watch -n 2 'netstat -antp | awk "\$4 ~ /.*:(1580|1590|1600|1943|1953|1963|2080|2090|2100|2443|2453|2463|5505|5515|6506|6516)$/"| sort -k 4'
 
$ # Just action without filtering pattern
$ awk -F, '{if (($1 ~ /vm[0-9]*/) && (substr($1, 3, length($1) - 1) + 0 < 97)) print $1 " " $2}' ../../vms.csv | while read vm ip; do

read

Parsing

Jshon

jq

References
Options
Option Description Remarks
-s, --slurp Instead of running the filter for each JSON object in the input, read the entire input stream into a large array and run the filter just once
-R, --raw-input Don’t parse the input as JSON. Instead, each line of text is passed to the filter as a string. jq -R 'fromjson? | ... '
-r, --raw-output If the filter’s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes.
Filters
Filter Syntax Description Remarks
Identity .
Object Identifier .key
Optional Object Identifier .key?
Generic Object Index .["key"]
Array Index .[n]
Array/String Slice .[n:m]
Array/Object Value Iterator .[] Return all of the elements of an array or all the values of the object
Optional Array/Object Value Iterator .[]? Like .[], but no errors will be output if . is not an array or object.
Comma , If two filters are separated by a comma, then the same input will be fed into both and the two filters’ output value streams will be concatenated in order
Piple | Combines two filters by feeding the output(s) of the one on the left into the input of the one on the right .a | .b | .c == .a.b.c
.[] | .foo == .[].foo (?)
Expressions
Expression Syntax Description Remarks
Array Constructor [] used to construct arrays [.foo, .bar, .baz], [.items[].name]
Object Constructor {}
Recursive Descent ..
rippled=`echo ${rippled}' '$peers_connected | jq -s '.[0] + { peers_found : [.[1].result.peers[].public_key] }'`

echo $rippleds | jq '.['$i'] | { name: .name, type: .type, peers_defined: .peers }'
Operators
Operator Syntax Description Remarks
Addition Operator + takes two filters, applies them both to the same input, and adds the results together
Variable / Symbolic Binding Operator ... as $identifier |
Alternative Operator // defaults
Functions
Function Syntax Description Remarks
Has Function has(key)
In Function in(object), in(array) sorts its input, which must be an array
Any Filter any, any(condition)
All Filter all, all(condition)
Map Function map(filter), map(function) run that filter for each element of the input array, and return the outputs in a new array
Select Function select(predicate) predicate = boolean expression
Sort Function sort, sort_by(path_expression)
Flatten Filter flatten, flatten(depth)
Object/Array Conversion Function to_entries, from_entries, with_entries Convert between an object and an array of key-value pairs
echo $rippleds3 | jq 'sort_by(if .type == "validator" then 1 else 2 end, .name)'
Readings
Examples
Navigating JSON node using dot operator
...
for (( i = ${start}; i <= ${end}; i++ )); do
  resp=`curl -ksS --header "Content-Type: application/json" \
    --header "Connection: keep-alive" \
    --data "{\"method\": \"ledger\", \"params\": [ { \"ledger_index\": ${i}, \"transactions\": true } ]}" \
    http://${addr}:${port} | jq '.'`

  status=`echo ${resp} | jq '.result.status'`
  ...
  close_time=`echo ${resp} | jq '.result.ledger.close_time_human'`
  ...
  txs=`echo ${resp} | jq '.result.ledger.transactions'`
  ...
  txs_num=`echo ${txs} | jq '. | length'`
  ...
done
Constructing new JSON object
$ curl -sSX POST \
>   http://tracker1/ \
>   -H 'Connection: keep-alive' \
>   -H 'Content-Type: application/json' \
>   -d '{
>     "method": "server_info",
>     "params": [ {} ]
> }' | jq '.result.info | {complete_ledgers: .complete_ledgers, hostid: .hostid, time: .time}'
{
  "complete_ledgers": "1247544-1440506,1440513-1460692,1460699-1468605,1468609-1479330,1479335-1567817",
  "hostid": "tracker1",
  "time": "2018-Oct-15 06:50:04.525106"
}
Merging distinct JSON objects using + operator
info=`curl -ksS --data '{"method":"server_info", "params": [{}]}' http://$addr:$port \
  | jq '.result.info | {build_ver: .build_version, complete_ledgers: .complete_ledgers, io_latency_ms: .io_latency_ms, peers: .peers, peer_disconnects: .peer_disconnects, server_state: .server_state, uptime: .uptime, validated_ledger_index: .validated_ledger.seq}'`

logging=`curl -ksS --data '{"method":"log_level", "params": [{}]}' http://$addr:$port \
  | jq '{base_log_level: .result.levels.base}'`

validators=`curl -ksS --data '{"method":"validators", "params": [{}]}' http://$addr:$port \
  | jq '{trusted_validators: .result.trusted_validator_keys | length, validation_quorum: .result.validation_quorum}'`

echo "{\"info\": $info, \"logging\": $logging, \"validators\": $validators}" | jq '.info + .logging + .validators'
Merging distinct JSON string using -s option and + operator
rippled=`echo $rippled' '$peers_connected | jq -s '.[0] + { peers_found : [.[1].result.peers[].public_key] }'`
Using select function
  1. #! /bin/bash
  2.  
  3. declare script_dir=`dirname -- "$0"`
  4. script_dir=$(cd "$script_dir" && pwd)
  5. declare -r config="$script_dir/../generated/ripple-network-config.json"
  6.  
  7. # TODO Sort by name
  8. declare -r rippleds=`cat "$config" | jq -r '.rippleds'`
  9. declare -r rippleds_cnt=`echo $rippleds | jq 'length'`
  10. declare -r protocol=https
  11.  
  12. declare rippled
  13. declare name
  14. declare peers_defined
  15. declare proxy
  16. declare addr
  17. declare port
  18. declare peers_connected
  19. declare validators
  20. for (( i = 0; i < $rippleds_cnt; i++ )); do
  21.   rippled=`echo $rippleds | jq '.['$i']'`
  22.  
  23.   name=`echo $rippled | jq -r '.name'`
  24.   echo $name
  25.  
  26.   peers_defined=`echo $rippled | jq -r '.peers'`
  27.   echo $peers_defined
  28.  
  29.   proxy=`echo $rippled | jq -r '.proxy.frontends[] | select((.backend == "rippled-rpc") and (.protocol == "'$protocol'"))'`
  30.   echo $proxy
  31.  
  32.   addr=`echo $proxy | jq -r '.address'`
  33.   echo $addr
  34.  
  35.   port=`echo $proxy | jq '.port'`
  36.   echo $port
  37.  
  38.   peers_connected=`curl -ksS -X POST \
  39.   --cacert "$script_dir/../files/tls/test-ca.crt" \
  40.   --cert "$script_dir/../files/tls/test-tls-client.crt" \
  41.   --key "$script_dir/../files/tls/test-tls-client.key" \
  42.   --cert-type PEM \
  43.   $protocol'://'$addr':'$port'/' \
  44.   -H 'Connection: close' \
  45.   -H 'Content-Type: application/json' \
  46.   -d '{"method": "peers", "params": [{}]}'`
  47.   echo $peers_connected
  48.  
  49.   validators=`curl -ksS -X POST \
  50.   --cacert "$script_dir/../files/tls/test-ca.crt" \
  51.   --cert "$script_dir/../files/tls/test-tls-client.crt" \
  52.   --key "$script_dir/../files/tls/test-tls-client.key" \
  53.   --cert-type PEM \
  54.   $protocol'://'$addr':'$port'/' \
  55.   -H 'Connection: close' \
  56.   -H 'Content-Type: application/json' \
  57.   -d '{"method": "validators", "params": [{}]}'`
  58.   echo $validators
  59. done
Filtering an array using map and select functions
  1. #! /bin/bash
  2.  
  3. # addrs : {label: address}*
  4. # tests : [source_label, target_label, expected]*
  5.  
  6. addrs2=(`hostname -I`)   # addresses for this host, usually 1 or 2 elements
  7. for addr2 in "${addrs2[@]}"; do
  8.   # find label for current address
  9.   label=`echo $addrs | jq -r 'to_entries[] | select(.value == "'$addr2'") | .key'`
  10.  
  11.   # select tests of which source is current address
  12.   tests2=`echo $tests | jq -r 'map(select(.[0] =="'$label'"))'`
  13.  
  14.   n=`echo $tests2 | jq 'length'`
  15.   for(( i = 0; i < $n; i++)); do
  16.     from=`echo $tests2 | jq -r '.['$i'][0]'`
  17.     to=`echo $tests2 | jq -r '.['$i'][1]'`
  18.     expected=`echo $tests2 | jq -r '.['$i'][2]'`
  19.  
  20.     fromAddr=`echo $addrs | jq -r '.'$from`
  21.     toAddr=`echo $addrs | jq -r '.'$to`
  22.  
  23.     # execute a test
  24.     ping -c $cnt -W $timeout -q -I $fromAddr $toAddr > /dev/null 2>&1
  25.  
  26.     # print out the result
  27.     if [ $? -eq 0 ]; then
  28.       if [ "$expected" == "Allowed" ]; then rc='INFO '; else rc='ERROR'; fi
  29.       echo "[$rc] ${from} -> $to : Expected: $expected, Actual: Allowed"
  30.     else
  31.       if [ "$expected" == "Disallowed" ]; then rc='INFO '; else rc='ERROR'; fi
  32.       echo "[$rc] $from -> $to : Expected: $expected, Actual: Disallowed"
  33.     fi
  34.   done
  35. done

Administration

Command Description Remarks
who Show who is logged on
w Show who is logged on and what they are doing.
useradd Create a new user or update default new user information
groupadd Create a new group
ulmit Get and set user limits.
mount
cron Daemon to execute scheduled commands.

who, w

useradd

groupadd

ulimit

Option Description Remarks
-n The maximum number of open file descriptors
-p The pipe buffer size
-s The maximum stack size
-u The maximum number of processes available to a single user
-v The maximum amount of virtual memory available to the process

mount

Mount Options
Option Description Filesystems Remarks
atime The inode access time is controlled by kernel defaults.
noatime Do not update inode access times on this filesystem.
relatime Update inode access times relative to modify or change time.
strictatime Allows to explicitly requesting full atime updates.
ro Mount the filesystem read-only.
rw Mount the filesystem read-write.
default Use default options: rw, suid, dev, exec, auto, nouser, async, and relatime.

cron

clusterssh

parallel

Terminator

Package Management

APT

Commands
utility command description remarks
apt-get APT package handling utility -- command-line interface
update resynchronize the package index files from their sources
upgrade install the newest versions of all packages currently installed on the system from the sources enumerated in /etc/apt/sources.list. Apply any update that does not involve removing components
dist-upgrade Update all packages, even those that require package removal
install
remove packages are removed leaving its configuration files on the system.
purge packages are removed and purged (any configuration files are deleted too).
clean clears out the local repository of retrieved package files.
autoclean clears out the local repository of retrieved package files that can no longer be downloaded, and are largely useless.
autoremove remove packages that were automatically installed to satisfy dependencies for other packages and are now no longer needed.
apt-cache query the APT cache
show displays the package records for the named packages.
showpkg displays information about the packages listed on the command line additional information
policy print out the priorities of each source with no arguments
apt-file searching files in packages for the APT package management system
apt provides a high-level commandline interface for the package management system. cann't be used inside script file
  • autoremove
    • In order to automatically remove any packages that were installed as dependencies that are no longer required by any packages, you can use the autoremove command:
    • If you wish to remove all of the associated configuration files from the dependencies being removed, you will want to add the --purge option to the autoremove command.
  • autoclean
    • Remove any package files on the local system that are associated with packages that are no longer available from the repositories by using the autoclean command.
Readings
Examples
$ # Update local package cache
$ sudo apt-get update

$ # Apply any update that does not involve removing components
$ sudo apt-get upgrade

$ # Update all packages, even those that require package removal
$ sudo apt-get dist-upgrade

$ # Dry run to update all packages
$ sudo apt-get -s dist-upgrade

$ # show detailed information about a package in your distribution's repositories
$ apt-cache show apache2

$ # show additional information about each of the candidates
$ apt-cache showpkg apache2

$ # show details about a .deb file
$ dpkg --info debfile.deb

$ # Perform a dry run of package actions
$ apt-get install -s apache2

$ # Fix broken dependencies and packages
$ sudo apt-get install -f

$ # Remove any packages that were installed as dependencies that are no longer required by any packages
$ sudo apt-get autoremove

$ # Remove any packages that were installed as dependencies that are no longer required by any packages with all of the associated configuration files
$ sudo apt-get --purge autoremove

$ # Remove any package files on the local system that are associated with packages that are no longer available from the repositories
$ sudo apt-get autoclean

dpkg

Commands
Command Description Remarks
dpkg package manager for Debian
dpkg-query a tool to query the dpkg database
Actions
Action Syntax Description Remarks
Install dpkg -i package_file... --install
Remove dpkg -r package... remove everything except conffiles --remove
Purge dpkg -P pakcage... remove everything, including conffiles --purge
Status
Status Description Remarks
rc the package has been removed, but that the configuration files remain
  • 1st character : desired state
Character Status Description
u Unknown an unknown state
i Install marked for installation
r Remove marked for removal
p Purge marked for purging
h Hold
  • 2nd character : current state
Character Status Description
n not-installed The package is not installed on your system.
i installed The package is unpacked and configured OK.
c config-files Only the configuration files of the package exist on the system.
u unpacked The package is unpacked, but not configured.
h half-installed The installation of the package has been started, but not completed for some reason.
W triggers-awaited The package awaits trigger processing by another package.
t triggers-pending The package has been triggered.
Readings

Monitoring & Diagnosis

Command/Tool Description Remarks
sysctl examining and changing kernel parameters at runtime
ps report a snapshot of the current processes
top display Linux tasks
lsof displays information about files open to Unix processes
strace
iostat collect and show operating system storage input and output statistics
netstat
ss show socket statistics
tcpdump
rsyslog the rocket-fast system for log processing
logrotate rotates, compresses, and mails system logs

ps

Options
Category Option Description Remarks
Simple Process Selection a Lift the BSD-style "only yourself" restriction, which is imposed upon the set of all processes when some BSD-style (without "-") options are used or when the ps personality setting is BSD-like. The set of processes selected in this manner is in addition to the set of processes selected by other means. An alternate description is that this option causes ps to list all processes with a terminal (tty), or to list all processes when used together with the x option.
x Lift the BSD-style "must have a tty" restriction, which is imposed upon the set of all processes when some BSD-style (without "-") options are used or when the ps personality setting is BSD-like. The set of processes selected in this manner is in addition to the set of processes selected by other means. An alternate description is that this option causes ps to list all processes owned by you (same EUID as ps), or to list all processes when used together with the a option.
Output Format Control u Display user-oriented format.
Output Modifires c Show the true command name. This is derived from the name of the executable file, rather than from the argv value. Command arguments and any modifications to them are thus not shown. This option effectively turns the args format keyword into the comm format keyword; it is useful with the -f format option and with the various BSD-style format options, which all normally display the command arguments. See the -f option, the format keyword args, and the format keyword comm.
o Specify user-defined format Identical to -o and --format
f ASCII art process hierarchy (forest)
--sort specify sorting order. Sorting syntax is [+|-]key[,[+|-]key[,...]]
Format Specifiers
  • Used with o or --sort options
Code Header Description Aliases Remarks
pid PID a number representing the process ID tgid(TGID)
ppid PPID a number representing the process ID tgid(TGID)
pgid PGID process group ID or, equivalently, the process ID of the process group leader pgrp(PGRP)
sid SID session ID or, equivalently, the process ID of the session leader sess(SESS)
tid TID light weight process (thread) ID of the dispatchable entity spid(SPID), lwp(LWP)
comm COMMAND command name (only the executable name) ucmd(CMD), ucomm(COMMAND)
command COMMAND command with all its arguments as a string args(COMMAND)
uid UID effective user ID euid(EUID)
user USER effective user name (textual user ID) euser(EUSER)
gid GID effective group ID number of the process as a decimal integer egid(EGID)
group GROUP effective group name of the process (textual group ID) egroup(EGROUP)
ruid RUID real user ID
ruser RUSER real user name (textual user ID)
rgid RGID real group ID
rgroup RGROUP real group name (textual group ID)
stat STAT multi-character process state
tname TTY Controlling terminal tt, tty
start STARTED time the command started
lstart STARTED time the command started long format ?
etime ELAPSED elapsed time since the process was started, in the form [[DD-]hh:]mm:ss
etimes ELAPSED elapsed time since the process was started, in seconds
time TIME cumulative CPU time, "[DD-]HH:MM:SS" format cputime
times TIME cumulative CPU time in seconds cputimes
nlwp NLWP number of lwps (threads) in the process thcount
rss RSS the non-swapped physical memory that a task has used (in KB) rsz Resident Set Size
vsz VSZ virtual memory size of the process in KiB vsize
Process State Codes
Category Code Meaning Remarks
Main R running or runnable (on run queue)
S interruptible sleep (waiting for an event to complete)
X dead (should never be seen)
Z defunct ("zombie") process, terminated but not reaped by its parent
Additional < high-priority (not nice to other users)
N low-priority (nice to other users)
L has pages locked into memory (for real-time and custom IO)
s is a session leader
l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
+ is in the foreground process group
Readings
  • What are “session leaders” in `ps`? (Aug 6 '11)
    • Sessions and process groups are just ways to treat a number of related processes as a unit. All the members of a process group always belong to the same session, but a session may have multiple process groups
Examples
$ ps axuww      #full command line
 
$ ps axuc       #true command name only, used before calling 'killall -r'
 
$ ps axucf      #true command name in hierarchy
 
$ ps axufww     #full command line in hirerachy
 
$ ps axuww -T   #including thread
 
$ ps axufww | grep -v grep | grep -E '^USER|haproxy' 
 
$ ps axu --sort=-time | head -n 5
 
$ ps axww o pid,user,group,command | grep -v grep | grep -E '^PID|rippled'
 
$ ps axH -o pid,ppid,pgid,sid,stat,tid,rss,comm,args --sort tid | grep -v grep | grep -E '^\s*PID|rippled'
 
$ ps axf o pid,ppid,pgid,sid,commad

top

Interactive Command
Category Command Title Description Remarks
GLOBAL Commands I Irix/Solaris Mode Toggle When operating in 'Solaris mode', a task's cpu usage will be divided by the total number of CPUs.
SUMMARY Area Commands 1 Toggle Single/Separate CPU States
m Toggle Memory/Swap Usage
TASK Area Commands/APPEARANCE z Color/Monochrome Toggle
f Fields Select Display separate screens where you can change which fields are displayed and their order
TASK Area Commands/CONTENT c Command Line/Program Name Toggle Whether or not the 'Command' column is currently visible
H Threads Toggle Displays all individual threads or a summation of all threads in a process.
u Show Specific User Only Will be prompted to enter the name of the user to display.
Fields
Letter Field Name Description Remarks
a PID Process Id
h PR Priority The priority of the task.
i NI Nice value The nice value of the task.
n %MEM Memory usage (RES) A task's currently used share of available physical memory.
o VIRT Virtual Image (kb) The total amount of virtual memory used by the task.
q RES Resident size (kb) The non-swapped physical memory a task has used. RES = CODE + DATA
r CODE Code size (kb) The amount of physical memory devoted to executable code. TRS (Text Resident Set)
s DATA Data+Stack size (kb) The amount of physical memory devoted to other than executable code DRS (Data Resident Set)
t SHR Shared Mem size (kb) The amount of shared memory used by a task.
w S Process Status The status of the task S, R, D, T or Z
Process Status Codes
Code Name Description
S Sleeping
R Running
D Uninterruptible sleep
T Traced or Stopped
Z Zombie
Readings

lsof

Predefined file descriptor
Name Description Remarks
cwd current working directory
rtd root directory
txt program text (code and data)
mem memory-mapped file
Options
Option Format Description Remarks
-n Inhibits the conversion of network numbers to host names for network files. Do not resolve hostnames
-P Inhibits the conversion of port numbers to port names for network files. Do not resolve port names
-i [i] [46][protocol][@hostname|hostaddr][:service|port] Selects the listing of files any of whose Internet address matches the address specified in i.
Samples
# List only network files with TCP state LISTEN
$ sudo lsof -nP -i TCP -s TCP:LISTEN

# List network files with TCP on 8080 port of any interface or internal address
$ sudo lsof -nP -i TCP@0.0.0.0:8080 -i TCP@192.168.1.31:8080

strace

iotop

hdparm

dd

iostat

netstat

Options
Option Description Remark
-a, --all Show both listening and non-listening (for TCP this means established connections) sockets.
-l, --listen Show only listening sockets.
-n, --numeric Show numerical addresses instead of trying to determine symbolic host, port or user names.
-t, --tcp Display only TCP connections. Linux
-u, --udp Display only UDP connections. Linux
-p, --program Show the PID and name of the program to which each socket belongs.
-o, --timers Include information related to networking timers.
-e, --extend Show the PID and name of the program to which each socket belongs.
Samples
$ netstat -antpe   # list all TCP connections in states of LISTEN, ESTABLISHED, TIME_WAIT and so on
 
$ netstat -lntp    # list TCP listening ports
 
$ netstat -lnup    # list UPD listening ports
Readings

ss

tcpdump

iperf

iperf3

Syslog

rsyslog

logrotate

Options
Option Description Remarks
-d, --debug Don't do anything, just test. implies -v
Readings

Networking

ip

  • ip man page
  • show / manipulate routing, devices, policy routing and tunnels

brctl

ping

Options
Option Description Remarks
-c count Stop after sending count ECHO_REQUEST packets.
-I interface_address Set source address to specified interface_address. numeric IP address or device name
-W timeout Time to wait for a response, in seconds.

traceroute

conntrack

PAC Manager

Security

PAM (Pluggable Authentication Modules)

  • http://www.linux-pam.org
  • Desc. : a suite of shared libraries that enable the local system administrator to choose how applications authenticate users.
Readings
Configuration
module_interface     control_flag     module_name module_arguments
Module Interfaces
Interface Description Remarks
auth authenticates use
account verifies that access is allowed
password used for changing user passwords
session configures and manages user sessions
Control Flags
Flag Description Remarks
required The module result must be successful for authentication to continue. If the test fails at this point, the user is not notified until the results of all module tests that reference that interface are complete.
requisite The module result must be successful for authentication to continue. However, if a test fails at this point, the user is notified immediately with a message reflecting the first failed required or requisite module test.
sufficient The module result is ignored if it fails.
optional The module result is ignored.
include
Modules
Module Description Interfaces Remarks
pam_succeed_if succeed or fail authentication based on characteristics of the account belonging to the user being authenticated or values of other PAM items. auth, account, password, session
pam_tally maintains a count of attempted accesses, can reset count on success, can deny access if too many attempts fail. auth, account
pam_tally2 maintains a count of attempted accesses, can reset count on success, can deny access if too many attempts fail. auth, account
pam_time restricts access to a system and or specific applications at various times of the day and on specific days or over various terminal lines. account
pam_motd display arbitrary motd (message of the day) files after a successful login. sseion
Tips and Tricks
  • Remarkable configurations
# lock the user with successive password failures (except for root user)
auth required pam_tally.so onerr=fail deny=5 no_magic_root

# lock the user with successive password failures for a considerable time (including root user)
auth required pam_tally2.so deny=5 even_deny_root unlock_time=1200
  • List Linux services that use PAM
$ ls /etc/pam.d/
  • Check if a program uses Linux-PAM or not
$ ldd /bin/su | grep 'libpam.so'

pass

  • Desc. : stores, retrieves, generates, and synchronizes passwords securely

iptables

References
  • iptables (on Wikipedia)
    • allows a system administrator to configure the tables[2] provided by the Linux kernel firewall (implemented as different Netfilter modules) and the chains and rules it stores.
  • iptables(8) man page
Parameters
Parameter Description Remarks
-p, --protocol [!] protocol The protocol of the rule or of the packet to check
-s, --src, --source [!] address[/mask] Source specification
-d, --dst, --destination [!] address[/mask] Destination specification
--dport, --destination-port [!] port[:port] Destination port or port range specification
--sport, --source-port [!] port[:port] Source port or port range specification
-i, --in-interface [!] name Name of an interface via which a packet was received
-o, --out-interface [!] name Name of an interface via which a packet is going to be sent
-j, --jump target This specifies the target of the rule; i.e., what to do if the packet matches it.
Readings

TCP Wrapper

SETools

Shell Script Test

Bats

shUnit2

misc

watch

screen

Tips and Tricks

Diagnosing and Monitoring Linux

Task Commandline Remarks
Identifying the product of Linux installed $ cat /etc/issue
$ cat /etc/*-release
Identifying kernel version $ cat /proc/version
Listing or identifying kernel parameters $ sysctl -a | more
$ sysctl net.ipv4 | more
$ sysctl net.ipv4.tcp_max_syn_backlog
Listing currently logged in users $ sudo w
Identifying CPU capacity $ cat /proc/cpuinfo
Identifying memory capacity and usage $ cat /proc/meminfo
Listing processes $ ps auxfww
Identifying threads of a specific process $ ps -T -p 31
Listing disks $ lsblk
Listing filesystems $ df -ahT
$ mount -l
Identifying details of a certain filesystem $ dumpe2fs -h /dev/xvda2
Identifying the filesystem a certain file belongs to $ df -h /var
Checking disk caching $ hdparm -W /dev/sda
Identifying TCP/IP ports in use $ netstat -anotup
Identifying sockets summary $ ss -s
Identifying the user limits of current login or session $ ulimit -a
Identifying the max. number of file handles for the entire system $ cat /proc/sys/fs/file-max
Identifying file handle usage $ cat /proc/sys/fs/file-nr
Counting the number of currently open files to a specific process $ lsof -a -p pid -d ^mem -d ^cwd -d ^rtd -d ^txt -d ^DEL | wc -l

Identifying the product of Linux installed

For Linux, /etc/issues file contains more detailed information on what Linux product it is.

$ cat /etc/issue
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
Kernel \r on an \m

For most of Linux distributions, /etc/*-release file is provided which contains more detailed and systematic information. The file would be lsb-release, os-release, redhat-release or so on as to the Linux distributions.

$ cat /etc/*-release
...

Identifying the environmental variables of current session

$ env
...

Identifying the locales of current session

$ locale
...

Identifying kernel version

$ cat /proc/version
Linux version 4.4.0-83-generic (buildd@lgw01-29) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017

Identifying kernel parameters

To print out all kernel parameters

$ sysctl -a | more

To print out a group of kernel parameters

$ sysctl net.ipv4 | more

To print out a specific kernel parameter

$ sysctl net.ipv4.tcp_syncookies
...
$ sysctl net.ipv4.conf.all.rp_filter
...
$ sysctl net.ipv4.tcp_max_syn_backlog
...

Identifying boot-up time

$ uptime -s

Or

$ who -b

Listing currently logged in users

$ sudo w

Identifying CPU capacity

$ cat /proc/cpuinfo

Identifying memory capacity and usage

$ cat /proc/meminfo
MemTotal:        4041600 kB
MemFree:          291372 kB
MemAvailable:    3537040 kB
...

Identifying threads of a specific process

$ ps -T -p 31

Listing disks

$ lsblk
...

Listing filesystems

To list filesystems with capacity and usage, use df

$ df -ahT
...

To list filesystems with mount options, use mount command

$ mount -l
...

Identifying the details of a certain filesystem

$ dumpe2fs -h /dev/xvda2
...

Identifying the filesystem a certain file belongs to

$ df -h /var   #asking what filesystem contains /var directory
...

Checking disk caching

$ hdparm -W /dev/sda
...

Identifying disk detail information

$ hdparm -i /dev/sda
...

Benchmarking disk performance

for i in {1..3}; do
  echo "Started &i/3 test:"
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=notrunc oflag=append,noatime
done   

for i in {1..3}; do
  echo "Started &i/3 test:"
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=fdatasync,notrunc oflag=append,noatime
done   

for i in {1..3}; do
  echo "Started &i/3 test:"
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=notrunc oflag=dsync,append,noatime
done

for i in {1..3}; do
  echo "Started &i/3 test:"
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=notrunc oflag=direct,append,noatime
done
References
Each CONV symbol may be:

  ascii     from EBCDIC to ASCII
  ebcdic    from ASCII to EBCDIC
  ibm       from ASCII to alternate EBCDIC
  block     pad newline-terminated records with spaces to cbs-size
  unblock   replace trailing spaces in cbs-size records with newline
  lcase     change upper case to lower case
  ucase     change lower case to upper case
  sparse    try to seek rather than write the output for NUL input blocks
  swab      swap every pair of input bytes
  sync      pad every input block with NULs to ibs-size; when used
            with block or unblock, pad with spaces rather than NULs
  excl      fail if the output file already exists
  nocreat   do not create the output file
  notrunc   do not truncate the output file
  noerror   continue after read errors
  fdatasync  physically write output file data before finishing
  fsync     likewise, but also write metadata

Each FLAG symbol may be:

  append    append mode (makes sense only for output; conv=notrunc suggested)
  direct    use direct I/O for data
  directory  fail unless a directory
  dsync     use synchronized I/O for data
  sync      likewise, but also for metadata
  fullblock  accumulate full blocks of input (iflag only)
  nonblock  use non-blocking I/O
  noatime   do not update access time
  nocache   Request to drop cache.  See also oflag=sync
  noctty    do not assign controlling terminal from file
  nofollow  do not follow symlinks
  count_bytes  treat 'count=N' as a byte count (iflag only)
  skip_bytes  treat 'skip=N' as a byte count (iflag only)
  seek_bytes  treat 'seek=N' as a byte count (oflag only)

Identifying TCP/UDP ports currently in use

You can identify TCP/IP ports currently in use using netstat command. The options of netstat is slightly different among operating systems.

For UNIX,

$ netstat -anotu

For Linux,

$ sudo netstat -anotup   #-a: all sockets, -n: numeric, -t: TCP, -u: UDP, -p: PID/program, -o: timers

You need root privilege to take effect of -p option To find out whether a given port is being used or not, use grep command.

$ sudo netstat -anotup | grep -E '(^Proto)|(8080)'

For Windows,

$ netstat -ano

For more about netstat, refer topics in Wikipedia.

Monitoring network traffic by interface card

watch -n 2 netstat -i

Capturing incoming HTTP request using tcpdump

$ sudo tcpdump -s 0 -A -i eth1 dst port 80

Identifying the user limits of current login or session

$ ulimit -a

Counting the number of currently open files to a specific process

$ lsof -a -p 12345 -d ^mem -d ^cwd -d ^rtd -d ^txt -d ^DEL | wc -l

or

$ ls -l /proc/12345/fd | wc -l

Readings

Diagnosing and Monitoring AIX

Identifying System Configuration

Using Common Shell Commands

Identifying the shell of your current login

To identify what shell a user is set to use by default, you can check SHELL variable.

$ echo $SHELL
/bin/bash
$ bin/ksh
$ echo $SHELL
/bin/bash

As the above example shows, SHELL variable contains the login default shell type not the one currently in use.

Hiding the output of command

To hide both the normal output and error output, redirect stdout and stderr to null device

% npm ls -g json >/dev/null 2>&1
% #or
% npm ls -g y18n &>/dev/null

To hide only the error output, redirect stderr to null device

% npm ls -g json 2>/dev/null

Listing files using find command excluding files with 'Permission denied'

When executing find command in simplest format, you may get lots of lines just saying that 'Permission denied'. Most cases, those are not what you want, and lots of permission denied lines can disturb you identifying the wanted result.

You can use stderr redirection to cut out permission denied files (or directories).

% find / -name '*.jar' 2>/dev/null

Finding files having specified name with full path

If you want to find files with extension of 'jar' and print them with full path, use find command with -exec operator like the following.

% find . -name '*.jar' -exec ls -l {} \;

For more about find command and -exec operator including strange '{}' or '\;' in the above example, refer the followings.

Finding files containing the specified word

% find /home |xargs grep "password"

For more about xargs, refer the followings.

Finding old directories or files to remove them

To find old files or directories and then work with them, use find command with -amin, -atime, -cmin, -ctime, -mmin or -mtime and -exec options.

% find . -maxdepth 1 -type d -ctime +10 -exec rm -Rf {} \;

Finding large files

To find large files(not directories) under current directory and list them in pages, use the following command.

% find . -type f -exec du -k {} 2>/dev/null \; | sort -nr | more

To filter out small files, you can use size option with find command, or to filter out some subdirectories you can redirect the result to grep command. The following command will list files whose size are more than 1 mega-byte under current directory recursively except the subdirectories starting with 'svn' in order of their size.

% find . -type f -size +1000000c -exec du -k {} 2>/dev/null \; | sort -nr | grep -E "\./svn.*" -v

Listing all files under a certain directory recursively

Using find command

% find /proc/sys -type f 2>/dev/null | more

Listing distinct file extensions of all files under a directory

% find . -type f -name "*.*" | sed -r 's/^.+(\.\w+)$/\1/' | sort | uniq

Counting files under a directory recurssively

% find . -type f -print | wc -l

Counting files in a tar file

% tar -tvf archive.tar | grep "^-.*" | wc -l

Inverse matching with grep command

To find lines not matching the specified patterns in a file, you can use -v option with grep command.

$ svn list -R http://.../repos1 | grep -v -E '(.*java|.*/)'

You don't need to be bothered to find out how to use complex negative patterns with regex.

Viewing files in octal or hexadecimal format - od

You can view non ascii base files in hexadecimal format using od command.

% od -A d -x journal.log

For more about od, refer the following.

Viewing file contents without line wrapping - less -S

% less -S known_hosts

Viewing the result of ps command without line wrapping

You can redirect the result to cat or less command, or use ww flag.

% ps auxf | cat
...
% ps auxfww
...
% ps auxf | less -+S

Viewing file contents without comments lines (starting with #)

% cat /etc/apt/sources.list | grep -P '^[^#].*'

or

% cat /etc/apt/sources.list | grep -Ev '(^#|^$)'

Sorting the file system usage result from the du command

You can sort the output of du command applying pipe to sort command.

% du -m | sort -n

For more about du and sort, read the followings.

Getting multiple files form the target URL using wget command

wget provide --accept or -A switch which can represent multiple files using comma separated list, wild card, or character class. But it's not that -A switch support regular expression.

$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '198[7-9].csv.bz2' http://stat-computing.org/dataexpo/2009/)"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '199[0-9].csv.bz2' http://stat-computing.org/dataexpo/2009/)"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '200[0-8].csv.bz2' http://stat-computing.org/dataexpo/2009/)"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A 'airports.csv, carriers.csv, plane-data.csv' http://stat-computing.org/dataexpo/2009/)"

For more, refer the following

Adding lines to a specific row of a file using sed command

$ sed -i '2 a export GOROOT=/usr/lib/go-1.8' .profile
$ sed -i '3 a export PATH=$PATH:$GOROOT/bin\n' .profile

Repeat command over piped targets using xargs

$ dpkg -l | grep "^rc" | awk '{print $2}' | xargs sudo dpkg --purge

Grep from a specific column

$ ls -l | awk '$3 == "root"'
...

$ netstat -antp | awk '$4 ~ /.*:(1580|1590|1600|1943|1953|1963|2080|2090|2100|2443|2453|2463|5505|5515|6506|6516)/'
...

$ watch -n 2 'netstat -antp | awk "\$4 ~ /.*:(1580|1590|1600|1943|1953|1963|2080|2090|2100|2443|2453|2463|5505|5515|6506|6516)$/"| sort -k 4'
...

Bash Programming

Directory containing the current script

The following script would work correctly with a directory containing spaces in its path.

script_dir=`dirname -- "$0"`
script_dir=$(cd "$script_dir" && pwd)

When the directory may contain spaces, $script_dir and its derivatives should always be double quoted like the following code

script_dir=`dirname -- "$0"`
script_dir=$(cd "$script_dir" && pwd)
readonly config="$script_dir/../generated/ripple-network-config.json"

cat "$config"

Looping array

For indexed array

fruits=(Apple Banana Kiwi)

for f in ${fruits[@]}; do
  echo $f
done;

# or looping in index order

for (( i = 0; i < ${#fruits[@]}; i++ )); do
  echo ${fruits[${i}]}
done;

For correlated arrays,

fruits=(Apple Banana Kiwi)
colors=(red yellow green)

for (( i=0; i<${#fruits[@]}; i++ )); do
  echo ${fruits[$i]} is ${colors[$i]}
done;

For associate array,

declare -Ar vm_params=( # kernel parameters for virtual memory
  [swappiness]=1
  [dirty_ratio]=10
  [dirty_background_ratio]=5
  [min_free_kbytes]=262144 #256MB
)

for param in "${!vm_params[@]}"; do
  echo "Updating 'vm.${param}' to '${vm_params[${param}]}'."
  sudo sysctl -wq vm.${param}=${vm_params[${param}]}

  if [ `grep -E "^\\s*vm.${param}\\s*=\\s*${vm_params[${param}]}.*" /etc/sysctl.conf | wc -l` -ne 1 ]; then
    sudo sed -r -i 's/^\s*(vm\.'"${param}"'.*)/#\1/g' /etc/sysctl.conf
    sudo sh -c "echo 'vm.${param}=${vm_params[${param}]}' >> /etc/sysctl.conf"
  fi
done

Looping arguments

Looping arguments by item,

declare command="curl -sSL ${url_base}/generate-tls-artifacts.sh | bash -s --"

for arg in "$@"; do
  if [ ${arg:0:1} == '-' ]; then
    command=${command}" ${arg}"
  else
    command=${command}" '${arg}'"
  fi
done

Looping arguments by index,

declare -A args
declare arg
for (( i = 1; i <= $#; i++ )); do
  arg=${@:$i:1}
  if [ ! ${arg:0:1} == '-' ]; then arg="'${arg}'"; fi
  args[$i]=${@:$i:1}
done
# echo ${args[@]}

2 dimentional array

declare -a m
m[0]='a b c d'
m[1]='e f g h'
m[2]='i j k l'
m[3]='m n o p'
m[4]='q r s t'

for r in "${m[@]}"; do
  echo $r
  for c in $r; do
    echo $c
  done
done

Read a file line by line

cat ./${infile} | while -r read line; do
  IFS=','; read -r -a fields <<< "${line}"
  blk_idx=${fields[0]}
  close_time=${fields[1]//\"/}
  tx_hash=${fields[2]//\"/}
 
  # ...
done;

awk script typical snippet

Use awk to structure target data table from the input source and process each row using while statement assigning each column to its own variable using read.
So typical script flow is awk ... | while read var1 var2 var3; do ...; done like the following real example.

awk -F, '{if (($1 ~ /vm[0-9]*/) && (substr($1, 3, length($1) - 1) + 0 < 97)) print $1 " " $2}' ../../vms.csv | while read vm ip; do

  no=${vm##vm}  # remove left 'vm'
  org_no=$(($(($((no - 1))/org_size)) + 1))
  peer_str="{\"requests\": \"grpcs://${ip}:7051\", \"events\": \"grpcs://${ip}:7053\", \"server-hostname\": \"peer${no}\", \"tls_cacerts\": \"tlsCerts/org${org_no}.com/tlsca${org_no}-cert.pem\"}"

  # Defines paired peer no : (vm1-vm2, vm3-vm4, ...)
  if [ $((no % 2)) -eq 1 ]; then
    no2=$((no + 1))
  else
    no2=$((no - 1))
  fi

  # For current peer vm
  sed -i -r 's#@@peer1@@#\"peer1\": '"$peer_str"',#' ./generated/vm${no}/channelConfig.json
  if [ $? -ne 0 ]; then
    echo "Fail to update 'peer1' part of './generated/vm${no}/channelConfig.json' file."
    exit 1
  else
    echo "Successfully updated 'peer1' part of './generated/vm${no}/channelConfig.json' file."
  fi

  # For paired peer vm
  sed -i -r 's#@@peer2@@#\"peer2\": '"$peer_str"'#' ./generated/vm${no2}/channelConfig.json
  if [ $? -ne 0 ]; then
    echo "Fail to update 'peer2' part of './generated/vm${no2}/channelConfig.json' file."
    exit 1
  else
    echo "Successfully updated 'peer2' part of './generated/vm${no2}/channelConfig.json' file."
  fi
done

parallel script typical snippet

For local tasks

dirs=('/boot' '/proc' '/sys' '/var' '/sbin');

for dir in ${dirs[@]}; do echo $dir; done | parallel --no-notice --bar --joblog /dev/stdout '
  files=$(sudo find {} -type f 2>/dev/null | wc -l);
  echo {} ":" ${files} "files"
'

For remote tasks

hosts=('192.168.100.1' '192.168.100.2' '192.168.100.3');

for host in ${hosts[@]}; do echo $host; done | parallel --no-notice --bar --joblog /dev/stdout '
  name=$(ssh -o StrictHostKeyChecking=no paul@{} echo "\`hostname\`");
  echo {} ":" ${name};
'

Using getopt to provide command line options

  1. readonly openssl_ver=`openssl version 2> /dev/null`
  2.  
  3. if [ $? -ne 0 ]; then
  4.   echo "OpenSSL is not installed or 'openssl' is not in the PATH."
  5.   echo "Check whether OpenSSL is installed or not, execute 'dpkg -l | grep -w openssl'."
  6.   echo "To install OpenSSL, execute 'sudo apt-get install openssl'."
  7.   exit 101
  8. fi
  9.  
  10. options=$(getopt -o hs:f: --long "subj:,filename:,help" --name 'generate-tls-artifacts-options' -- "$@");
  11.  
  12. if [ $? -ne 0 ]; then
  13.   echo "Unable to parse command line, For help, try '-h' option."
  14.   echo ""
  15.   exit 300
  16. fi
  17.  
  18. eval set -- "$options"
  19.  
  20. declare filename='test-tls'  # only file name part without extension or directory
  21. declare subj=
  22. while true; do
  23.   case "$1" in
  24.     -h | --help )
  25.       echo "Show help"
  26.       exit 0
  27.       shift ;;
  28.     -s | --subj )
  29.       if [ -z "$2" ]; then
  30.         echo "-s or --subj option requires argument like '-s \"/C=ZZ/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown\"'."
  31.         exit 301
  32.       else
  33.         subj=${2## }
  34.         subj=${subj%% }
  35.       fi
  36.       shift 2 ;;
  37.     -f | --filename )
  38.       if [ -z "$2" ]; then
  39.         echo "-f or --filename option requires argument like '-s test-tls-server'."
  40.         exit 302
  41.       else
  42.         filename=$2
  43.       fi
  44.       shift 2 ;;
  45.     -- ) shift; break ;;
  46.    esac
  47. done
  48.  
  49. # echo 'subj='${subj}
  50. # echo 'filename='${filename}
  51.  
  52. if [ -z "${subj}" ]; then
  53.   subj="/C=ZZ/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown"
  54.   echo "No subject (identity for the generated key and certifiate) is specified."
  55.   echo "Default subject '/C=ZZ/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown' will be used."
  56.   echo "To specify subejct use -s or --subj option. For more, refer help using -h option."
  57.   echo ""
  58. fi

Managing Packages

Ubuntu

Installing a new software package
  1. Update the package information
  2. Check whether or not the software package is already installed
  3. Search and review the software package to install
  4. Install or upgrade the software package
  5. (Optionally) Confirm all the files installed by the package

Not using apt, which is preferred 'cause apt doesn't support in-script usage well.

$ sudo apt-get update                                 # update package information
...
$ dpkg -l | awk '{print $2}' | grep golang-1.8   # check previous installation
...
$ sudo apt-cache --names-only search ^golang | more   # search available packages
...
$ sudo apt-cache show golang-1.8                      # review the software to install
...
$ sudo apt-get install golang-1.8                     # install the software
...
$ sudo apt-file list golang-1.8                       # confirm all the files installed
...
$ dpkg-query -L golang-1.8                            # another way to confirm all the files installed

Using apt

$ sudo apt update                          # update package information
...
$ sudo apt list *golang-1.8* --installed   # check previous installation
...
$ sudo apt search ^golang | more           # search available packages
...
$ sudo apt show golang-1.8                 # review the software to install
...
$ sudo apt install golang-1.8              # install the software
...
$ dpkg-query -L golang-1.8                 # confirm all the files installed
  • References
    • apt-get
    • apt-cache
    • apt-file : searching files in packages for the APT package management system.
    • apt : provides a high-level commandline interface for the package management system.
Installing a software package specifying version with wildcard
$ sudo apt-get install nodejs=6.10.2*
Listing all installed packages
$ sudo dpkg -l | awk '{print $2}'

Or

$ sudo apt list --installed

Debian

$ sudo dpkg -l | grep -E '\stelegraf'
...
$ sudo dpkg -i telegraf_1.8.2-1_amd64.deb
...

Cent OS

$ sudo yum list installed | grep nginx    # check 'nginx' is installed or not

$

Misc

Preceed configuration for Ubuntu 16.04 automatic installation

### References
###   https://help.ubuntu.com/16.04/installation-guide/amd64/apb.html
###   https://www.debian.org/releases/stable/amd64/apb.html.en
###   https://help.ubuntu.com/16.04/installation-guide/example-preseed.txt

### Localization
# Language, Country and Locales
d-i debian-installer/language string en
d-i debian-installer/country string KR
d-i debian-installer/locale string en_US.UTF-8
d-i localechooser/supported-locales multiselect en_US.UTF-8

# Keyboard
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/xkb-keymap select us


### Network
d-i netcfg/enable boolean false
d-i netcfg/disable_autoconfig boolean true
d-i netcfg/get_ipaddress string 192.168.1.42
d-i netcfg/get_netmask string 255.255.255.0
d-i netcfg/get_gateway string 192.168.1.1
d-i netcfg/get_nameservers string 192.168.1.1
d-i netcfg/confirm_static boolean true
d-i netcfg/hostname string chainz001


### Clock and Time-zone
d-i clock-setup/utc boolean true
d-i time/zone string Asia/Seoul
d-i clock-setup/ntp boolean false


### Account
d-i passwd/root-login boolean false
d-i passwd/make-user boolean tue

d-i passwd/user-fullname string Blockchain User
d-i passwd/username string chainz
d-i passwd/user-password password skccz
d-i passwd/user-password-crypted password [crypt(3) hash]
d-i passwd/user-uid string 1010
d-i passwd/user-default-groups string audio cdrom video
d-i user-setup/allow-password-weak boolean true
d-i user-setup/encrypt-home boolean false


### Partitioning
d-i partman-auto/disk string /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde
d-i partman-auto/method string lvm

d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true

# d-i partman-auto-lvm/guided_size string max

### https://help.ubuntu.com/16.04/installation-guide/amd64/apc.html
### https://help.ubuntu.com/16.04/installation-guide/amd64/apcs03.html
### https://sources.debian.org/src/debian-installer/20171204/doc/devel/partman-auto-recipe.txt/#L195
### https://www.claudiokuenzler.com/blog/513/debian-ubuntu-preseed-create-two-volume-groups-same-disk
d-i partman-auto/expert_recipe string \
  lvm_wo_raid :: \
    512 512 512 ext3 \
      $primary{ } \
      $bootable{ } \
      device{ /dev/sda1 } \
      method{ format } \
      format{ } \
      use_filesystem{ } \
      filesystem{ ext3 } \
      mountpoint{ /boot } \
      label{ boot } \
    . \
    2048 2048 -1 ext3 \
      $defaultignore{ } \
      $primary{ } \
      method{ lvm } \
      device{ /dev/sda2 } \
      vg_name{ vg1 } \
    . \
    2048 2048 -1 ext3 \
      $defaultignore{ } \
      $primary{ } \
      method{ lvm } \
      device{ /dev/sdb } \
      vg_name{ vg1 } \
    . \
    2048 2048 -1 ext3 \
      $defaultignore{ } \
      $primary{ } \
      method{ lvm } \
      device{ /dev/sdc } \
      vg_name{ vg1 } \
    . \
    2048 2048 -1 ext3 \
      $defaultignore{ } \
      $primary{ } \
      method{ lvm } \
      device{ /dev/sdd } \
      vg_name{ vg1 } \
    . \
    2048 2048 -1 ext3 \
      $defaultignore{ } \
      $primary{ } \
      method{ lvm } \
      device{ /dev/sde } \
      vg_name{ vg1 } \
    . \    
    6144 6144 6144 linux-swap \
      $lvmok{ } \
      in_vg{ vg1 } \
      lv_name{ swap } \
      method{ swap } \
      format { } \
    . \
    6144 6144 -1 ext4 \
      $lvmok{ } \
      in_vg{ vg1 } \
      lv_name{ root } \
      method{ format } \
      format{ } \
      use_filesystem{ } \
      filesystem{ ext4 } \
      mountpoint{ / } \
      label{ root } \
    . \
    6144 6144 6144 ext4 \
      $lvmok{ } \
      in_vg{ vg1 } \
      lv_name{ var } \
      method{ format } \
      format{ } \
      use_filesystem{ } \
      filesystem{ ext4 } \
      mountpoint{ /var } \
      label{ var } \
    . \
    100 100 100 ext4 \
      $lvmok{ } \
      in_vg{ vg1 } \
      lv_name{ tmp } \
      method{ format } \
      format{ } \
      use_filesystem{ } \
      filesystem{ ext4 } \
      mountpoint{ /tmp } \
      label{ tmp } \
    .
   
d-i partman-auto/choose_recipe select lvm_wo_raid
 
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

# "uuid", "traditional" or "label"
#d-i partman/mount_style select uuid


### Base system installation

d-i base-installer/install-recommends boolean true
d-i base-installer/kernel/image string linux-generic


### Apt setup

d-i apt-setup/restricted boolean true
d-i apt-setup/universe boolean true
d-i apt-setup/backports boolean true

# Uncomment this if you don't want to use a network mirror.
#d-i apt-setup/use_mirror boolean false

d-i apt-setup/services-select multiselect security
d-i apt-setup/security_host string security.ubuntu.com
d-i apt-setup/security_path string /ubuntu

d-i apt-setup/local0/repository string \
       http://local.server/ubuntu xenial main
d-i apt-setup/local0/comment string local server
d-i apt-setup/local0/source boolean true
d-i apt-setup/local0/key string http://local.server/key

d-i debian-installer/allow_unauthenticated boolean false
#d-i apt-setup/multiarch string amd64


### Package selection

# ''We recommend always including the standard task.''
tasksel tasksel/first multiselect standard

d-i pkgsel/language-packs multiselect en, ko

# "none" : no automatic updates
# "unattended-upgrades" : install security updates automatically
# "landscape" : manage system with Landscape
d-i pkgsel/update-policy select none

popularity-contest popularity-contest/participate boolean false

# By default, the system's locate database will be updated after the
# installer has finished installing most packages. This may take a while, so
# if you don't want it, you can set this to "false" to turn it off.
#d-i pkgsel/updatedb boolean true


### Boot loader installation
### @TODO What if with no bootloader ?

d-i grub-installer/skip boolean false
d-i lilo-installer/skip boolean true

d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true

d-i grub-installer/bootdev  string /dev/sda
#d-i grub-installer/bootdev string default

#d-i debian-installer/add-kernel-opts string nousb


### Finishing up the installation

d-i finish-install/reboot_in_progress note
#d-i cdrom-detect/eject boolean false

d-i debian-installer/exit/halt boolean false
d-i debian-installer/exit/poweroff boolean false
Community content is available under CC-BY-SA unless otherwise noted.