Introduction to /etc/passwd and /etc/shadow files in Linux systems (Debian/Ubuntu/CentOS/RHEL etc.)

Linux operating systems store all username and password (including administrators/root) in /etc/passwd and /etc/shadow file.

/etc/passwd

Each user has a line of corresponding record which records basic attributes. Only root/administrators can modify it. All other users have read only access to it.

/etc/shadow

As name suggested, this file is like shadow of “passwd” file. The record in “shadow” file is corresponding to the records in “passwd” file. Records is “shadow” file is automatically produced by “pwconv” command based on “passwd” file. Only root/administrators have read and write access to “shadow” file, other users can’t read it.

File permission for passwd and shadow
File permission for passwd and shadow

About /etc/passwd

sudo vi /etc/passwd
partial passwd file
partial passwd file
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin

There are 7 columns for each record

ColumnDescription
1Username
2Placeholder, x = password is required to login, empty = password is not required to login
3User UID
4User GID
5Extra information, Full name, contact information etc.
6Home directory
7Login shell, /bin/bash = Login to system shell enabled, /sbin/nologin = User can’t login

About /etc/shadow

sudo vi /etc/shadow
partial shadow file
partial shadow file
root:!:18313:0:99999:7:::
daemon:*:18313:0:99999:7:::
bin:*:18313:0:99999:7:::
sys:*:18313:0:99999:7:::
sync:*:18313:0:99999:7:::
games:*:18313:0:99999:7:::
man:*:18313:0:99999:7:::
lp:*:18313:0:99999:7:::
mail:*:18313:0:99999:7:::
news:*:18313:0:99999:7:::

There are 8 columns for each record

ColumnDecription
1Username
2Password (!! = no password, encrypted if password is set)
3Days between last change of password and 01/01/1970
4Minimum password age (Validated days)
5Maximum password age (Validated days)
6Buffer time (Days) after the password is expired (After the password is expired, for how many days the user can change the password, old password can’t be used to login again during this period of time)
7Number of days after password expires that account is disabled
8Date which the account is disabled (Days since 01/01/1979)
9Not used yet

How to: Run Linux commands with time limit/timeout (Kill process/command after some time)

Sometimes we want to stop or kill the command after a period of time, so that we don’t get stuck with that command and wasting resources etc. To specify timeout or time limit for Linux command, we can use timeout command

Command Usage/Parameters

timeout [OPTION] DURATION COMMAND [ARG]...

DURATION is integer or floating point with unit

s: Seconds (Default)

m: Minutes

h: Hours

d: Days

Without units appended, by default it is considered as seconds.

If the DURATION is 0, the timeout is disabled.

Basic Usage

Timeout ping command after 3 seconds

timeout 3 ping 127.0.0.1
timeout 3 ping 127.0.0.1
timeout 3 ping 127.0.0.1

Timeout ping command after 3 minutes

timeout 3m ping 127.0.0.1

Timeout ping command after 3 days

timeout 1d ping 127.0.0.1

Timeout ping command after 3.2 seconds

timeout 3.2s ping 127.0.0.1

Send specific signal after timeout

By default if signal is not specified, timeout command will use “SIGTERM” signal after timeout. We can use -s (-signal) switch to specific which signal to send after timeout

e.g. Send SIGKILL signal to ping command after 3 seconds

sudo timeout -s SIGKILL 3s ping 127.0.0.1
sudo timeout -s SIGKILL 3s ping 127.0.0.1
sudo timeout -s SIGKILL 3s ping 127.0.0.1

We can use the name of the signal or the number of the signal

e.g. We can use 9 as SIGKILL to achieve same result

sudo timeout -s 9 3s ping 127.0.0.1
sudo timeout -s 9 3s ping 127.0.0.1
sudo timeout -s 9 3s ping 127.0.0.1

To list all acceptable signal, we can use kill -l to find out

kill -l
[email protected]:~# kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX
kill -l
kill -l

Stop frozen process

SIGTERM, the default signal can be ignored by some processes, thus the program will keep running. To make sure the process is killed, we can use -k (–kill after) switch with specified time. When the time limited reached, force to kill the process.

e.g. Let the shell script run for 2 minutes, if it did not exit, then kill after 5 seconds

timeout -k 5s 2m sh test.sh

By default the timeout command will run in background, if we want to run it in foreground, refer to following example

timeout --foreground 2m ./test.sh

timeout help

Usage: timeout [OPTION] DURATION COMMAND [ARG]...
  or:  timeout [OPTION]
Start COMMAND, and kill it if still running after DURATION.
Mandatory arguments to long options are mandatory for short options too.
      --preserve-status
                 exit with the same status as COMMAND, even when the
                   command times out
      --foreground
                 when not running timeout directly from a shell prompt,
                   allow COMMAND to read from the TTY and get TTY signals;
                   in this mode, children of COMMAND will not be timed out
  -k, --kill-after=DURATION
                 also send a KILL signal if COMMAND is still running
                   this long after the initial signal was sent
  -s, --signal=SIGNAL
                 specify the signal to be sent on timeout;
                   SIGNAL may be a name like 'HUP' or a number;
                   see 'kill -l' for a list of signals
  -v, --verbose  diagnose to stderr any signal sent upon timeout
      --help     display this help and exit
      --version  output version information and exit
DURATION is a floating point number with an optional suffix:
's' for seconds (the default), 'm' for minutes, 'h' for hours or 'd' for days.
A duration of 0 disables the associated timeout.
If the command times out, and --preserve-status is not set, then exit with
status 124.  Otherwise, exit with the status of COMMAND.  If no signal
is specified, send the TERM signal upon timeout.  The TERM signal kills
any process that does not block or catch that signal.  It may be necessary
to use the KILL (9) signal, since this signal cannot be caught, in which
case the exit status is 128+9 rather than 124.
GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
Full documentation at: <https://www.gnu.org/software/coreutils/timeout>
or available locally via: info '(coreutils) timeout invocation'

Linux Command Line/ Terminal Disk Space Usage tool (Find largest folder/file)

For finding largest file/folder or showing disk space usage on Windows, refer to this one: How to: Find Largest file on Windows, Windows 7, Windows 10, Microsoft Windows, Windows Server (Disk Space Usage)

Ncdu (NCurses Disk Usage) is a command line tool to view and analyse disk space usage on Linux.

It can be easily installed on most Linux systems with package management system.

ncdu on Kali Linux 2020
ncdu on Kali Linux 2020

Debian/Kali Linux/Ubuntu etc. Linux installation

sudo apt install ncdu -y
 
OR
 
sudo aptitude ncdu -y

RHEL/CentOS/Fedora etc. Linux installation

If EPEL repo is not installed yet, we have to install EPEL repo first

sudo yum -y install epel-release

Next, we can now install ncdu

sudo yum install ncdu -y

Using ncdu is simple.

Show current working directory info

ncdu

Show info for a folder e.g. “/etc”

ncdu /etc

To show more info about a folder while in ncdu, press “i” key (Press “i” again to dismiss)

ncdu - i
ncdu – i

Press Shift + ? to show help document while in ncdu

ncdu help
ncdu help

Press “q” key to quit menus and the ncdu program


How to: Use shortcut keys/Key combinations in Linux Terminal

1 Tab

When entering command, enter beginning of the command, file name or folder name or command option then press “Tab” key, it will complete the rest for you automatically or show all possible results.

2 Ctrl + C

Terminate/Kill the command or process, it will terminate the running process immediately. (signal SIGINT). It can be intercepted by a program, thus the program can clean itself up before exiting or not exit at all.

3 Ctrl + Z

Suspending a process by sending the SIGSTOP signal, it cannot be intercepted by the program.

4 Ctrl + D

Exit the current terminal. If you are using SSH, it will close it. If you are using a terminal directly, it will close the terminal window.

5 Ctrl + L

Clear terminal screen, same effect as “clear” command

6 Ctrl + A

Move the type cursor to the beginning of the line (Same as pressing “Home” key on keyboard)

7 Ctrl + E

Move the type cursor to the end of the line (Same as pressing “End” key on keyboard)

8 Ctrl + U

Wipe the line and move the type cursor to the beginning of the line (Instead of use “Backspace” key to clear the line slowly)

9 Ctrl + K

Wipe the content from the type cursor to the end of the line

10 Ctrl + W

Clear a word

Before Ctrl + W

Before Ctrl + W
Before Ctrl + W

After Ctrl + W

After Ctrl + W
After Ctrl + W

11 Ctrl + Y

It will paste text removed by Ctrl + U, Ctrl + U and Ctrl + K. If you have deleted text by mistake, this will be helpful.

12 Ctrl + P

Review last command, use repetitively to go back further. Many Terminal provides this review function by PageUp key as well. Some provide the review function by using up arrow key as well (↑).

13 Ctrl + N

Similar usage as Ctrl + P but opposite direction, this command navigate to more recent commands. Many Terminal provides this review function by PageDown key as well. Some provide the review function by using down arrow key as well (↓).

14 Ctrl + R

Used for search history commands

Bonus:

Alternatively, we can use “history” command to show all history command

To search from history command, we can use “history | grep searchTerm”


What’s the difference between CentOS Linux vs CentOS Stream

1 If you just want to test CentOS or use it as server: Download CentOS Linux

2 If you want to know what’s difference, read on

In short, CentOS Stream is in-between Fedora and RHEL, it provides clearner vision of what the next version of RHEL will be, give developer time to plan and create next-generation applications, so that they are compatible with future RHEL version. Whereas the CentOS Linux is the normal version we should use if we just want to setup another server.

Resources


Useful commands to look up system information, install package etc. (Linux, Debian, Ubuntu, Kali Linux, RedHat, CentOS etc.)

System

uname -a               #Show kernel Info/Operating System/CPU Info
head -n 1 /etc/issue   #Show operating system Name
cat /proc/cpuinfo      #Show detail about CPU
hostname               #Show host name
lspci -tv              #List all PCI devices
lsusb -tv              #List all USB devices
lsmod                  #List all loaded kernel modules
env                    #Show environment variables

Processes

top        #Show real-time status of processes
ps -ef     #Show all processes

Services

systemctl list-units                        #List all running services
 systemctl list-units -a                        #List all services, including inactive units
 systemctl list-units -a --state=inactive    #List all inactive units

Users

w                       #Show active users
id <UserName>           #Show info about the user
last                    #Show user login log
cut -d: -f1 /etc/passwd #Show all users
cut -d: -f1 /etc/group  #Show all groups
crontab -l              #Show all cron jobs (Scheduled jobs) for current user

Resources

free -m                       #Show RAM, SWAP usage
df -h                         #Show partition usage
du -sh                        #Show folder/directory size
grep MemTotal /proc/meminfo   #Show total RAM
grep MemFree /proc/meminfo    #Show free/available RAM
uptime                        #Show system uptime, users, load
cat /proc/loadavg             #Show system load

Disk & Partition

mount | column -t    #Show mounted partition info
fdisk -l             #Show all partitions
swapon -s            #Show all swap partition info

Networking

ifconfig        #Show all interface properties
iptables -L     #Show firewall (iptable) configuration
route -n        #Show routing table
netstat -lntp   #Show all listening ports
netstat -antp   #Show all established connections
netstat -s      #Show statistics about Ethernet

apt-get

update            #Update package list
upgrade           #Upgrade packages
install           #Install new packages
remove            #Remove packages
autoremove        #Remove packages that were automatically installed to satisfy dependencies for other packages and are now no longer needed
purge             #Remove package and configuration files
source            #fetch source packages
build-dep         #Install all dependencies for 'packagename'
dist-upgrade      #Upgrade distro
full-upgrade      #Upgrade distro (For newer 'apt')
dselect-upgrade   #Resolves the delta between the currently-installed packages and the states requested in the list of available packages, and performs the necessary actions to reconcile the two.
clean             #Clears out the local repository of retrieved package files
autoclean         #Another method used to clear out the local repository of downloaded package files, just like clean . The difference between clean and autoclean is that the latter only removes package files that can no longer be downloaded from their sources, and are very likely to be useless.
check             #It does an update of the package lists and checks for broken dependencies
-h        #Local help document
-q        #Output to log (Do not indicate progress)
-qq       #Only output errors
-d        #Only download (Do not Extract or Install)
-s        #Simulate the command, does not install the package for real
-y        #Answer Yes for all commands
-f        #Try to fix dependency issues
-m        #Try to continue even the archive can't be located
-u        #Also show the list of upgrade package
-b        #Compile after getting the source (-V to show detailed version number)
-c=?      #Show the configuration file
-o=?      #Set custom/arbitrary configuration e.g. -d dir::cache=/tmp
#Examples
 
apt-cache search PACKAGE                 #Search package
apt-cache show PACKAGE                   #Get details about the package (Manual, size, version etc.)
apt list --installed                     #List all installed packages
apt-get install PACKAGE                  #Install the packag
apt-get install PACKAGE --reinstall      #Reinstall the package
apt-get -f install                       #Repair install (-f = –fix-missing)
apt-get remove PACKAGE                   #Remove package
apt-get remove PACKAGE --purge           #Remove package and configuration files
apt-get update                           #Update source info (repositories and PPAs)
apt-get upgrade                          #Upgrade installed packages
apt-get dist-upgrade                     #Upgrade system
apt full-upgrade                         #Upgrade system (Newer apt)
apt-get dselect-upgrade                  #Use dselect to upgrade
apt-cache depends PACKAGE                #List the dependencies in human form
apt-cache showpkg PACKAGE                #find the details about dependencies with
apt-cache rdepends PACKAGE               #Searches through the APT cache to locate dependencies for an application, and knows how to emulate the results
apt-get build-dep PACKAGE                #Install all dependencies for 'PACKAGE'
apt-get source PACKAGE                   #Download the source for this PACKAGE
apt-get clean                            #Clears out the local repository of retrieved package files
apt-get autoclean                        #Another method used to clear out the local repository of downloaded package files, just like clean . The difference between clean and autoclean is that the latter only removes package files that can no longer be downloaded from their sources, and are very likely to be useless.
apt-get check                            #It does an update of the package lists and checks for broken dependencies

RPM

rpm -qa                                      #List all installed packages
rpm -qpi Linux-1.4-6.i368.rpm                #Show detail about the package
rpm -qpl Linux-1.4-6.i368.rpm                #Show dependencies
rpm -ivh Linux-1.4-6.i368.rpm                #Begin to inatll
rpm -ivh -replacepkgs Linux-1.4-6.i368.rpm   #Force to install
rpm -e PACKAGE                               #Uninstall package
rpm -uvh filename                            #Upgrade by patch (The filename must be the upgrade patch not package)
rpm -i https://contoso.com/packagename.rpm   #Install package from internet
rpm -Va                                      #Search for corrupted filename
rpm -qf filename                             #Find out which package does the file belongs to