Introduction to /etc/passwd and /etc/shadow files in Linux systems (Debian/Ubuntu/CentOS/RHEL etc.)

Linux operating systems store all username and password (including administrators/root) in /etc/passwd and /etc/shadow file.

/etc/passwd

Each user has a line of corresponding record which records basic attributes. Only root/administrators can modify it. All other users have read only access to it.

/etc/shadow

As name suggested, this file is like shadow of “passwd” file. The record in “shadow” file is corresponding to the records in “passwd” file. Records is “shadow” file is automatically produced by “pwconv” command based on “passwd” file. Only root/administrators have read and write access to “shadow” file, other users can’t read it.

File permission for passwd and shadow
File permission for passwd and shadow

About /etc/passwd

sudo vi /etc/passwd
partial passwd file
partial passwd file
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin

There are 7 columns for each record

ColumnDescription
1Username
2Placeholder, x = password is required to login, empty = password is not required to login
3User UID
4User GID
5Extra information, Full name, contact information etc.
6Home directory
7Login shell, /bin/bash = Login to system shell enabled, /sbin/nologin = User can’t login

About /etc/shadow

sudo vi /etc/shadow
partial shadow file
partial shadow file
root:!:18313:0:99999:7:::
daemon:*:18313:0:99999:7:::
bin:*:18313:0:99999:7:::
sys:*:18313:0:99999:7:::
sync:*:18313:0:99999:7:::
games:*:18313:0:99999:7:::
man:*:18313:0:99999:7:::
lp:*:18313:0:99999:7:::
mail:*:18313:0:99999:7:::
news:*:18313:0:99999:7:::

There are 8 columns for each record

ColumnDecription
1Username
2Password (!! = no password, encrypted if password is set)
3Days between last change of password and 01/01/1970
4Minimum password age (Validated days)
5Maximum password age (Validated days)
6Buffer time (Days) after the password is expired (After the password is expired, for how many days the user can change the password, old password can’t be used to login again during this period of time)
7Number of days after password expires that account is disabled
8Date which the account is disabled (Days since 01/01/1979)
9Not used yet

How to: Upgrade Roundcube webmail easily with terminal/command

Roundcube is an open source web/online MUA (mail user agent)

Note!: Don’t forget to change the download link and folder name for wget and Install/Update (Step 2 and 4)

#1 Switch to /tmp directory
cd /tmp
 
#2 Download the package with wget
wget https://github.com/roundcube/roundcubemail/releases/download/1.4.3/roundcubemail-1.4.3-complete.tar.gz
 
#3 Extract the package
tar xf roundcubemail-*.tar.gz
 
#4 Install/Update
./roundcubemail-1.4.3/bin/installto.sh /destinationFolder/roundcube

Extended Reading

MUA (mail user agent) Is used for users to read, compose, and send email. Examples of MUAs are Roundcube, SquirrelMail, pine, Microsoft Outlook etc.

MTA (mail transfer agent) Is used for the transport, delivery, and forwarding of email. Examples of MTAs like SMTP servers are POSTFIX, sendmail etc.


How to: Run Linux commands with time limit/timeout (Kill process/command after some time)

Sometimes we want to stop or kill the command after a period of time, so that we don’t get stuck with that command and wasting resources etc. To specify timeout or time limit for Linux command, we can use timeout command

Command Usage/Parameters

timeout [OPTION] DURATION COMMAND [ARG]...

DURATION is integer or floating point with unit

s: Seconds (Default)

m: Minutes

h: Hours

d: Days

Without units appended, by default it is considered as seconds.

If the DURATION is 0, the timeout is disabled.

Basic Usage

Timeout ping command after 3 seconds

timeout 3 ping 127.0.0.1
timeout 3 ping 127.0.0.1
timeout 3 ping 127.0.0.1

Timeout ping command after 3 minutes

timeout 3m ping 127.0.0.1

Timeout ping command after 3 days

timeout 1d ping 127.0.0.1

Timeout ping command after 3.2 seconds

timeout 3.2s ping 127.0.0.1

Send specific signal after timeout

By default if signal is not specified, timeout command will use “SIGTERM” signal after timeout. We can use -s (-signal) switch to specific which signal to send after timeout

e.g. Send SIGKILL signal to ping command after 3 seconds

sudo timeout -s SIGKILL 3s ping 127.0.0.1
sudo timeout -s SIGKILL 3s ping 127.0.0.1
sudo timeout -s SIGKILL 3s ping 127.0.0.1

We can use the name of the signal or the number of the signal

e.g. We can use 9 as SIGKILL to achieve same result

sudo timeout -s 9 3s ping 127.0.0.1
sudo timeout -s 9 3s ping 127.0.0.1
sudo timeout -s 9 3s ping 127.0.0.1

To list all acceptable signal, we can use kill -l to find out

kill -l
[email protected]:~# kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX
kill -l
kill -l

Stop frozen process

SIGTERM, the default signal can be ignored by some processes, thus the program will keep running. To make sure the process is killed, we can use -k (–kill after) switch with specified time. When the time limited reached, force to kill the process.

e.g. Let the shell script run for 2 minutes, if it did not exit, then kill after 5 seconds

timeout -k 5s 2m sh test.sh

By default the timeout command will run in background, if we want to run it in foreground, refer to following example

timeout --foreground 2m ./test.sh

timeout help

Usage: timeout [OPTION] DURATION COMMAND [ARG]...
  or:  timeout [OPTION]
Start COMMAND, and kill it if still running after DURATION.
Mandatory arguments to long options are mandatory for short options too.
      --preserve-status
                 exit with the same status as COMMAND, even when the
                   command times out
      --foreground
                 when not running timeout directly from a shell prompt,
                   allow COMMAND to read from the TTY and get TTY signals;
                   in this mode, children of COMMAND will not be timed out
  -k, --kill-after=DURATION
                 also send a KILL signal if COMMAND is still running
                   this long after the initial signal was sent
  -s, --signal=SIGNAL
                 specify the signal to be sent on timeout;
                   SIGNAL may be a name like 'HUP' or a number;
                   see 'kill -l' for a list of signals
  -v, --verbose  diagnose to stderr any signal sent upon timeout
      --help     display this help and exit
      --version  output version information and exit
DURATION is a floating point number with an optional suffix:
's' for seconds (the default), 'm' for minutes, 'h' for hours or 'd' for days.
A duration of 0 disables the associated timeout.
If the command times out, and --preserve-status is not set, then exit with
status 124.  Otherwise, exit with the status of COMMAND.  If no signal
is specified, send the TERM signal upon timeout.  The TERM signal kills
any process that does not block or catch that signal.  It may be necessary
to use the KILL (9) signal, since this signal cannot be caught, in which
case the exit status is 128+9 rather than 124.
GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
Full documentation at: <https://www.gnu.org/software/coreutils/timeout>
or available locally via: info '(coreutils) timeout invocation'

How to: Use curl to submit Google Form

Before crafting the curl command, we need to find the “name” attributes for the text boxes and the value for them, we can use built-in developer tools from different browsers to find that out, record the value of the “name”

Now we can craft the curl command

curl https://docs.google.com/forms/d/e/stringoForTheGoogleForm/formResponse -d ifq -d "entry.1773835858=text1" -d "entry.784619947=text2" -d submit=Submit

Make sure after the string from copied URL we put “/formResponse”, it there is “viewform” at the end of the URL replace that with “formResponse”


“entry.1773835858” and “entry.784619947” are the “name” value for two text boxes

Now we can use the above curl command to submit the Google Form


Linux Command Line/ Terminal Disk Space Usage tool (Find largest folder/file)

For finding largest file/folder or showing disk space usage on Windows, refer to this one: How to: Find Largest file on Windows, Windows 7, Windows 10, Microsoft Windows, Windows Server (Disk Space Usage)

Ncdu (NCurses Disk Usage) is a command line tool to view and analyse disk space usage on Linux.

It can be easily installed on most Linux systems with package management system.

ncdu on Kali Linux 2020
ncdu on Kali Linux 2020

Debian/Kali Linux/Ubuntu etc. Linux installation

sudo apt install ncdu -y
 
OR
 
sudo aptitude ncdu -y

RHEL/CentOS/Fedora etc. Linux installation

If EPEL repo is not installed yet, we have to install EPEL repo first

sudo yum -y install epel-release

Next, we can now install ncdu

sudo yum install ncdu -y

Using ncdu is simple.

Show current working directory info

ncdu

Show info for a folder e.g. “/etc”

ncdu /etc

To show more info about a folder while in ncdu, press “i” key (Press “i” again to dismiss)

ncdu - i
ncdu – i

Press Shift + ? to show help document while in ncdu

ncdu help
ncdu help

Press “q” key to quit menus and the ncdu program


How to: Use shortcut keys/Key combinations in Linux Terminal

1 Tab

When entering command, enter beginning of the command, file name or folder name or command option then press “Tab” key, it will complete the rest for you automatically or show all possible results.

2 Ctrl + C

Terminate/Kill the command or process, it will terminate the running process immediately. (signal SIGINT). It can be intercepted by a program, thus the program can clean itself up before exiting or not exit at all.

3 Ctrl + Z

Suspending a process by sending the SIGSTOP signal, it cannot be intercepted by the program.

4 Ctrl + D

Exit the current terminal. If you are using SSH, it will close it. If you are using a terminal directly, it will close the terminal window.

5 Ctrl + L

Clear terminal screen, same effect as “clear” command

6 Ctrl + A

Move the type cursor to the beginning of the line (Same as pressing “Home” key on keyboard)

7 Ctrl + E

Move the type cursor to the end of the line (Same as pressing “End” key on keyboard)

8 Ctrl + U

Wipe the line and move the type cursor to the beginning of the line (Instead of use “Backspace” key to clear the line slowly)

9 Ctrl + K

Wipe the content from the type cursor to the end of the line

10 Ctrl + W

Clear a word

Before Ctrl + W

Before Ctrl + W
Before Ctrl + W

After Ctrl + W

After Ctrl + W
After Ctrl + W

11 Ctrl + Y

It will paste text removed by Ctrl + U, Ctrl + U and Ctrl + K. If you have deleted text by mistake, this will be helpful.

12 Ctrl + P

Review last command, use repetitively to go back further. Many Terminal provides this review function by PageUp key as well. Some provide the review function by using up arrow key as well (↑).

13 Ctrl + N

Similar usage as Ctrl + P but opposite direction, this command navigate to more recent commands. Many Terminal provides this review function by PageDown key as well. Some provide the review function by using down arrow key as well (↓).

14 Ctrl + R

Used for search history commands

Bonus:

Alternatively, we can use “history” command to show all history command

To search from history command, we can use “history | grep searchTerm”


What’s the difference between CentOS Linux vs CentOS Stream

1 If you just want to test CentOS or use it as server: Download CentOS Linux

2 If you want to know what’s difference, read on

In short, CentOS Stream is in-between Fedora and RHEL, it provides clearner vision of what the next version of RHEL will be, give developer time to plan and create next-generation applications, so that they are compatible with future RHEL version. Whereas the CentOS Linux is the normal version we should use if we just want to setup another server.

Resources


Basics about Network configuration in Linux, IP commands, configuration files etc.

Table of Contents

1 Some useful basic IP commands

1.1 Use network/Interface configuration files to make permanent changes.

For CentOS/RHEL/Fedora etc.

File: /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"
BOOTPROTO=static
ONBOOT=yes TYPE="Ethernet"
IPADDR=10.0.0.10
NAME="System eth0"
HWADDR=00:53:78:2C:7D:9E
GATEWAY=10.0.0.1

For Debian/Ubuntu/Kali Linux etc.

File: /etc/network/interfaces

auto eth0
iface eth0 inet static
address 10.0.0.10
netmask 255.255.255.0
gateway 10.0.0.1

Restart network services to make the changes take effect

sudo /etc/init.d/networking restart
 
OR
 
sudo service restart networking
 
OR
 
systemctl restart networking

1.2 Assign IP address to a specific interface (eth0 in this example) (nonpersistent, will be lost after system reboot)

sudo ip addr add 10.0.0.10 dev eth0

1.3 Remove IP address from a specific interface

sudo ip addr del 10.0.0.10/24 dev eth0

1.4 Check IP address

sudo ip addr
 
OR
 
sudo ip addr show
 
OR
 
sudo ifconfig

1.5 Enable Network interface

sudo ip link set eth0 up

1.6 Disable Network interface

sudo ip link set eth0 down

1.7 Check routing table

sudo ip route show

1.8 Add Static route

sudo ip route del 10.0.0.0/24

1.9 Add persistent static routes

For CentOS/RHEL/Fedora etc.

File: /etc/sysconfig/network-scripts/route-eth0

Add following

10.0.0.0/24 via 192.168.5.20 dev eth0

For Debian/Ubuntu/Kali Linux etc.

File: /etc/network/interfaces

Add following

up ip route add 10.0.0.0/24 via 192.168.5.20 dev eth0

Restart network services to make the changes take effect

sudo /etc/init.d/networking restart
 
OR
 
sudo service restart networking
 
OR
 
systemctl restart networking

1.10 Add default gateway

sudo ip route add default via 10.0.0.1

2 Network configuration file

For CentOS/RHEL/Fedora etc.

File: /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
   #Alias name for the NIC
BOOTPROTO={static|dhcp|none|bootp}
   #Boot protocol, static|none;dhcp
IPADDR=192.168.10.10
   #Set IP address
NETMASK=255.255.255.0
   #Netmask
GATEWAY=192.168.10.1
   #Gateway
ONBOOT=yes|no
   #Activate the network port or not, on boot
HWADDR=00:1E:0B:8F:B0:D0
   #MAC address, if same as the default MAC address of the hardware, this line can be omitted 
DNS1=202.106.0.20
   #Specifiy DNS server
USERCTL=yes|no
   #Users (non-admin/root) allowed to enable/disable this port or not
PEERDNS=yes|no
   #Accept/Reject the DNS server from DHCP while BOOTPROTO is dhcp

For Debian/Ubuntu/Kali Linux etc.

File: /etc/network/interfaces

auto eth1
     #Automatically connect to Ethernet on boot
iface eth1 inet static
     #Assign IP address by static/dhcp
address 192.168.72.8
     #IP address
netmask 255.255.255.0
     #Netmask
gateway 192.168.72.1
     #Default gateway
dns-nameservers 8.8.8.8 4.4.2.2
     #DNS server

3 Hosts configuration

File: /etc/hosts

192.168.0.10 internalserver.mynet

4 Network Interface Controller (NIC) Naming

lo: Localhost loop

ppp#: Point-to-Point Protocol

eth: Ethernet

5 Network management tool

network tool and NetworkManager tool.

network

Restart network

sudo /etc/init.d/network restart

NetworkManager

It can be used to manager network easily, when X Window is not available this GUI tool can be used to manage network without the need to edit configuration file manually.

nmtui
nmtui
nmtui
NetworkManager TUI - nmtui
NetworkManager TUI – mntui

6 NetworkManager cli

NetworkManager provides cli tools as well alongside nmtui

nmcli con show
     #Get UUID table
nmcli dev
     #Check network device status
nmcli r wifi off
     #Turn off wifi
nmcli - commands
nmcli – commands

Start NetworkManager on boot

chkconfig NetworkManager on
 
OR
 
 systemctl enable NetworkManager

Start NetworkManager immediately

service NetworkManager start
 
OR
 
 systemctl start NetworkManager

How to: Find which process is causing high CPU usage Linux/Ubutu/Debian/Kali Linux/CentOS/RHEL

“top” command

top shows CPU usage in real time.

By default, it lists process by their CPU usage, refreshes every 5 seconds.

We can use following command to show top 10 processes with highest CPU usage.

top -b | head -10
[email protected]:~# top -b | head -10
top - 02:05:40 up 20:24,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 165 total,   1 running, 164 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.1 us,  3.1 sy,  0.0 ni, 93.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3913.3 total,   2066.4 free,    617.4 used,   1229.6 buff/cache
MiB Swap:   4094.0 total,   3850.2 free,    243.8 used.   3042.1 avail Mem 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  61649 root      20   0    9144   3560   3164 R   6.7   0.1   0:00.01 top
      1 root      20   0  166420   5908   3936 S   0.0   0.1   0:04.54 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.03 kthreadd
  • -b : Batch mode.
  • head -10: Display first 10 lines in the output.
  • PID : Unique ID of the process.
  • USER : Owner of the process.
  • PR : priority of the process.
  • NI : The NICE value of the process.
  • VIRT : How much virtual memory used by the process.
  • RES : How much physical memory used by the process.
  • SHR : How much shared memory used by the process.
  • S : This indicates the status of the process: S=sleep R=running Z=zombie.
  • %CPU : The percentage of CPU used by the process.
  • %MEM : The percentage of RAM used by the process.
  • TIME+ : How long the process being running.
  • COMMAND : Name of the process.

“ps” command

“ps” stands for “processes status”, it display the information about the active/running processes on the system.

We can use following command to find out high CPU usage processes.

ps -eo pid,ppid,%mem,%cpu,cmd --sort=-%cpu | head
[email protected]:~# ps -eo pid,ppid,%mem,%cpu,cmd --sort=-%cpu | head
    PID    PPID %MEM %CPU CMD
     78       2  0.0  0.4 [kswapd0]
    639     576  4.1  0.4 /usr/lib/xorg/Xorg :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
    262       2  0.0  0.2 [kworker/1:1H-kblockd]
    267       2  0.0  0.1 [kworker/0:1H-kblockd]
    889     823  0.8  0.1 xfwm4 --display :0.0 --sm-client-id 24f144caf-a490-40f0-afc0-fd75665210e1
      1       0  0.1  0.0 /sbin/init splash
      2       0  0.0  0.0 [kthreadd]
      3       2  0.0  0.0 [rcu_gp]
      4       2  0.0  0.0 [rcu_par_gp]

To see the command name instead of full path.

ps -eo pid,ppid,%mem,%cpu,comm --sort=-%cpu | head
[email protected]:~# ps -eo pid,ppid,%mem,%cpu,comm --sort=-%cpu | head
    PID    PPID %MEM %CPU COMMAND
     78       2  0.0  0.4 kswapd0
    639     576  4.1  0.4 Xorg
    262       2  0.0  0.2 kworker/1:1H-kblockd
    267       2  0.0  0.1 kworker/0:1H-kblockd
    889     823  0.8  0.1 xfwm4
      1       0  0.1  0.0 systemd
      2       0  0.0  0.0 kthreadd
      3       2  0.0  0.0 rcu_gp
      4       2  0.0  0.0 rcu_par_gp
  • -e : Select all processes.
  • -o : To customize a output format.
  • –sort=-%cpu : Sort the ouput based on CPU usage.
  • head : To display first 10 lines of the output
  • PID : Unique ID of the process.
  • PPID : Unique ID of the parent process.
  • %MEM : The percentage of RAM used by the process.
  • %CPU : The percentage of CPU used by the process.
  • Command : Name of the process.

htop

htop is a command line utility that allows you to interactively monitor your system’s vital resources or server’s processes in real time

You might need to install htop first

#Debian/Ubuntu/Kali Linux etc.
sudo apt install htop
 
#CentOS/RHEL etc.
sudo yum install htop

To launch htop (Use “q” key to exit)

htop
htop
htop

We can easily sort the processes by their Priority, Nice, Virtual memory usage, Memory, CPU and running time etc. Simply by clicking on the column header.

glances

glances is another utility which can be easily installed and used system resource monitoring tool.

Install

#Debian/Ubuntu/Kali Linux etc.
sudo apt install glances
 
#CentOS/RHEL etc.
sudo yum install glances 

Launch (Use “q” key to exit)

glances