Different ways to search in Linux/Debian/Ubuntu/Kali Linux/CentOS/RHEL etc.

locate – Locate files

It can find files quickly but it depends on updatedb to update the index, updatedb runs once everyday but we can run it manually.

# Update the index manually if necessary
sudo updatedb
 
# Find/Search for "myfile"
locate myfile

whereis, which – Locate command/executable files

whereis ProgramName
 
which ProgramName

whereis: Will search for executable, source code and documents from default installation folder (Usually, it’s the default folder when installing with root)

Default folders are:

/bin
/sbin
/usr/bin
/usr/lib
/usr/local/ma
etc.

which: Will show results from environment variables, it’s very useful for finding where the actual program/executable located

e.g. Copy source code to current directory without typing complete path

cp 'which myScript.sh'

If which ProgramName returns /usr/bin/which: no ProgramName in (/home/usr/bin:/bin), it means the ProgramName is not located in environment variables, it can’t be executed directly by it’s name

find – Find files with multiple conditions

Search by folder

# Find files with name "myFile" under root directory
find / -name myFile

If executing the above command with non-root user, it will return many errors (Permission denied), we can use following command to ignore all errors but keep normal output

find / -name 2>/dev/null

What is 2>/dev/null ?: Refer to “I/O redirection in Linux” section in Linux – Basics, Useful Terminal commands and Basic File manipulation, (QuickStart)

Search by time

Show log files which were modified within 5 minutes in current working directory

find . -name '*.log' -mmin -5

(To show log files which were modified within 1 day in current working directory)

find . -name '*.log' -mtime -1

Search by Size

e.g. Search all jpg and bmp files

find . \( -name "*.jpg" -o -name "*.bmp" ) | less
 
or
 
find . -regex ".*\(\.jpg\|\.bmp\)$"
 
# Find files over 10G
find . -type f -size +100G

Limit search folder depth

Search for any .txt files within maximum 3 folder depth

find . -maxdepth 3 -name *.txt

Search for files which are not…

Search for files which are not .txt files

find . -not -name *.txt

find, grep Search for file contents

To list name of all .txt files with Mytext in the content

find . -name *.txt -exec grep -l 'Mytext' {} \;
 
or
 
find . -name *.txt | xargs grep -l 'Mytext'
 
or
 
grep -rl 'Mytext'

To view matched previous 2 lines and next 3 lines

grep -A 3 -B 2 'Mytext' a.txt

To view match counts

# Exact match
grep -c 'Mytext' a.txt
 
# Ignore case
grcp -ci 'Mytext' a.txt

Count number of series in FASTA and FASTQ

grep '^>' test.fa
 
grep '^+$' test.fq

^: Start with …

$: End with …

Match/Show unmatched lines

grep -v 'Mytext' a.txt

Regular Expression

By default, grep uses basic regular expression, add -E switch to use extended regular expression, add -P switch to use perl format regular expression

e.g. Remove all blank lines from text file

grep -v '^$' a.txt >a-without-blank-lines.txt

Extended reading

More about “find” command can be found here: How to: Use “find” command in Linux (Debian, Ubuntu, Kali Linux, CentOS, RHEL/RedHat etc.)

Other techniques related to grep can be found here: How to: Search in Linux, How to: Use grep command, How to: Use grep to search

More on whereis, which commands: Linux, Ubuntu etc. How to find where the program is installed


Basic/Common Special symbols/Special characters in Linux

Home Directory ~

# Back to home directory
cd ~
 
# Change to sub directory within home directory
cd ~/Desktop

Current Directory .

ls -al
ls -al

. in above image means current directory

# Run script.sh in current directory
./script.sh

Parent Directory ..

cd ..
cd ..
cd ..
# Change to /test/ folder in parent folder directory
cd ../test/

Path Separator, System root folder /

ls /etc
# Change to system root folder
cd /

Comment #

#Comments are ignored by Bash shell
Comment
Comment

We can also comment part of the string from variable

# Define variable
test_string="abc def"
 
# echo the variable with abc being commented out
echo ghi ${test_string#abc}
 
# echo full variable
echo $test_string
Comment part of the string/variable
Comment part of the string/variable

Single Character Wildcard ?

? Can be used for this purpose

ls file?.txt
ls file?.txt
ls file?.txt

Since file.txt doesn’t have any characters right after file, it is not in the result.

To match exactly how many characters with ?, we use corresponding number of ?

ls ????.txt
ls ????.txt
ls ????.txt

Wildcard *

Match any character including space

ls file*
ls file*
ls file*

Match any type of files

ls file.*

Character Set Wildcard []

Match at least one of the characters in the []

ls file0[123].txt
ls file0[123].txt
ls file0[123].txt
ls file[012][012].txt
ls file[012][012].txt

Shell Command Separator (Run till the last one anyway) ;

command 1; command 2; command 3
command 1; command 2; command 3
command 1; command 2; command 3

When using Shell Command Separator “;” no matter the previous command succeeded or not, it will run till the last command

Shell Command Separator (Stop if there is error/failed) &&

command 1 && command 2 && command 3
command 1 && command 2 && command 3
command 1 && command 2 && command 3

Background Process &

Add & behind the command to run the command in the background

command &
Run the command in the background
Run the command in the background

1709 is the process ID of this background task

Input Redirection <

command < file
sort test file
sort test file
wc test
 
wx < test

(< does not display source filename)

Output Redirection >

ls > ListOfFiles.txt
 
# To read the file again
cat ListOfFiles.txt

It can be used with stderr, stdin, stdout (See bottom of this page)

cat test.txt 2> err.txt
Use with stderr
Use with stderr

Pipe |

command 1 | command 2 | command 3
cat test | grep [Aa] | sort -r
Demo or pipe
Demo or pipe

1 Use cat to read test, pip to grep command

2 Use grep to filter the test, only left with text including A and a, pipe to sort command

3 Use sort to reverse sort the text

Pipeline logical NOT and History Operator !

[ ! -d ./testfolder ] && echo "Folder testfolder does not exist"

[ ! -d ./testfolder ]: Check if the folder named testfolder exists

If does not exist, echo the text Folder testfolder does not exist

Run history command

!number
 
# e.g.
!210
Run history command
Run history command
# Run last command
!!
Run last command
Run last command

Variable Expressions $

In Bash shell, $ usually means variables

We can use echo to view the values of the variable

echo $PATH
 
echo $USER
 
echo $HOME
Show values in variables
Show values in variables

We can define variables in following ways

year=2020
MyName=Fred
Define variable

We can use {} to perform other advanced manipulation of text

# Define variable
string=12345qwert
 
# Output the string
echo ${string}
 
# Output text begin from index of 3 (Begin from 0)
echo ${string:3}
 
# Output 3 characters begin from index 0
echo ${string:0:3}
 
# Output 2 characters begin from index 2
echo ${string:2:2}

Quoting Special Characters “” ”

Use “” to stop special character functions (But not $)

echo "$string"
Double quotes
Double quotes

Use single quotes ” to disable all special character’s function

echo '$string'
Single quotes
Single quotes

We can also use backslash \ to prevent the following character to be functioning as a special character

echo "\$string"
Backslash
Backslash

Extend Reading

More on redirection, pipeline,stdin,stdout,stderr | Linux – Basics, Useful Terminal commands and Basic File manipulation, (QuickStart)

More on flow control, if else, loop, variable Linux Bash/Shell simple, basic flow control


How to: Install VMware Tools in Linux

Install from ISO

1 Launch VMware Workstation

2 Start the Virtual Machine

3 Right click on the Virtual Machine Name on the tab

4 Click on “Install VMware Tools”

Parameter for “./vmware-install.pl”

-d: Use default/suggested answers for questions

-f: Force install

# Debian/Ubuntu etc.
#Mount to /mnt
sudo mount /dev/sr0 /mnt
#Extract, install
tar -zxvf /mnt/VMwareTools-versionNumber.tar.gz -C ~
cd vmware-tools-distrib/
sudo ./vmware-install.pl -f -d

When done, eject the ISO, reboot the operating system

sudo unmout /mnt
eject
sudo reboot

# CentOS/Fedora/RHEL etc.
#Mount to /mnt
mount /dev/sr0 /mnt
#Extract, install
tar -zxvf /mnt/VMwareTools-versoinNumber.tar.gz -C ~
cd vmware-tools-distrib/
./vmware-install.pl -f -d

Install VMware Tools from Online source/Package Manager

# Debian/Ubuntu etc.
#System without desktop environment
sudo apt install open-vm-tools
 
#System with desktop environment
sudo apt install open-vm-tools-desktop
# CentOS/Fedora/RHEL etc.
#System without desktop environment
dnf -y install open-vm-tools
or
yum -y install open-vm-tools
 
#System with desktop environment
dnf -y install open-vm-tools-desktop
or
yum -y install open-vm-tools-desktop

How to: Check Linux disk/partition information (Alternative to df)

To check Linux partition/disk usage, mount, file system types information etc. we can use df command which mentioned in “Useful commands to look up system information, install package etc. (Linux, Debian, Ubuntu, Kali Linux, RedHat, CentOS etc.)

We can also use another alternative to df which is di

1 Install

# Debian/Ubuntu/Kali Linux etc.
sudo apt install di
 
# CentOS/Fedora/RHEL
sudo dnf install di

2 Usage

2.1 Default output

[email protected]:~$ di
Filesystem         Mount               Size     Used    Avail %Used  fs Type
/dev/sda2          /                 196.3G    15.2G   171.1G   13%  ext4   
/dev/sda1          /boot/efi         511.0M     0.1M   510.8M    0%  vfat   
tmpfs              /dev/shm            1.9G     0.0G     1.9G    0%  tmpfs  
tmpfs              /run              391.3M    15.8M   375.4M    4%  tmpfs  
tmpfs              /run/lock           5.0M     0.0M     5.0M    0%  tmpfs  
tmpfs              /run/user/1000    391.3M     0.0M   391.3M    0%  tmpfs  
tmpfs              /sys/fs/cgroup      1.9G     0.0G     1.9G    0%  tmpfs
di
di

(By default di uses human readable units, with df we need to use -h switch “df -h”)

2.2 Show detailed mount point information, special device name etc.

di -A
[email protected]:~$ di -A
Mount          fs Type Filesystem 
        Options                                                                                               
            Size     Used     Free %Used  %Free 
            Size     Used    Avail %Used  %Free 
            Size     Used    Avail %Used  
           Inodes     IUsed     IFree %IUsed
/              ext4    /dev/sda2  
        rw,relatime,errors=remount-ro                                                                         
          196.3G    15.2G   181.2G    8%    92%  
          196.3G    25.2G   171.1G   13%    87%  
          186.3G    15.2G   171.1G    8%  
         13165632    570127  12595505    4% 
/boot/efi      vfat    /dev/sda1  
        rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro 
          511.0M     0.1M   510.8M    0%   100%  
          511.0M     0.1M   510.8M    0%   100%  
          511.0M     0.1M   510.8M    0%  
                0         0         0    0% 
/dev/shm       tmpfs   tmpfs      
        rw,nosuid,nodev                                                                                       
            1.9G     0.0G     1.9G    0%   100%  
            1.9G     0.0G     1.9G    0%   100%  
            1.9G     0.0G     1.9G    0%  
           500837         1    500836    0% 
/run           tmpfs   tmpfs      
        rw,nosuid,nodev,noexec,relatime,size=400672k,mode=755                                                 
          391.3M    15.8M   375.4M    4%    96%  
          391.3M    15.8M   375.4M    4%    96%  
          391.3M    15.8M   375.4M    4%  
           500837       668    500169    0% 
/run/lock      tmpfs   tmpfs      
        rw,nosuid,nodev,noexec,relatime,size=5120k                                                            
            5.0M     0.0M     5.0M    0%   100%  
            5.0M     0.0M     5.0M    0%   100%  
            5.0M     0.0M     5.0M    0%  
           500837         4    500833    0% 
/run/user/1000 tmpfs   tmpfs      
        rw,nosuid,nodev,relatime,size=400668k,mode=700,uid=1000,gid=1000                                      
          391.3M     0.0M   391.3M    0%   100%  
          391.3M     0.0M   391.3M    0%   100%  
          391.3M     0.0M   391.3M    0%  
           500837        28    500809    0% 
/sys/fs/cgroup tmpfs   tmpfs      
        ro,nosuid,nodev,noexec,mode=755                                                                       
            1.9G     0.0G     1.9G    0%   100%  
            1.9G     0.0G     1.9G    0%   100%  
            1.9G     0.0G     1.9G    0%  
           500837        17    500820    0% 
di -A
di -A

2.3 Show all mounted devices

di -a
[email protected]:~$ di -a
Filesystem         Mount               Size     Used    Avail %Used  fs Type            
/dev/sda2          /                 196.3G    15.2G   171.1G   13%  ext4               
/dev/sda1          /boot/efi         511.0M     0.1M   510.8M    0%  vfat               
udev               /dev                1.9G     0.0G     1.9G    0%  devtmpfs           
hugetlbfs          /dev/hugepages      0.0M     0.0M     0.0M    0%  hugetlbfs          
mqueue             /dev/mqueue         0.0M     0.0M     0.0M    0%  mqueue             
devpts             /dev/pts            0.0M     0.0M     0.0M    0%  devpts             
tmpfs              /dev/shm            1.9G     0.0G     1.9G    0%  tmpfs              
proc               /proc               0.0M     0.0M     0.0M    0%  proc               
systemd-1          /proc/sys/fs/bi     0.0M     0.0M     0.0M    0%  autofs             
binfmt_misc        /proc/sys/fs/bi     0.0M     0.0M     0.0M    0%  binfmt_misc        
tmpfs              /run              391.3M    15.8M   375.4M    4%  tmpfs              
tmpfs              /run/lock           5.0M     0.0M     5.0M    0%  tmpfs              
sunrpc             /run/rpc_pipefs     0.0M     0.0M     0.0M    0%  rpc_pipefs         
tmpfs              /run/user/1000    391.3M     0.0M   391.3M    0%  tmpfs              
/dev/fuse          /run/user/1000/     0.0M     0.0M     0.0M    0%  fuse               
gvfsd-fuse         /run/user/1000/     0.0M     0.0M     0.0M    0%  fuse.gvfsd-fuse    
vmware-vmblock     /run/vmblock-fu     0.0M     0.0M     0.0M    0%  fuse.vmware-vmblock
sysfs              /sys                0.0M     0.0M     0.0M    0%  sysfs              
efivarfs           /sys/firmware/e     0.0M     0.0M     0.0M    0%  efivarfs           
none               /sys/fs/bpf         0.0M     0.0M     0.0M    0%  bpf                
tmpfs              /sys/fs/cgroup      1.9G     0.0G     1.9G    0%  tmpfs              
cgroup             /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup             
cgroup             /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup             
cgroup             /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup             
cgroup             /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup             
cgroup             /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup             
cgroup             /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup             
cgroup             /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup             
cgroup             /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup             
cgroup             /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup             
cgroup             /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup             
cgroup             /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup             
cgroup2            /sys/fs/cgroup/     0.0M     0.0M     0.0M    0%  cgroup2            
fusectl            /sys/fs/fuse/co     0.0M     0.0M     0.0M    0%  fusectl            
pstore             /sys/fs/pstore      0.0M     0.0M     0.0M    0%  pstore             
debugfs            /sys/kernel/deb     0.0M     0.0M     0.0M    0%  debugfs            
securityfs         /sys/kernel/sec     0.0M     0.0M     0.0M    0%  securityfs
di -a
di -a

2.4 Produce CSV style output

di -c
[email protected]:~$ di -c
s,m,b,u,v,p,T
"/dev/sda2","/","196.3G","15.2G","171.1G",13%,"ext4"
"/dev/sda1","/boot/efi","511.0M","0.1M","510.8M",0%,"vfat"
"tmpfs","/dev/shm","1.9G","0.0G","1.9G",0%,"tmpfs"
"tmpfs","/run","391.3M","15.8M","375.4M",4%,"tmpfs"
"tmpfs","/run/lock","5.0M","0.0M","5.0M",0%,"tmpfs"
"tmpfs","/run/user/1000","391.3M","0.0M","391.3M",0%,"tmpfs"
"tmpfs","/sys/fs/cgroup","1.9G","0.0G","1.9G",0%,"tmpfs"
di -c
di -c

2.5 Output size in Gigabyte

di -g

We can also use -k or -m to indicate output in K or M

[email protected]:~$ di -g
Filesystem         Mount              Gibis     Used    Avail %Used  fs Type
/dev/sda2          /                  196.3     15.2    171.1   13%  ext4   
/dev/sda1          /boot/efi            0.5      0.0      0.5    0%  vfat   
tmpfs              /dev/shm             1.9      0.0      1.9    0%  tmpfs  
tmpfs              /run                 0.4      0.0      0.4    4%  tmpfs  
tmpfs              /run/lock            0.0      0.0      0.0    0%  tmpfs  
tmpfs              /run/user/1000       0.4      0.0      0.4    0%  tmpfs  
tmpfs              /sys/fs/cgroup       1.9      0.0      1.9    0%  tmpfs
di -g
di -g

2.6 Display information for specific file system

di -I filesystem
 
e.g.
di -I tmpfs
[email protected]:~$ di -I tmpfs 
Filesystem         Mount               Size     Used    Avail %Used  fs Type
tmpfs              /dev/shm            1.9G     0.0G     1.9G    0%  tmpfs  
tmpfs              /run              391.3M    15.8M   375.4M    4%  tmpfs  
tmpfs              /run/lock           5.0M     0.0M     5.0M    0%  tmpfs  
tmpfs              /run/user/1000    391.3M     0.0M   391.3M    0%  tmpfs  
tmpfs              /sys/fs/cgroup      1.9G     0.0G     1.9G    0%  tmpfs
di -I tmpfs
di -I tmpfs

2.7 Skip title row

di -n
[email protected]:~$ di -n
/dev/sda2          /                 196.3G    15.2G   171.1G   13%  ext4   
/dev/sda1          /boot/efi         511.0M     0.1M   510.8M    0%  vfat   
tmpfs              /dev/shm            1.9G     0.0G     1.9G    0%  tmpfs  
tmpfs              /run              391.3M    15.8M   375.4M    4%  tmpfs  
tmpfs              /run/lock           5.0M     0.0M     5.0M    0%  tmpfs  
tmpfs              /run/user/1000    391.3M     0.0M   391.3M    0%  tmpfs  
tmpfs              /sys/fs/cgroup      1.9G     0.0G     1.9G    0%  tmpfs
di -n
di -n

2.8 Show total count at the bottom of the output

di -t
[email protected]:~$ di -t
Filesystem         Mount               Size     Used    Avail %Used  fs Type
/dev/sda2          /                 196.3G    15.2G   171.1G   13%  ext4   
/dev/sda1          /boot/efi         511.0M     0.1M   510.8M    0%  vfat   
tmpfs              /dev/shm            1.9G     0.0G     1.9G    0%  tmpfs  
tmpfs              /run              391.3M    15.8M   375.4M    4%  tmpfs  
tmpfs              /run/lock           5.0M     0.0M     5.0M    0%  tmpfs  
tmpfs              /run/user/1000    391.3M     0.0M   391.3M    0%  tmpfs  
tmpfs              /sys/fs/cgroup      1.9G     0.0G     1.9G    0%  tmpfs  
                   Total             201.4G    15.2G   176.2G   13%
di -t
di -t

2.9 Sort the output

# Sort by name of the mount point
di -sm
[email protected]:~$ di -sm
Filesystem         Mount               Size     Used    Avail %Used  fs Type
/dev/sda2          /                 196.3G    15.2G   171.1G   13%  ext4   
/dev/sda1          /boot/efi         511.0M     0.1M   510.8M    0%  vfat   
tmpfs              /dev/shm            1.9G     0.0G     1.9G    0%  tmpfs  
tmpfs              /run              391.3M    15.8M   375.4M    4%  tmpfs  
tmpfs              /run/lock           5.0M     0.0M     5.0M    0%  tmpfs  
tmpfs              /run/user/1000    391.3M     0.0M   391.3M    0%  tmpfs  
tmpfs              /sys/fs/cgroup      1.9G     0.0G     1.9G    0%  tmpfs
di -sm
di -sm

-s: sort

m: Order by mountpoint (Default)

n: Do not sort (Use the order from “/etc/fstab”)

s: Order by name of special device

t: Order by type of fil esystem

r: Reverse the order

Sort switch can be mixed, e.g. di -stsrm: Type, Device, Mount point in reverse order. di strsrm: Type, Device in reverse order, mount point.

2.10 Specify output format

# Output mount point name
di -fm
 
# Output file system type
di -ft
[email protected]:~$ di -fm
Mount          
/              
/boot/efi      
/dev/shm       
/run           
/run/lock      
/run/user/1000 
/sys/fs/cgroup
di -fm
di -fm
di -ft
di -ft

man page of di

di(1)                                                                                                  General Commands Manual                                                                                                  di(1)
Name
       di - disk information
Synopsis
       di [-AacghHklLmnPqRtZ] [-B block-size] [-d display-size] [-f format] [-I include-fstyp-list] [-s sort-type] [-w block-width] [-W inode-width] [-x exclude-fstyp-list] [-X debug-level] [-z zone-name] [file [...]]
       mi
Description
       di  Displays  usage information on mounted filesystems.  Block values are reported in a human readable format.  If the user or group has a disk quota, the values reported are adjusted according the quotas that apply to the
       user.
       If file is specified, the usage information for the partition on which file is located is printed.
       Unless the -a flag is specified, the following mounted filesystems will not normally be displayed: filesystems with total space <= 0; loopback filesystems that are duplicates of other normally mounted filesystems (filesys‐
       tem type of 'lofs' , 'none', or 'nullfs'); loopback filesystems that are part of a zone (Solaris).
       Filesystems that the user does not have permissions to access will not be displayed at all.
       mi Displays the mounted filesystem information.
       Several options may be specified to control the output of di and mi:
       -A     Print all fields (used for debugging).  Mount points and special device names are printed at full width.
       -a     (compatibility: --all)
              Prints all mounted devices (normally, those with 0 total space are not printed - e.g. /dev/proc, /dev/fd).
       -B     block-size (compatibility: --block-size, -b)
              Change the base block size from 1024 (default) to the size specified.  block-size may be one of: k - 1024 bytes, si - 1000 bytes, or a specific size.
       -c     (alias: --csv-output)
              Comma separated values are output.  The titles are output as the format string specifiers.  Totals are turned off. See also the -n flag.
       -C     (alias: --csv-tabs)
              Values are output with tab separators.  See also the -c option.
       -d     display-size (alias: --display-size)
              Display  the  usage  in  units  specified  by  display-size.   Note  that if the base block size is SI units of 1000 bytes, the display size is calculated using those units.  display-size may be one of: 512 - POSIX,
              k - kilobytes, m - megabytes, g - gigabytes, t - terabytes, p - petabytes, e - exabytes, z - zettabytes, y - yottabytes, h - Scaled alternative 1, H - Scaled alternative 2, or a specific value to use  as  the  block
              size.
              Block display sizes greater than 1024 bytes are displayed with a precision of one decimal place after the radix.
              The Scaled alternatives scale the sizes displayed and appends a suffix (e.g. 48.0k, 3.4M).
              With scaled alternative 1, sizes within a line may scale to different units.
              Scaled alternative 2 scales all the sizes in each individual line to the same unit size (the largest needed).
       -f     format Use the specified format string format.  See the Format Strings section.
       -g     (alias for: -dg)
              Display sizes in gigabytes.
       -h     (alias for: -dh)
              Display partition sizes in scaled alternative 1 format.
       --help
              Display some basic usage information.
       -H     (alias for: -dH; compatibility: --human-readable)
              Display partition sizes in scaled alternative 2 format.
       -I     include-fstype-list (compatibility: -F, --type)
              Include  only  the  file  system  types listed in include-fstyp-list.  The list is a comma separated list of file system types.  Multiple -I options may be specified.  If the 'fuse' filesystem type is specified, all
              fuse* filesystems will be included.
              e.g. -I nfs,tmpfs or -I nfs -I tmpfs.
       --inodes
              Ignored.  Use the -f option.
       -k     (alias for: -dk)
              Display sizes in Kbytes.
       -l     (compatibility: --local)
              Display only local file systems.
       -L     Turn off check for duplicate filesystems (loopback (lofs/none) mounts).
       -m     (alias for: -dm)
              Display sizes in megabytes.
       -n     Do not print a header line above the list of file systems.  Useful when parsing the output of di.
       --no-sync
              Ignored.
       -P     (compatibility: --portability)
              Output format is POSIX standard.  512 byte block size is the default.  The -k option may be used.
       --print-type
              Ignored.  Use the -f option.
       -q     Disable quota checks.
       -R     (also: --dont-resolve-symlinks)
              Do not resolve symlinks (for mount points that have a trailing UUID).
       -s     sort-type
              Use sort-type to sort the output.  The output of di is normally sorted by mount point.  The following sort flags may be used to change the sort order: m - by mount point (default); n - leave unsorted (as it  appears
              in  the  mount  table); s - by special device name; T - by total space; f - by free space; a - by available space; t - by filesystem type; r - reverse the sort order; This will apply to all sort flags following this
              sort flag.
              These sort options may be combined in any order.  e.g.:
              di -stsrm # by type, special, reversed mount;
              di -strsrm # by type, reversed special, mount.
       --si   An alias for -dH -Bsi.
       --sync Ignored.
       -t     (compatibility: --total)
              Print a totals line below the list of file systems.  Pooled filesystems (zfs, advfs) have only the main pool added to the total.  It is up to the user to exclude (using the -x option)  read-only  filesystems  (cdfs,
              iso9660), swap-based (memfs, mfs, tmpfs) filesystems and user (fuse*) filesystems.  Excluding the 'fuse' filesystem will exclude all fuse* filesystems.
       -w     block-width
              Set the print width for block values.  The default is eight.
       -v     Ignored.
       --version
              Display di's version and default format string.
       -W     inode-width
              Set the print width for inode values.  Default is seven.
       -x     exclude-fstype-list (compatibility: --exclude-type)
              Exclude  the  file  system  types  listed in exclude-fstyp-list.  The list is a comma separated list of file system types.  Multiple -x options may be specified.  If the 'fuse' filesystem type is excluded, all fuse*
              filesystems will be excluded.  e.g. -x nfs,tmpfs or -x nfs -x tmpfs.
       -X     level
              Set the program's debugging level to debug-level.
       -z     zone-name
              Display the filesystems for the specified zone.  The zone must be visible to the user.
       -Z     (alias for: -z all)
              Display the filesystems for all visible zones.
Format Strings
       The output of di may be specified via a format string.  This string may be given either via the -f command line option or as part of the DI_ARGS environment variable.  The format string may specify the following columns:
       m      Print the name of the mount point.
       M      Print the name of the mount point, at full length.  The mount point is formatted to the maximum width necessary for the longest mount point name.
       s      Print the file system name (special device or remote mount point).
       S      Print the file system name (special device or remote mount point), at full length.  The file system name is formatted to the maximum width necessary for the longest file system name.
       t      Print the file system type.
       T      Print the file system type at full length.  The file system type is formatted to the maximum width necessary for the longest file system type.
       Total Available
       b      Print the total number of megabytes on the file system.
       B      Print the total number of megabytes on the file system available for use by normal users.
       In Use
       u      Print the number of megabytes in use on the file system (actual number of megabytes used = total - free).
       c      Print the number of megabytes not available for use by normal users (total - available).
       Free
       f      Print the number of free (unused) megabytes on the file system.
       v      Print the number of megabytes available for use by normal users.
       Percentage Used
       p      Print the percentage of megabytes not available for use by normal users (number of megabytes not available for use / total disk space).
       1      Print the percentage of total megabytes in use (actual number of megabytes used / total disk space).
       2      Print the percentage of megabytes in use, BSD-style.  Represents the percentage of user-available space in use.  Note that values over 100% are possible (actual number of megabytes used /  disk  space  available  to
              non-root users).
       Percentage Free
       a      Print the percentage of megabytes available for use by normal users (number of megabytes available for use / total disk space).
       3      Print the percentage of total megabytes free (actual number of megabytes free / total disk space).
       Inodes
       i      Print the total number of file slots (inodes) that can be created on the file system.
       U      Print the number of file slots in use.
       F      Print the number of file slots available.
       P      Print the percentage of file slots in use.
       Mount Information
       I      Print the time the filesystem was mounted.  This column is not supported on all systems.
       O      Print the filesystem mount options.
       The default format string for di is smbuvpT.
       The default format string for mi is MSTIO.
       The format string may also contain any other character not listed above.  The character will be printed as is.  e.g. di -f 'mbuvp|iUFP' will print the character '|' between the disk usage and the file slot usage.  The com‐
       mand sequence:
              di -f 'mbuvp
              miUFP'
       will print two lines of data for each filesystem.
Examples
       Various df equivalent format strings for System V release 4 are:
              /usr/bin/df -v     di -P -f msbuf1
              /usr/bin/df -k     di -dk -f sbcvpm
              /usr/ucb/df        di -dk -f sbuv2m
       GNU df:
              df                 di -dk -f SbuvpM -w 10
              df -T              di -dk -f STbuvpM -w 10
       AIX df:
              df                 di -d 512 -f Sbf1UPM -w 10
              df -I              di -d 512 -f Sbuf1M
              df -I -M           di -d 512 -f SMbuf1 -w 10
       HP-UX bdf:
              bdf                di -d k -f Sbuv2M
              bdf -i             di -d k -f Sbuv2UFPM
       If you like your numbers to add up/calculate the percentage correctly, try one of the following format strings:
              di -f SMbuf1T
              di -f SMbcvpT
              di -f SMBuv2T
Environment Variables
       The DI_ARGS environment variable may be used to specify command line arguments.  e.g. If you always want gigabytes displayed, set DI_ARGS equal to "-dg".  Any command line arguments specified will override the DI_ARGS  en‐
       vironment variable.
       The DI_LOCALE_DIR environment variable may be used to specify the location of the di program's locale message files.
       The GNU df POSIXLY_CORRECT, and DF_BLOCK_SIZE and the BSD BLOCKSIZE environment variables are honored.
Note
       For filesystems that do not report available space (e.g. System V release 3), the number of available space is set to the free space.
WARNING
       Do not replace your system's df command with this program.  You will in all likelihood break your installation procedures.
See Also
       df(1), fstab(5), getmnt(2), getmntinfo(2), mnttab(4), mount(1M) statfs(2), statvfs(2)
Bugs
       Send bug reports to: brad.lanam.di_at_gmail.com
       Known Issues:
       di will probably not process a zettabyte or yottabyte sized filesystem properly due to overflow of a long long.
Website
       https://gentoo.com/di/
Author
       This program is Copyright 1994-2011 by Brad Lanam.
       Brad Lanam, Walnut Creek, CA (brad.lanam.di_at_gmail.com)
                                                                                                             17 Jan 2013                                                                                                        di(1)

How to: Check/Change/Set/Modify Linux (Debian/Ubuntu/Kali Linux) date/time/timezone etc. (and Calculate past/future date/time) in Terminal

Check/Calculate time

Show current time

date
date
date

Calculate past/future time

date --date="1 day ago"
 
date --date="2 hours ago"
 
date --date="3 days"
 
date --date="2 hours"
Calculate dates/time
Calculate dates/time

timedatectl
 
timedatectl status
timedatectl
timedatectl
timedatectl status
timedatectl status

Show/Change timezone

Show timezone

timedatectl list-timezones

Set timezone

timedatectl set-timezone Africa/Abidjan

Change time by using “timedatectl”

(HH:MM:SS, in 24 hour format)

sudo timedatectl set-time 00:00:00
Use timedatectl set-time to change time
Use timedatectl set-time to change time

Change time by using “date”

(HH:MM:SS, in 24 hour format)

sudo date +%T -s "23:00:00"

(12 hour format)

sudo date +%T%p -s "2:00:00AM"
 
sudo date +%T%p -s "2:00:00PM"

Change date

Set the date and time to 03 Feb 2001, 01:00:00

sudo date --set="20010203 01:00:00"
 
sudo timedatectl set-time '2001-02-03 01:00:00'

Set date

sudo timedatectl set-time 2001-02-03
 
sudo date --date="2001-02-03"

Create custom date format (+)

sudo date +"Year : %Y Day : %d Month : %m"
 
# Output
Year : 2001 Day : 03 Month : 02
 
sudo +%D
 
# Output
02/03/01
 
sudo date +"%a %b %d %y"
 
# Output
Sat Feb 03 01
 
sudo date +"%A %B %d %Y"
 
# Output
Saturday February 03 2001
 
sudo date +"%A %B %d %Y %T"
 
# Output
Saturday February 03 2001 00:27:30
 
sudo date +"%A %B-%d-%Y %c"
 
# Output
Saturday February-03-2001 Sat 03 Feb 2001 00:28:17 EST

%D: Year/Month/Day format.

Show CMOS Time

sudo hwclock
hwclock
hwclock

How to: Remove junk file from Linux easily

tmpwatch/tmpreaper is the tool we can use for this purpose. It can be used to remove files and folders which haven’t being used for a specific time. (find command can be used to achieve the same result as well)

By default, tmpwatch/tmpreaper can use access time, inode change time, modification time to remove files.

Usually, tmpwatch/tmpreaper is used to remove files from “/tmp” folder and other folders like old log files.

Warning: Do not run this command within root folder “/”!

1 Installation

# CentOS/RHEL/Fedora
sudo dnf install tmpwatch
 
# Older version of CentOS
sudo yum install tmpwatch
 
# Debian/Ubuntu Kali Linux etc.
sudo apt install tmpreaper
 
# OpenSUSE
sudo zypper install tmpwatch

2 Using tmpwatch/tmpreaper

tmpwatch and tmpreaper have identical commands/switches.

2.1 Remove files which haven’t being accessed over X days

tmpreaper 30d /var/log/

30d = 30 days

d: Days

h: hours

m: minutes

s: seconds

By default, without specifying the unit it is considered as hours (In following example, 24 hours)

# Remove files haven't being accessed over 10 hours
tmpreaper 24 /var/log/

2.2 Remove files which haven’t being modified over X days

tmpreaper -m 30d /var/log/

2.3 Remove symbolic links

tmpreaper -s 1d /var/log/

2.4 Remove all files (Include files, folders and symbolic files)

tmpreaper -a 1d /var/log/

2.5 Skip folder/directory when deleting

tmpreaper -am --nodirs /var/log/

2.6 Test deleting

tmpreaper -t 5d /var/log/

2.7 Force deleting

tmpreaper -f 3d /var/log/

2.8 Skip files when deleting

tmpreaper --protect '*.txt' 3d /var/log/

tmpreaper can be used with cron job, e.g. delete tmp or log file from 90 days ago every night at 2:00.


Introduction to /etc/passwd and /etc/shadow files in Linux systems (Debian/Ubuntu/CentOS/RHEL etc.)

Linux operating systems store all username and password (including administrators/root) in /etc/passwd and /etc/shadow file.

/etc/passwd

Each user has a line of corresponding record which records basic attributes. Only root/administrators can modify it. All other users have read only access to it.

/etc/shadow

As name suggested, this file is like shadow of “passwd” file. The record in “shadow” file is corresponding to the records in “passwd” file. Records is “shadow” file is automatically produced by “pwconv” command based on “passwd” file. Only root/administrators have read and write access to “shadow” file, other users can’t read it.

File permission for passwd and shadow
File permission for passwd and shadow

About /etc/passwd

sudo vi /etc/passwd
partial passwd file
partial passwd file
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin

There are 7 columns for each record

ColumnDescription
1Username
2Placeholder, x = password is required to login, empty = password is not required to login
3User UID
4User GID
5Extra information, Full name, contact information etc.
6Home directory
7Login shell, /bin/bash = Login to system shell enabled, /sbin/nologin = User can’t login

About /etc/shadow

sudo vi /etc/shadow
partial shadow file
partial shadow file
root:!:18313:0:99999:7:::
daemon:*:18313:0:99999:7:::
bin:*:18313:0:99999:7:::
sys:*:18313:0:99999:7:::
sync:*:18313:0:99999:7:::
games:*:18313:0:99999:7:::
man:*:18313:0:99999:7:::
lp:*:18313:0:99999:7:::
mail:*:18313:0:99999:7:::
news:*:18313:0:99999:7:::

There are 8 columns for each record

ColumnDecription
1Username
2Password (!! = no password, encrypted if password is set)
3Days between last change of password and 01/01/1970
4Minimum password age (Validated days)
5Maximum password age (Validated days)
6Buffer time (Days) after the password is expired (After the password is expired, for how many days the user can change the password, old password can’t be used to login again during this period of time)
7Number of days after password expires that account is disabled
8Date which the account is disabled (Days since 01/01/1979)
9Not used yet

How to: Run Linux commands with time limit/timeout (Kill process/command after some time)

Sometimes we want to stop or kill the command after a period of time, so that we don’t get stuck with that command and wasting resources etc. To specify timeout or time limit for Linux command, we can use timeout command

Command Usage/Parameters

timeout [OPTION] DURATION COMMAND [ARG]...

DURATION is integer or floating point with unit

s: Seconds (Default)

m: Minutes

h: Hours

d: Days

Without units appended, by default it is considered as seconds.

If the DURATION is 0, the timeout is disabled.

Basic Usage

Timeout ping command after 3 seconds

timeout 3 ping 127.0.0.1
timeout 3 ping 127.0.0.1
timeout 3 ping 127.0.0.1

Timeout ping command after 3 minutes

timeout 3m ping 127.0.0.1

Timeout ping command after 3 days

timeout 1d ping 127.0.0.1

Timeout ping command after 3.2 seconds

timeout 3.2s ping 127.0.0.1

Send specific signal after timeout

By default if signal is not specified, timeout command will use “SIGTERM” signal after timeout. We can use -s (-signal) switch to specific which signal to send after timeout

e.g. Send SIGKILL signal to ping command after 3 seconds

sudo timeout -s SIGKILL 3s ping 127.0.0.1
sudo timeout -s SIGKILL 3s ping 127.0.0.1
sudo timeout -s SIGKILL 3s ping 127.0.0.1

We can use the name of the signal or the number of the signal

e.g. We can use 9 as SIGKILL to achieve same result

sudo timeout -s 9 3s ping 127.0.0.1
sudo timeout -s 9 3s ping 127.0.0.1
sudo timeout -s 9 3s ping 127.0.0.1

To list all acceptable signal, we can use kill -l to find out

kill -l
[email protected]:~# kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX
kill -l
kill -l

Stop frozen process

SIGTERM, the default signal can be ignored by some processes, thus the program will keep running. To make sure the process is killed, we can use -k (–kill after) switch with specified time. When the time limited reached, force to kill the process.

e.g. Let the shell script run for 2 minutes, if it did not exit, then kill after 5 seconds

timeout -k 5s 2m sh test.sh

By default the timeout command will run in background, if we want to run it in foreground, refer to following example

timeout --foreground 2m ./test.sh

timeout help

Usage: timeout [OPTION] DURATION COMMAND [ARG]...
  or:  timeout [OPTION]
Start COMMAND, and kill it if still running after DURATION.
Mandatory arguments to long options are mandatory for short options too.
      --preserve-status
                 exit with the same status as COMMAND, even when the
                   command times out
      --foreground
                 when not running timeout directly from a shell prompt,
                   allow COMMAND to read from the TTY and get TTY signals;
                   in this mode, children of COMMAND will not be timed out
  -k, --kill-after=DURATION
                 also send a KILL signal if COMMAND is still running
                   this long after the initial signal was sent
  -s, --signal=SIGNAL
                 specify the signal to be sent on timeout;
                   SIGNAL may be a name like 'HUP' or a number;
                   see 'kill -l' for a list of signals
  -v, --verbose  diagnose to stderr any signal sent upon timeout
      --help     display this help and exit
      --version  output version information and exit
DURATION is a floating point number with an optional suffix:
's' for seconds (the default), 'm' for minutes, 'h' for hours or 'd' for days.
A duration of 0 disables the associated timeout.
If the command times out, and --preserve-status is not set, then exit with
status 124.  Otherwise, exit with the status of COMMAND.  If no signal
is specified, send the TERM signal upon timeout.  The TERM signal kills
any process that does not block or catch that signal.  It may be necessary
to use the KILL (9) signal, since this signal cannot be caught, in which
case the exit status is 128+9 rather than 124.
GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
Full documentation at: <https://www.gnu.org/software/coreutils/timeout>
or available locally via: info '(coreutils) timeout invocation'