How to: Monitor CPU/GPU/HDD temperature in Linux (Debian/Ubuntu/Kali Linux/CentOS/RHEL etc.) easily

1 glances

Monitors CPU usage, RAM usage, SWAP usage, system load, process list, hard drive I/O, Network I/O, sensors (temperature), battery, file system usage, Docker, Monitor, alarm, system information, up time etc.

Install on Debian/Ubuntu/Kali Linux

sudo apt install glances -y

Usage

sudo glances
glances
glances

2 sensor

By default, it’s installed on some Linux distros like Ubuntu etc. It can show CPU, HDD, temperature, fan RPM etc.

Install on Debian/Ubuntu/Kali Linux

sudo apt install lm-sensors

Usage

# Initialize/Detect sensors
sudo sensors-detect
 
# Show sensors information
sudo sensors

Output

coretemp-isa-0000 Adapter: ISA adapter Core 0:       +47.0°C  (high = +105.0°C, crit = +105.0°C) Core 1:       +46.0°C  (high = +105.0°C, crit = +105.0°C) acpitz-virtual-0 Adapter: Virtual device temp1:        +51.0°C  (crit = +105.0°C) thinkpad-isa-0000 Adapter: ISA adapter fan1:         674 RPM temp1:        +50.0°C temp2:        +51.0°C temp3:        +52.0°C temp4:         -1.0°C temp5:         +0.0°C temp6:         +0.0°C temp7:        +32.0°C temp8:         +0.0°C

Different ways to search in Linux/Debian/Ubuntu/Kali Linux/CentOS/RHEL etc.

locate – Locate files

It can find files quickly but it depends on updatedb to update the index, updatedb runs once everyday but we can run it manually.

# Update the index manually if necessary
sudo updatedb
 
# Find/Search for "myfile"
locate myfile

whereis, which – Locate command/executable files

whereis ProgramName
 
which ProgramName

whereis: Will search for executable, source code and documents from default installation folder (Usually, it’s the default folder when installing with root)

Default folders are:

/bin
/sbin
/usr/bin
/usr/lib
/usr/local/ma
etc.

which: Will show results from environment variables, it’s very useful for finding where the actual program/executable located

e.g. Copy source code to current directory without typing complete path

cp 'which myScript.sh'

If which ProgramName returns /usr/bin/which: no ProgramName in (/home/usr/bin:/bin), it means the ProgramName is not located in environment variables, it can’t be executed directly by it’s name

find – Find files with multiple conditions

Search by folder

# Find files with name "myFile" under root directory
find / -name myFile

If executing the above command with non-root user, it will return many errors (Permission denied), we can use following command to ignore all errors but keep normal output

find / -name 2>/dev/null

What is 2>/dev/null ?: Refer to “I/O redirection in Linux” section in Linux – Basics, Useful Terminal commands and Basic File manipulation, (QuickStart)

Search by time

Show log files which were modified within 5 minutes in current working directory

find . -name '*.log' -mmin -5

(To show log files which were modified within 1 day in current working directory)

find . -name '*.log' -mtime -1

Search by Size

e.g. Search all jpg and bmp files

find . \( -name "*.jpg" -o -name "*.bmp" ) | less
 
or
 
find . -regex ".*\(\.jpg\|\.bmp\)$"
 
# Find files over 10G
find . -type f -size +100G

Limit search folder depth

Search for any .txt files within maximum 3 folder depth

find . -maxdepth 3 -name *.txt

Search for files which are not…

Search for files which are not .txt files

find . -not -name *.txt

find, grep Search for file contents

To list name of all .txt files with Mytext in the content

find . -name *.txt -exec grep -l 'Mytext' {} \;
 
or
 
find . -name *.txt | xargs grep -l 'Mytext'
 
or
 
grep -rl 'Mytext'

To view matched previous 2 lines and next 3 lines

grep -A 3 -B 2 'Mytext' a.txt

To view match counts

# Exact match
grep -c 'Mytext' a.txt
 
# Ignore case
grcp -ci 'Mytext' a.txt

Count number of series in FASTA and FASTQ

grep '^>' test.fa
 
grep '^+$' test.fq

^: Start with …

$: End with …

Match/Show unmatched lines

grep -v 'Mytext' a.txt

Regular Expression

By default, grep uses basic regular expression, add -E switch to use extended regular expression, add -P switch to use perl format regular expression

e.g. Remove all blank lines from text file

grep -v '^$' a.txt >a-without-blank-lines.txt

Extended reading

More about “find” command can be found here: How to: Use “find” command in Linux (Debian, Ubuntu, Kali Linux, CentOS, RHEL/RedHat etc.)

Other techniques related to grep can be found here: How to: Search in Linux, How to: Use grep command, How to: Use grep to search

More on whereis, which commands: Linux, Ubuntu etc. How to find where the program is installed


How to: Search in Linux, How to: Use grep command, How to: Use grep to search

“grep” is very useful when searching text.

On Microsoft Windows, we can use “dngrep” to achieve similar results

grep command

grep is a command-line utility for searching plain-text data sets for lines that match a regular expression. Its name comes from the ed command g/re/p (globally search a regular expression and print), which has the same effect: doing a global search with the regular expression and printing all matching lines. [1]

Some basic grep usage

grep 'test' filename #Search test in file filename
grep 'test' file1 file2 #Search test from file1 and file2
cat filename | grep 'test' #Print out content from filename then search for test
grep --color=never 'test' filename #Search test from filename and don't highlight the results
grep --color=auto 'test' filename #Search test from filename and displays color in the output unless the output is piped to a command, or redirected to a file
grep --color=always 'test' filename #Search test from filename and highlight the matched string/results
grep 'test' test.txt
grep ‘test’ test.txt
grep --color[auto/never/always] 'test' test.txt
grep –color[auto/never/always] ‘test’ test.txt

Simple search

Search www from /etc/passwd file

grep www /etc/passwd
grep www /etc/passwd
grep www /etc/passwd

Search www from /etc/passwd file with case insensitive switch “-i” (Means WWW www WWw wwW WwW wWw will be included in the results as well)

grep -i 'www' /etc/passwd

Search recursively

Search all files from a folder for a text

e.g. Search “configured” from “/etc” folder

grep -r 'disabled' /etc
 
or
 
grep -R 'disabled' /etc
grep -r 'disabled' /etc
grep -r ‘disabled’ /etc

To eliminate file name

grep -hr 'disabled' /etc
 
or
 
grep -h -r 'disabled' /etc
grep -hr 'disabled' /etc
grep -hr ‘disabled’ /etc

Search with “-w” match only whole words switch

grep -w 'word' filename
grep -w test test.txt
grep -w test test.txt

Use egrep to search two different words

egrep -w 'word1|word2' filename
egrep 'test|testaaa' test.txt
egrep ‘test|testaaa’ test.txt
egrep -w 'test|testaaa' test.txt
egrep -w ‘test|testaaa’ test.txt

Show count of matching results & Show number of the row the results at in the file

#Show count of matching results
grep -c -w test test.txt
#Show number of the row the results at in the file
grep -n -w test test.txt
Show count of matching results, Show number of the row the results at in the file
Show count of matching results, Show number of the row the results at in the file

Search with inverse match (Exclude the string)

grep -v excludeWord filename
grep -v text test.txt
grep -v text test.txt
grep -vn text test.txt
grep -vn text test.txt

Use grep with Linux pipeline

grep -i 'searchTerm' command
command | grep -i 'model'

Show hard drive name

dmesg | egrep '(s|h)d[a-z]'

Show cpu model

#With pipeline
cat /proc/cpuinfo | grep -i 'Model'
 
#Without pipeine
grep -i 'Model' /proc/cpuinfo
grep with/without Linux pipeline
grep with/without Linux pipeline

List files which includes the search term

Search all “.txt” files which contains “test”

grep -l 'test' *.txt
grep -l 'test' *.txt
grep -l ‘test’ *.txt

Some switches of grep command

-cCount of occurrence
-hRemove file name and path from results
-iCase insensitive
-lPrint only names of FILEs with selected lines
-nPrint line numbers
-RRecursive search, obey all symbolic links
-rRecursive search all folders
-vReverse match
-wMatch only whole words
–colorApply/Disable color scheme for search results

Help page of grep

Usage: grep [OPTION]... PATTERNS [FILE]...
Search for PATTERNS in each FILE.
Example: grep -i 'hello world' menu.h main.c
PATTERNS can contain multiple patterns separated by newlines.
Pattern selection and interpretation:
  -E, --extended-regexp     PATTERNS are extended regular expressions
  -F, --fixed-strings       PATTERNS are strings
  -G, --basic-regexp        PATTERNS are basic regular expressions
  -P, --perl-regexp         PATTERNS are Perl regular expressions
  -e, --regexp=PATTERNS     use PATTERNS for matching
  -f, --file=FILE           take PATTERNS from FILE
  -i, --ignore-case         ignore case distinctions in patterns and data
      --no-ignore-case      do not ignore case distinctions (default)
  -w, --word-regexp         match only whole words
  -x, --line-regexp         match only whole lines
  -z, --null-data           a data line ends in 0 byte, not newline
Miscellaneous:
  -s, --no-messages         suppress error messages
  -v, --invert-match        select non-matching lines
  -V, --version             display version information and exit
      --help                display this help text and exit
Output control:
  -m, --max-count=NUM       stop after NUM selected lines
  -b, --byte-offset         print the byte offset with output lines
  -n, --line-number         print line number with output lines
      --line-buffered       flush output on every line
  -H, --with-filename       print file name with output lines
  -h, --no-filename         suppress the file name prefix on output
      --label=LABEL         use LABEL as the standard input file name prefix
  -o, --only-matching       show only nonempty parts of lines that match
  -q, --quiet, --silent     suppress all normal output
      --binary-files=TYPE   assume that binary files are TYPE;
                            TYPE is 'binary', 'text', or 'without-match'
  -a, --text                equivalent to --binary-files=text
  -I                        equivalent to --binary-files=without-match
  -d, --directories=ACTION  how to handle directories;
                            ACTION is 'read', 'recurse', or 'skip'
  -D, --devices=ACTION      how to handle devices, FIFOs and sockets;
                            ACTION is 'read' or 'skip'
  -r, --recursive           like --directories=recurse
  -R, --dereference-recursive  likewise, but follow all symlinks
      --include=GLOB        search only files that match GLOB (a file pattern)
      --exclude=GLOB        skip files that match GLOB
      --exclude-from=FILE   skip files that match any file pattern from FILE
      --exclude-dir=GLOB    skip directories that match GLOB
  -L, --files-without-match  print only names of FILEs with no selected lines
  -l, --files-with-matches  print only names of FILEs with selected lines
  -c, --count               print only a count of selected lines per FILE
  -T, --initial-tab         make tabs line up (if needed)
  -Z, --null                print 0 byte after FILE name
Context control:
  -B, --before-context=NUM  print NUM lines of leading context
  -A, --after-context=NUM   print NUM lines of trailing context
  -C, --context=NUM         print NUM lines of output context
  -NUM                      same as --context=NUM
      --color[=WHEN],
      --colour[=WHEN]       use markers to highlight the matching strings;
                            WHEN is 'always', 'never', or 'auto'
  -U, --binary              do not strip CR characters at EOL (MSDOS/Windows)
When FILE is '-', read standard input.  With no FILE, read '.' if
recursive, '-' otherwise.  With fewer than two FILEs, assume -h.
Exit status is 0 if any line (or file if -L) is selected, 1 otherwise;
if any error occurs and -q is not given, the exit status is 2.
Report bugs to: [email protected]
GNU grep home page: <http://www.gnu.org/software/grep/>
General help using GNU software: <https://www.gnu.org/gethelp/>

man page of grep

GREP(1)                                                                                                     User Commands                                                                                                     GREP(1)
NAME
       grep, egrep, fgrep, rgrep - print lines that match patterns
SYNOPSIS
       grep [OPTION...] PATTERNS [FILE...]
       grep [OPTION...] -e PATTERNS ... [FILE...]
       grep [OPTION...] -f PATTERN_FILE ... [FILE...]
DESCRIPTION
       grep  searches  for  PATTERNS  in  each FILE.  PATTERNS is one or more patterns separated by newline characters, and grep prints each line that matches a pattern.  Typically PATTERNS should be quoted when grep is used in a
       shell command.
       A FILE of “-” stands for standard input.  If no FILE is given, recursive searches examine the working directory, and nonrecursive searches read standard input.
       In addition, the variant programs egrep, fgrep and rgrep are the same as grep -E, grep -F, and grep -r, respectively.  These variants are deprecated, but are provided for backward compatibility.
OPTIONS
   Generic Program Information
       --help Output a usage message and exit.
       -V, --version
              Output the version number of grep and exit.
   Pattern Syntax
       -E, --extended-regexp
              Interpret PATTERNS as extended regular expressions (EREs, see below).
       -F, --fixed-strings
              Interpret PATTERNS as fixed strings, not regular expressions.
       -G, --basic-regexp
              Interpret PATTERNS as basic regular expressions (BREs, see below).  This is the default.
       -P, --perl-regexp
              Interpret PATTERNS as Perl-compatible regular expressions (PCREs).  This option is experimental when combined with the -z (--null-data) option, and grep -P may warn of unimplemented features.
   Matching Control
       -e PATTERNS, --regexp=PATTERNS
              Use PATTERNS as the patterns.  If this option is used multiple times or is combined with the -f (--file) option, search for all patterns given.  This option can be used to protect a pattern beginning with “-”.
       -f FILE, --file=FILE
              Obtain patterns from FILE, one per line.  If this option is used multiple times or is combined with the -e (--regexp) option, search for all patterns given.  The empty file  contains  zero  patterns,  and  therefore
              matches nothing.
       -i, --ignore-case
              Ignore case distinctions in patterns and input data, so that characters that differ only in case match each other.
       --no-ignore-case
              Do  not ignore case distinctions in patterns and input data.  This is the default.  This option is useful for passing to shell scripts that already use -i, to cancel its effects because the two options override each
              other.
       -v, --invert-match
              Invert the sense of matching, to select non-matching lines.
       -w, --word-regexp
              Select only those lines containing matches that form whole words.  The test is that the matching substring must either be at the beginning of the line, or preceded by a non-word constituent character.  Similarly, it
              must be either at the end of the line or followed by a non-word constituent character.  Word-constituent characters are letters, digits, and the underscore.  This option has no effect if -x is also specified.
       -x, --line-regexp
              Select only those matches that exactly match the whole line.  For a regular expression pattern, this is like parenthesizing the pattern and then surrounding it with ^ and $.
       -y     Obsolete synonym for -i.
   General Output Control
       -c, --count
              Suppress normal output; instead print a count of matching lines for each input file.  With the -v, --invert-match option (see below), count non-matching lines.
       --color[=WHEN], --colour[=WHEN]
              Surround the matched (non-empty) strings, matching lines, context lines, file names, line numbers, byte offsets, and separators (for fields and groups of context lines) with escape sequences to display them in color
              on the terminal.  The colors are defined by the environment variable GREP_COLORS.  The deprecated environment variable GREP_COLOR is still supported, but its setting does not have priority.  WHEN is  never,  always,
              or auto.
       -L, --files-without-match
              Suppress normal output; instead print the name of each input file from which no output would normally have been printed.  The scanning will stop on the first match.
       -l, --files-with-matches
              Suppress normal output; instead print the name of each input file from which output would normally have been printed.  The scanning will stop on the first match.
       -m NUM, --max-count=NUM
              Stop  reading  a  file  after  NUM  matching  lines.   If the input is standard input from a regular file, and NUM matching lines are output, grep ensures that the standard input is positioned to just after the last
              matching line before exiting, regardless of the presence of trailing context lines.  This enables a calling process to resume a search.  When grep stops after NUM matching lines,  it  outputs  any  trailing  context
              lines.  When the -c or --count option is also used, grep does not output a count greater than NUM.  When the -v or --invert-match option is also used, grep stops after outputting NUM non-matching lines.
       -o, --only-matching
              Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line.
       -q, --quiet, --silent
              Quiet; do not write anything to standard output.  Exit immediately with zero status if any match is found, even if an error was detected.  Also see the -s or --no-messages option.
       -s, --no-messages
              Suppress error messages about nonexistent or unreadable files.
   Output Line Prefix Control
       -b, --byte-offset
              Print the 0-based byte offset within the input file before each line of output.  If -o (--only-matching) is specified, print the offset of the matching part itself.
       -H, --with-filename
              Print the file name for each match.  This is the default when there is more than one file to search.
       -h, --no-filename
              Suppress the prefixing of file names on output.  This is the default when there is only one file (or only standard input) to search.
       --label=LABEL
              Display  input  actually  coming  from  standard input as input coming from file LABEL.  This can be useful for commands that transform a file's contents before searching, e.g., gzip -cd foo.gz | grep --label=foo -H
              'some pattern'.  See also the -H option.
       -n, --line-number
              Prefix each line of output with the 1-based line number within its input file.
       -T, --initial-tab
              Make sure that the first character of actual line content lies on a tab stop, so that the alignment of tabs looks normal.  This is useful with options that prefix their output to the actual content: -H,-n,  and  -b.
              In order to improve the probability that lines from a single file will all start at the same column, this also causes the line number and byte offset (if present) to be printed in a minimum size field width.
       -u, --unix-byte-offsets
              Report Unix-style byte offsets.  This switch causes grep to report byte offsets as if the file were a Unix-style text file, i.e., with CR characters stripped off.  This will produce results identical to running grep
              on a Unix machine.  This option has no effect unless -b option is also used; it has no effect on platforms other than MS-DOS and MS-Windows.
       -Z, --null
              Output a zero byte (the ASCII NUL character) instead of the character that normally follows a file name.  For example, grep -lZ outputs a zero byte after each file name instead of the  usual  newline.   This  option
              makes  the  output  unambiguous,  even  in  the presence of file names containing unusual characters like newlines.  This option can be used with commands like find -print0, perl -0, sort -z, and xargs -0 to process
              arbitrary file names, even those that contain newline characters.
   Context Line Control
       -A NUM, --after-context=NUM
              Print NUM lines of trailing context after matching lines.  Places a line containing a group separator (--) between contiguous groups of matches.  With the -o or --only-matching option,  this  has  no  effect  and  a
              warning is given.
       -B NUM, --before-context=NUM
              Print  NUM  lines  of  leading  context  before matching lines.  Places a line containing a group separator (--) between contiguous groups of matches.  With the -o or --only-matching option, this has no effect and a
              warning is given.
       -C NUM, -NUM, --context=NUM
              Print NUM lines of output context.  Places a line containing a group separator (--) between contiguous groups of matches.  With the -o or --only-matching option, this has no effect and a warning is given.
   File and Directory Selection
       -a, --text
              Process a binary file as if it were text; this is equivalent to the --binary-files=text option.
       --binary-files=TYPE
              If a file's data or metadata indicate that the file contains binary data, assume that the file is of type TYPE.  Non-text bytes indicate binary data; these are either output bytes that are improperly encoded for the
              current locale, or null input bytes when the -z option is not given.
              By  default, TYPE is binary, and grep suppresses output after null input binary data is discovered, and suppresses output lines that contain improperly encoded data.  When some output is suppressed, grep follows any
              output with a one-line message saying that a binary file matches.
              If TYPE is without-match, when grep discovers null input binary data it assumes that the rest of the file does not match; this is equivalent to the -I option.
              If TYPE is text, grep processes a binary file as if it were text; this is equivalent to the -a option.
              When type is binary, grep may treat non-text bytes as line terminators even without the -z option.  This means choosing binary versus text can affect whether a pattern matches a file.   For  example,  when  type  is
              binary the pattern q$ might match q immediately followed by a null byte, even though this is not matched when type is text.  Conversely, when type is binary the pattern . (period) might not match a null byte.
              Warning:  The -a option might output binary garbage, which can have nasty side effects if the output is a terminal and if the terminal driver interprets some of it as commands.  On the other hand, when reading files
              whose text encodings are unknown, it can be helpful to use -a or to set LC_ALL='C' in the environment, in order to find more matches even if the matches are unsafe for direct display.
       -D ACTION, --devices=ACTION
              If an input file is a device, FIFO or socket, use ACTION to process it.  By default, ACTION is read, which means that devices are read just as if they were ordinary files.  If ACTION is skip,  devices  are  silently
              skipped.
       -d ACTION, --directories=ACTION
              If  an  input  file  is  a  directory,  use ACTION to process it.  By default, ACTION is read, i.e., read directories just as if they were ordinary files.  If ACTION is skip, silently skip directories.  If ACTION is
              recurse, read all files under each directory, recursively, following symbolic links only if they are on the command line.  This is equivalent to the -r option.
       --exclude=GLOB
              Skip any command-line file with a name suffix that matches the pattern GLOB, using wildcard matching; a name suffix is either the whole name, or a trailing part that starts with  a  non-slash  character  immediately
              after  a  slash  (/) in the name.  When searching recursively, skip any subfile whose base name matches GLOB; the base name is the part after the last slash.  A pattern can use *, ?, and [...] as wildcards, and \ to
              quote a wildcard or backslash character literally.
       --exclude-from=FILE
              Skip files whose base name matches any of the file-name globs read from FILE (using wildcard matching as described under --exclude).
       --exclude-dir=GLOB
              Skip any command-line directory with a name suffix that matches the pattern GLOB.  When searching recursively, skip any subdirectory whose base name matches GLOB.  Ignore any redundant trailing slashes in GLOB.
       -I     Process a binary file as if it did not contain matching data; this is equivalent to the --binary-files=without-match option.
       --include=GLOB
              Search only files whose base name matches GLOB (using wildcard matching as described under --exclude).
       -r, --recursive
              Read all files under each directory, recursively, following symbolic links only if they are on the command line.  Note that if no file operand is given, grep searches the working directory.  This  is  equivalent  to
              the -d recurse option.
       -R, --dereference-recursive
              Read all files under each directory, recursively.  Follow all symbolic links, unlike -r.
   Other Options
       --line-buffered
              Use line buffering on output.  This can cause a performance penalty.
       -U, --binary
              Treat  the  file(s)  as binary.  By default, under MS-DOS and MS-Windows, grep guesses whether a file is text or binary as described for the --binary-files option.  If grep decides the file is a text file, it strips
              the CR characters from the original file contents (to make regular expressions with ^ and $ work correctly).  Specifying -U overrules this guesswork, causing all files to be read and passed to the matching mechanism
              verbatim; if the file is a text file with CR/LF pairs at the end of each line, this will cause some regular expressions to fail.  This option has no effect on platforms other than MS-DOS and MS-Windows.
       -z, --null-data
              Treat  input and output data as sequences of lines, each terminated by a zero byte (the ASCII NUL character) instead of a newline.  Like the -Z or --null option, this option can be used with commands like sort -z to
              process arbitrary file names.
REGULAR EXPRESSIONS
       A regular expression is a pattern that describes a set of strings.  Regular expressions are constructed analogously to arithmetic expressions, by using various operators to combine smaller expressions.
       grep understands three different versions of regular expression syntax: “basic” (BRE), “extended” (ERE) and “perl” (PCRE).  In GNU grep there is no difference in available functionality between basic and extended syntaxes.
       In  other  implementations,  basic  regular  expressions  are  less  powerful.  The following description applies to extended regular expressions; differences for basic regular expressions are summarized afterwards.  Perl-
       compatible regular expressions give additional functionality, and are documented in pcresyntax(3) and pcrepattern(3), but work only if PCRE is available in the system.
       The fundamental building blocks are the regular expressions that match a single character.  Most characters, including all letters and digits, are regular expressions that match themselves.  Any meta-character with special
       meaning may be quoted by preceding it with a backslash.
       The period . matches any single character.  It is unspecified whether it matches an encoding error.
   Character Classes and Bracket Expressions
       A  bracket  expression  is  a  list  of characters enclosed by [ and ].  It matches any single character in that list.  If the first character of the list is the caret ^ then it matches any character not in the list; it is
       unspecified whether it matches an encoding error.  For example, the regular expression [0123456789] matches any single digit.
       Within a bracket expression, a range expression consists of two characters separated by a hyphen.  It matches any single character that sorts between the two characters, inclusive, using the locale's collating sequence and
       character  set.   For  example,  in  the  default  C  locale,  [a-d] is equivalent to [abcd].  Many locales sort characters in dictionary order, and in these locales [a-d] is typically not equivalent to [abcd]; it might be
       equivalent to [aBbCcDd], for example.  To obtain the traditional interpretation of bracket expressions, you can use the C locale by setting the LC_ALL environment variable to the value C.
       Finally, certain named classes of characters are predefined within bracket expressions, as follows.  Their names are self explanatory,  and  they  are  [:alnum:],  [:alpha:],  [:blank:],  [:cntrl:],  [:digit:],  [:graph:],
       [:lower:],  [:print:],  [:punct:],  [:space:], [:upper:], and [:xdigit:].  For example, [[:alnum:]] means the character class of numbers and letters in the current locale.  In the C locale and ASCII character set encoding,
       this is the same as [0-9A-Za-z].  (Note that the brackets in these class names are part of the symbolic names, and must be included in addition to the brackets delimiting the bracket expression.)  Most meta-characters lose
       their special meaning inside bracket expressions.  To include a literal ] place it first in the list.  Similarly, to include a literal ^ place it anywhere but first.  Finally, to include a literal - place it last.
   Anchoring
       The caret ^ and the dollar sign $ are meta-characters that respectively match the empty string at the beginning and end of a line.
   The Backslash Character and Special Expressions
       The  symbols \< and \> respectively match the empty string at the beginning and end of a word.  The symbol \b matches the empty string at the edge of a word, and \B matches the empty string provided it's not at the edge of
       a word.  The symbol \w is a synonym for [_[:alnum:]] and \W is a synonym for [^_[:alnum:]].
   Repetition
       A regular expression may be followed by one of several repetition operators:
       ?      The preceding item is optional and matched at most once.
       *      The preceding item will be matched zero or more times.
       +      The preceding item will be matched one or more times.
       {n}    The preceding item is matched exactly n times.
       {n,}   The preceding item is matched n or more times.
       {,m}   The preceding item is matched at most m times.  This is a GNU extension.
       {n,m}  The preceding item is matched at least n times, but not more than m times.
   Concatenation
       Two regular expressions may be concatenated; the resulting regular expression matches any string formed by concatenating two substrings that respectively match the concatenated expressions.
   Alternation
       Two regular expressions may be joined by the infix operator |; the resulting regular expression matches any string matching either alternate expression.
   Precedence
       Repetition takes precedence over concatenation, which in turn takes precedence over alternation.  A whole expression may be enclosed in parentheses to override these precedence rules and form a subexpression.
   Back-references and Subexpressions
       The back-reference \n, where n is a single digit, matches the substring previously matched by the nth parenthesized subexpression of the regular expression.
   Basic vs Extended Regular Expressions
       In basic regular expressions the meta-characters ?, +, {, |, (, and ) lose their special meaning; instead use the backslashed versions \?, \+, \{, \|, \(, and \).
EXIT STATUS
       Normally the exit status is 0 if a line is selected, 1 if no lines were selected, and 2 if an error occurred.  However, if the -q or --quiet or --silent is used and a line is selected, the exit status is 0 even if an error
       occurred.
ENVIRONMENT
       The behavior of grep is affected by the following environment variables.
       The  locale  for category LC_foo is specified by examining the three environment variables LC_ALL, LC_foo, LANG, in that order.  The first of these variables that is set specifies the locale.  For example, if LC_ALL is not
       set, but LC_MESSAGES is set to pt_BR, then the Brazilian Portuguese locale is used for the LC_MESSAGES category.  The C locale is used if none of these environment variables are set, if the locale catalog is not installed,
       or if grep was not compiled with national language support (NLS).  The shell command locale -a lists locales that are currently available.
       GREP_OPTIONS
              This variable specifies default options to be placed in front of any explicit options.  As this causes problems when writing portable scripts, this feature will be removed in a future release of grep, and grep warns
              if it is used.  Please use an alias or script instead.
       GREP_COLOR
              This variable specifies the color used to highlight matched (non-empty) text.  It is deprecated in favor of GREP_COLORS, but still supported.  The mt, ms, and mc capabilities of GREP_COLORS have  priority  over  it.
              It  can only specify the color used to highlight the matching non-empty text in any matching line (a selected line when the -v command-line option is omitted, or a context line when -v is specified).  The default is
              01;31, which means a bold red foreground text on the terminal's default background.
       GREP_COLORS
              Specifies the colors and other attributes used to highlight various parts of the output.  Its value is a colon-separated list of capabilities that defaults to  ms=01;31:mc=01;31:sl=:cx=:fn=35:ln=32:bn=32:se=36  with
              the rv and ne boolean capabilities omitted (i.e., false).  Supported capabilities are as follows.
              sl=    SGR  substring for whole selected lines (i.e., matching lines when the -v command-line option is omitted, or non-matching lines when -v is specified).  If however the boolean rv capability and the -v command-
                     line option are both specified, it applies to context matching lines instead.  The default is empty (i.e., the terminal's default color pair).
              cx=    SGR substring for whole context lines (i.e., non-matching lines when the -v command-line option is omitted, or matching lines when -v is specified).  If however the boolean rv capability and the  -v  command-
                     line option are both specified, it applies to selected non-matching lines instead.  The default is empty (i.e., the terminal's default color pair).
              rv     Boolean value that reverses (swaps) the meanings of the sl= and cx= capabilities when the -v command-line option is specified.  The default is false (i.e., the capability is omitted).
              mt=01;31
                     SGR  substring  for  matching  non-empty  text  in  any matching line (i.e., a selected line when the -v command-line option is omitted, or a context line when -v is specified).  Setting this is equivalent to
                     setting both ms= and mc= at once to the same value.  The default is a bold red text foreground over the current line background.
              ms=01;31
                     SGR substring for matching non-empty text in a selected line.  (This is only used when the -v command-line option is omitted.)  The effect of the sl= (or cx= if rv) capability remains active when  this  kicks
                     in.  The default is a bold red text foreground over the current line background.
              mc=01;31
                     SGR  substring for matching non-empty text in a context line.  (This is only used when the -v command-line option is specified.)  The effect of the cx= (or sl= if rv) capability remains active when this kicks
                     in.  The default is a bold red text foreground over the current line background.
              fn=35  SGR substring for file names prefixing any content line.  The default is a magenta text foreground over the terminal's default background.
              ln=32  SGR substring for line numbers prefixing any content line.  The default is a green text foreground over the terminal's default background.
              bn=32  SGR substring for byte offsets prefixing any content line.  The default is a green text foreground over the terminal's default background.
              se=36  SGR substring for separators that are inserted between selected line fields (:), between context line fields, (-), and between groups of adjacent lines when nonzero context is specified (--).  The default  is
                     a cyan text foreground over the terminal's default background.
              ne     Boolean  value  that prevents clearing to the end of line using Erase in Line (EL) to Right (\33[K) each time a colorized item ends.  This is needed on terminals on which EL is not supported.  It is otherwise
                     useful on terminals for which the back_color_erase (bce) boolean terminfo capability does not apply, when the chosen highlight colors do not affect the background, or when EL is too slow or  causes  too  much
                     flicker.  The default is false (i.e., the capability is omitted).
              Note that boolean capabilities have no =... part.  They are omitted (i.e., false) by default and become true when specified.
              See  the  Select Graphic Rendition (SGR) section in the documentation of the text terminal that is used for permitted values and their meaning as character attributes.  These substring values are integers in decimal
              representation and can be concatenated with semicolons.  grep takes care of assembling the result into a complete SGR sequence (\33[...m).  Common values to concatenate include 1 for bold, 4  for  underline,  5  for
              blink,  7  for  inverse,  39  for default foreground color, 30 to 37 for foreground colors, 90 to 97 for 16-color mode foreground colors, 38;5;0 to 38;5;255 for 88-color and 256-color modes foreground colors, 49 for
              default background color, 40 to 47 for background colors, 100 to 107 for 16-color mode background colors, and 48;5;0 to 48;5;255 for 88-color and 256-color modes background colors.
       LC_ALL, LC_COLLATE, LANG
              These variables specify the locale for the LC_COLLATE category, which determines the collating sequence used to interpret range expressions like [a-z].
       LC_ALL, LC_CTYPE, LANG
              These variables specify the locale for the LC_CTYPE category, which determines the type of characters, e.g., which characters are whitespace.  This category also determines the character encoding, that  is,  whether
              text is encoded in UTF-8, ASCII, or some other encoding.  In the C or POSIX locale, all characters are encoded as a single byte and every byte is a valid character.
       LC_ALL, LC_MESSAGES, LANG
              These variables specify the locale for the LC_MESSAGES category, which determines the language that grep uses for messages.  The default C locale uses American English messages.
       POSIXLY_CORRECT
              If  set, grep behaves as POSIX requires; otherwise, grep behaves more like other GNU programs.  POSIX requires that options that follow file names must be treated as file names; by default, such options are permuted
              to the front of the operand list and are treated as options.  Also, POSIX requires that unrecognized options be diagnosed as “illegal”, but since they are not really against the law the default is to  diagnose  them
              as “invalid”.  POSIXLY_CORRECT also disables _N_GNU_nonoption_argv_flags_, described below.
       _N_GNU_nonoption_argv_flags_
              (Here  N  is  grep's numeric process ID.)  If the ith character of this environment variable's value is 1, do not consider the ith operand of grep to be an option, even if it appears to be one.  A shell can put this
              variable in the environment for each command it runs, specifying which operands are the results of file name wildcard expansion and therefore should not be treated as options.  This behavior is available  only  with
              the GNU C library, and only when POSIXLY_CORRECT is not set.
NOTES
       This man page is maintained only fitfully; the full documentation is often more up-to-date.
COPYRIGHT
       Copyright 1998-2000, 2002, 2005-2020 Free Software Foundation, Inc.
       This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
BUGS
   Reporting Bugs
       Email  bug  reports to the bug-reporting address ⟨[email protected]⟩.  An email archive ⟨https://lists.gnu.org/mailman/listinfo/bug-grep⟩ and a bug tracker ⟨https://debbugs.gnu.org/cgi/pkgreport.cgi?package=grep⟩ are avail‐
       able.
   Known Bugs
       Large repetition counts in the {n,m} construct may cause grep to use lots of memory.  In addition, certain other obscure regular expressions require exponential time and space, and may cause grep to run out of memory.
       Back-references are very slow, and may require exponential time.
EXAMPLE
       The following example outputs the location and contents of any line containing “f” and ending in “.c”, within all files in the current directory whose names contain “g” and end in “.h”.  The -n option outputs line numbers,
       the -- argument treats expansions of “*g*.h” starting with “-” as file names not options, and the empty file /dev/null causes file names to be output even if only one file name happens to be of the form “*g*.h”.
         $ grep -n -- 'f.*\.c$' *g*.h /dev/null
         argmatch.h:1:/* definitions and prototypes for argmatch.c
       The only line that matches is line 1 of argmatch.h.  Note that the regular expression syntax used in the pattern differs from the globbing syntax that the shell uses to match file names.
SEE ALSO
   Regular Manual Pages
       awk(1), cmp(1), diff(1), find(1), perl(1), sed(1), sort(1), xargs(1), read(2), pcre(3), pcresyntax(3), pcrepattern(3), terminfo(5), glob(7), regex(7).
   Full Documentation
       A complete manual ⟨https://www.gnu.org/software/grep/manual/⟩ is available.  If the info and grep programs are properly installed at your site, the command
              info grep
       should give you access to the complete manual.
GNU grep 3.4                                                                                                  2019-12-29                                                                                                      GREP(1)

Resources

Wikipedia – grep


How to: Find which process is causing high CPU usage Linux/Ubutu/Debian/Kali Linux/CentOS/RHEL

“top” command

top shows CPU usage in real time.

By default, it lists process by their CPU usage, refreshes every 5 seconds.

We can use following command to show top 10 processes with highest CPU usage.

top -b | head -10
[email protected]:~# top -b | head -10
top - 02:05:40 up 20:24,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 165 total,   1 running, 164 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.1 us,  3.1 sy,  0.0 ni, 93.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3913.3 total,   2066.4 free,    617.4 used,   1229.6 buff/cache
MiB Swap:   4094.0 total,   3850.2 free,    243.8 used.   3042.1 avail Mem 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  61649 root      20   0    9144   3560   3164 R   6.7   0.1   0:00.01 top
      1 root      20   0  166420   5908   3936 S   0.0   0.1   0:04.54 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.03 kthreadd
  • -b : Batch mode.
  • head -10: Display first 10 lines in the output.
  • PID : Unique ID of the process.
  • USER : Owner of the process.
  • PR : priority of the process.
  • NI : The NICE value of the process.
  • VIRT : How much virtual memory used by the process.
  • RES : How much physical memory used by the process.
  • SHR : How much shared memory used by the process.
  • S : This indicates the status of the process: S=sleep R=running Z=zombie.
  • %CPU : The percentage of CPU used by the process.
  • %MEM : The percentage of RAM used by the process.
  • TIME+ : How long the process being running.
  • COMMAND : Name of the process.

“ps” command

“ps” stands for “processes status”, it display the information about the active/running processes on the system.

We can use following command to find out high CPU usage processes.

ps -eo pid,ppid,%mem,%cpu,cmd --sort=-%cpu | head
[email protected]:~# ps -eo pid,ppid,%mem,%cpu,cmd --sort=-%cpu | head
    PID    PPID %MEM %CPU CMD
     78       2  0.0  0.4 [kswapd0]
    639     576  4.1  0.4 /usr/lib/xorg/Xorg :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
    262       2  0.0  0.2 [kworker/1:1H-kblockd]
    267       2  0.0  0.1 [kworker/0:1H-kblockd]
    889     823  0.8  0.1 xfwm4 --display :0.0 --sm-client-id 24f144caf-a490-40f0-afc0-fd75665210e1
      1       0  0.1  0.0 /sbin/init splash
      2       0  0.0  0.0 [kthreadd]
      3       2  0.0  0.0 [rcu_gp]
      4       2  0.0  0.0 [rcu_par_gp]

To see the command name instead of full path.

ps -eo pid,ppid,%mem,%cpu,comm --sort=-%cpu | head
[email protected]:~# ps -eo pid,ppid,%mem,%cpu,comm --sort=-%cpu | head
    PID    PPID %MEM %CPU COMMAND
     78       2  0.0  0.4 kswapd0
    639     576  4.1  0.4 Xorg
    262       2  0.0  0.2 kworker/1:1H-kblockd
    267       2  0.0  0.1 kworker/0:1H-kblockd
    889     823  0.8  0.1 xfwm4
      1       0  0.1  0.0 systemd
      2       0  0.0  0.0 kthreadd
      3       2  0.0  0.0 rcu_gp
      4       2  0.0  0.0 rcu_par_gp
  • -e : Select all processes.
  • -o : To customize a output format.
  • –sort=-%cpu : Sort the ouput based on CPU usage.
  • head : To display first 10 lines of the output
  • PID : Unique ID of the process.
  • PPID : Unique ID of the parent process.
  • %MEM : The percentage of RAM used by the process.
  • %CPU : The percentage of CPU used by the process.
  • Command : Name of the process.

htop

htop is a command line utility that allows you to interactively monitor your system’s vital resources or server’s processes in real time

You might need to install htop first

#Debian/Ubuntu/Kali Linux etc.
sudo apt install htop
 
#CentOS/RHEL etc.
sudo yum install htop

To launch htop (Use “q” key to exit)

htop
htop
htop

We can easily sort the processes by their Priority, Nice, Virtual memory usage, Memory, CPU and running time etc. Simply by clicking on the column header.

glances

glances is another utility which can be easily installed and used system resource monitoring tool.

Install

#Debian/Ubuntu/Kali Linux etc.
sudo apt install glances
 
#CentOS/RHEL etc.
sudo yum install glances 

Launch (Use “q” key to exit)

glances

Linux Bash/Shell simple, basic flow control

“if else” and pass variable to the script

if [condition]
 then
     //if block code
 else
     // else block code
 fi
#!/bin/bash
# https://dannyda.com
if [ $1 == $2 ];then
	echo "Numbers are equal ("$1" = "$2")"
elif [ $1 -gt $2 ];then
	echo "1st Number is greater than 2nd Number ("$1" > "$2")"
elif [ $1 -lt $2 ];then
	echo "1st Number is smaller than 2nd Number ("$1" < "$2")"
else
	echo "error"
fi
if else, pass variables to script
if else, pass variables to script

for Loop

for [var] in [list]
do
    //command1
    //command2
done
#!/bin/bash
# https://dannyda.com
for variable in 1 2 3 4 $(echo 7)
do
	echo "The value is: $variable"
done
for i in {1..3}
do
	echo "Loop $i times"
done
echo "2 Done"
echo
for i in {0..10..2}
do 
	echo "Step of 2, $i times"
done
echo "3 Done"
echo
for loop
for loop

while Loop

while [condition]
do
    //command1
    //command2
done
while [condition]; do commands; done
#!/bin/bash
# https://dannyda.com
i=0
while [[ $i<5 ]]
do
    echo $i
    let "i++"
done
echo "Job Done"
echo
i=0
while [[ $i<5 ]]; do echo $1 $((i++)); done
while loop
while loop

while Loop with input (Interactive script)

while read [condition]
do
    //command1
    //command2
done
#!/bin/bash
# https://dannyda.com
echo '<CTRL-D> to exit'
while read input
do
	echo "The number is $input"
done
while with input (interactive script)
while with input (interactive script)

until Loop

Until loop is opposite to while loop, it execute the given commands as long as the given condition evaluates is false

until [condition]
do
    //command1
    //command2
done
#!/bin/bash
# https://dannyda.com
i=0
until [ $i -gt 5 ]
do
  echo Counter: $i
  ((i++))
done
echo "-----------"
i=0
while [ $i -lt 6 ]
do
  echo Counter: $i
  ((i++))
done
until loop
until loop

Infinite while Loop

Use Ctrl + C to stop/exit the loop

while true
do
    //command1
    //command2
done
#!/bin/bash
# https://dannyda.com
while true
do
	echo a
done
infinite while loop
infinite while loop

Infinite for Loop

Use Ctrl + C to stop/exit the loop

for (( ; ; ))
do
    //command1
    //command2
done

infinite for loop
infinite for loop

Options/Case/Switch/Select Case

case  $variable  in
     pattern1)
         command1
         command2
         …
         ….
     ;;
     pattern2)
         command1
         command2
         …
         ….
     ;;
     patternN)
         command1
         command2
         …
         ….
     ;;
     *)
         command1
         command2
         …
         ….
 esac
case  $variable  in
     pattern1|pattern2|pattern3)
         command1
         command2
         …
         ….
     ;;
     pattern4|pattern5|pattern6)
         command1
         command2
         …
         ….
     ;;
     pattern7|pattern8|patternN)
         command1
         command2
         …
         ….
     ;;
     *)
         command1
         command2
         …
         ….
 esac
esacThe EOF for case
)the end of the pattern
*)Else, if doesn’t match anything
;;The end of current case clause
#!/bin/bash
# https://dannyda.com
case $1 in
    a) echo 'Case a'
    ;;
    b) echo 'Case b'
    ;;
    3) echo 'Case 3'
    ;;
    *) echo 'Not Case a or Case b or Case 3'
    ;;
esac
Case
Case

Different ways to delete/remove folder/directory in Linux (Ubuntu, Kali Linux etc.) (Empty and non-empty folders)

rmdir

Used for removing empty folders only. If the folder is not empty, error will be returned.

rmdir folderName
rmdir
rmdir

rm

Used for removing empty and non-empty folders (and files as well)

#Force to remove folder and sub-folders without warning.
rm -rf folderName
#Remove multiple folders
rm -rf folder1 folder2 folder3
rm -rf aa, rm -rf b c
rm -rf aa, rm -rf b c

Using “rm” and “find” command to do conditional deleting

  • Find a single folder within current folder, then delete it.
find . -type d -name "folder" -exec rm -rf {} +
Find and Delete single folder
Find and Delete single folder
.Current folder
-type dOnly search folder/dirctory
-nameName of the folder
-exec rm -rfRun rm -rf command, delete the folder
{} +append the command to the end of the rm -rf
  • Delete all empty folders
find . -type d -empty -delete
Find and delete all empty folders
Find and delete all empty folders
-emptyOnly delete empty folders
-deleteDelete all empty folders, including empty sub-folders

How to: Create/Add/Delete/Remove/List Users/Groups with Command Prompt/Command line in Windows (net command)

Keywords: Windows command prompt, command line, cmd, Add Users, Create Users, Delete Users, Remove Users, List Users, Add local groups, Create local groups, Delete local groups, List local groups, net command

Launch the Command Prompt (In Admin mode)

We should launch the Command Prompt in Administrator mode.

  • Use Win + X key combination -> “Windows PowerShell (Admin)”

or

  • Open start menu -> Type “cmd” -> Right click on “Command Prompt” -> Run as administrator

Add a New User

net user username password /add

net user user1 1234 /add
net user username password /add
net user username password /add

Add a user to a group

net localgroup administrator username /add

e.g. add user to local administrator group

net localgroup administrators user1 /add
net localgroup administrators user1 /add
net localgroup administrators user1 /add

Add a New Local Group

net localgroup newGroupName /add

net localgroup newgroup /add
net localgroup newgroup /add
net localgroup newgroup /add

List All Users

net user

net user
net user
net user

List information about a specific User

net user userName

net user win10
net user win10
net user win10

List All Users with Sid

WMIC useraccount get name

WMIC useraccount get name,sid

WMIC useraccount get name
WMIC useraccount get name,sid
WMIC useraccount get name,sid
WMIC useraccount get name,sid

List All Local Groups

net localgroup

net localgroup
net localgroup
net localgroup

Remove a User from a Local Group

net localgroup groupName userName /delete

net localgroup administrators user1 /delete
net localgroup administrators user1 /delete
net localgroup administrators user1 /delete

Remove a Local Group

net localgroup groupname /delete

net localgroup mygroup /detele
net localgroup mygroup /detele
net localgroup mygroup /detele

Remove a User

net user username /delete

net user user1 /delete
net user user1 /delete
net user user1 /delete

Linux – Basics, Useful Terminal commands and Basic File manipulation, (QuickStart)

Basics/ Useful Terminal commands

Home Directory

Where most of the time the user files are stored.

Usually, the home directory is stored under “/home/” e.g. If we have created a user named “Bob” his home directory will be “/home/bob/” Bob will have full control over this folder including Create, Retrieve, Update, Delete (CRUD).

The corresponding folder for Microsoft Windows is usually “C:\Users\bob\”

Check current working directory

Which directory am I working in?

pwd
pwd
pwd

Switch directory/Directory navigation

Switch working directory, navigating through different directory/folders

e.g. Current working directory is “/tmp”, we want to navigate to “/tmp/test/”

cd test
cd
cd

Or using the absolute path

cd /tmp/test

List files and folders under a folder

List files and folders under current directory

ls
ls
ls

list files and folders under specific path

ls /tmp/test
ls /tmp/test
ls /tmp/test

Color scheme for “ls” command

BlueDirectory/Folder
RedArchive files
WhiteText file

Create a new text file

Create a “b.txt” text file under current working directory. File extension like “txt” is not necessary, but it’s easier for users when the extension is present.

touch b.txt
touch b.txt
touch b.txt

Create a new folder

mkdir newFolder

We can also use standard stream redirection to create file and write content to the text file.

ls > ls.txt
ls > ls.txt
ls > ls.txt

The output of “ls” command will be redirected and written to the “ls.txt” file

Copy, Delete, Move/Rename

Copy

cp ls.txt ls1.txt
cp
cp

Delete

rm ls1.txt
rm ls1.txt
rm ls1.txt
  • -f: –force ignore nonexistent files and arguments, never prompt
  • -r, -R: –recursive remove directories and their contents recursively
  • /*: Anything under root directory (Warning: Use carefully and do not try /* without full backup)
#Delete trash folder and everything within it
rm -rf trash

Rename

mv ls.txt ls1.txt

Move

#Under same working directory/folder
mv ls.txt ls1.txt
#With full path
mv /tmp/test/ls.txt /tmp/test/ls1.txt 
#Move to another folder with same file name
mv /tmp/test/ls.txt /tmp/ls.txt
#Move to another folder with different file name
mv /tmp/test/ls.txt /tmp/new.txt

Search/Find file

Find

Search from root level with “find”

find / -name abc.txt
find / -name abc.txt
find / -name abc.txt

Search from other specific path with “find”

find /tmp/test -name abc.txt

locate

Search with “locate”

updatedb
locate abc.txt
locate abc.txt
locate abc.txt

Search from other specific path with “locate”

updatedb
locate "/bob/*abc.txt"
locate "/root/*.abc.txt"
locate “/root/*.abc.txt”

“find” is usually slower than “locate”

It’s best to do “updatedb” before using “locate”

Search text in file

grep

Search text file (case-sensitive by default)

grep TextWeWantToSearch /tmp/test/textfile.txt

Search test file (ignore case)

grep -i TextWeWantToSearch /tmp/test/textfile.txt
grep, search text from file
grep, search text from file

Search all “txt” files under same directory

grep SearchCriteria /tmp/test/*.txt
grep, search text from any txt file under folder
grep, search text from any txt file under folder

Add “-n” switch to show line numbers

grep -n SearchCriteria /tmp/test/*.txt
grep with line numbers
grep with line numbers

If we want to search sub-directories as well, we need to add “-R” (recursive) parameter

grep -nR SearchCriteria /tmp/test/*
grep including sub-directories and any file with line number
grep including sub-directories and any file with line number

Autocomplete

When typing command, e.g. “fin” then hit “Tab” key twice.

Autocomplete
Autocomplete

If we continue to type “c” at the end, then hit “Enter”, “finecore” will automatically completed.

Same technique can be used for folder names as well. If there is only one folder start with “mus” then after we hit “Tab” key, music will be automatically completed, If there is multiple folder stars with “mus”, then hit “Tab” key twice, options will appear just like for other commands.

Basic File manipulation

Check metadata (Privilege, Size etc.)

ls -l
ls -l
ls -l

Basically the layout has 7 sections

File permissions + No. of hard links + File owner + Group the file belongs + File Size + File modification time + File Name

The 1st character represent file type

dFolder/Directory
File
sSocket
lLink

etc.

ls, output explained
ls, output explained

Following 9 characters:

r: Read, w: Write, x: Execute -: No privilege

The following group of three characters represent owner’s permission, next group of three characters represent the permission for the group which the user belongs to, last three characters represent privilege for others who is not owner and not the group the file belongs to.

Add “-h” switch together with “-l” to list files and folders with human readable size format

ls -hl
ls -hl
ls -hl

Add “-a” switch to show all files and folders including hidden ones

ls -al

We can also use stat to find detailed information about a file/folder/directory

stat filename

Output

[email protected]:/# stat home
   File: home
   Size: 4096            Blocks: 8          IO Block: 4096   directory
 Device: 801h/2049d      Inode: 5242881     Links: 2
 Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
 Access: 2020-01-05 15:31:07.026808761 +1100
 Modify: 2019-11-09 20:18:26.000000000 +1100
 Change: 2019-12-01 13:40:26.043449434 +1100
  Birth: -

Change/Modify file permission

Extended reading: Unix/Linux (Ubuntu, Debian, Kali Linux etc.) Privilege Management

Method 1: “chmod” with characters

+: Add, -: Remove, =: set permission.

a: all, o: others, u:user/owner,g:group

Set Read,Write,Execute rights for the file owner/user

chmod u=rwx a.txt

Add write permission for others

chmod o+w a.txt

Remove read permission from group

chmod g-r a.txt

Add execute permission for all users

chmod a+x a.sh

If we do not define which user/group etc to set the permission for. It will change the permission for all, including owner, group, others.

chmod +x a.sh

Method 2: “chmod” with numbers

4: r, 2: w, 1: x

They adds up in group of three.

Following command gives owner, group and others: rwx, rw, r permission. 7 = sum of r,w,x. 6 = sum of r, w. 1 = x.

chmod 761 a.txt

Change/Modify Access time/Modify time

Change access and modify time to 2001-01-01 00:00:01.000000000

touch -t 200101010000.01 file
touch -t 200101010000.01 a
touch -t 200101010000.01 a

Other ways to define date/time

touch -d "2001-01-01 00:00:01" -a file
 
touch -d "5 hours ago" -a file
 
touch -d "next monday" -m file

-d: Human readable date format

-a: access time

-m: modification time

Find out File type

file FileName
file /folder/*
file *
file *
file FileName
file FileName

Can’t run downloaded Program/Execute script etc.

We need to give the program or the script execute permission for it to run.

We can use “chmod” command which we have mentioned before.

Folder/Directory/File disk usage

#All file and folders under this folder
du -h *
du -h *, ls-hl
du -h *, ls-hl

-h switch represents human readable unit

Note: “du” outputs size occupied on the disk not the actual file size, whereas “ls -hl” outputs file size.

du -h a.png
du -h a.png
a.png Properties in Microsoft Windows
a.png Properties in Microsoft Windows

-s: Show total size of all files and sub-folders

-S: Show size of individual folders

du -hs
du -hS
du -hs, du -hS
du -hs, du -hS

Use “df” to show size of system folders

df
df -h
df -h, df
df -h, df

Compress/Extract file/Archive file

Deal with .tar, .tar.gz, .bz2 files

Create archive (.tar)

tar -cvf archive.tar folderName/

-c: Create archive

-v: Show files added to tarball

-f: Specify file name

Extract archive

tar -xvf archive.tar

-x: Extract

(-C: To extract to other location)

Create archive (.tar.gz, .tar.bz2)

To create “.tar.gz” use “-z”

tar -zcvf archive.tar.gz folderName/

To create “tar.bz2” use “-j”

tar -jcvf archive.tar.bz2 folderName/

To list content within archive file

tar -tvf archive.tar
tar -tvf archive.tar.gz
tar -tvf archive.tar.bz2

To extract “.tar.gz” and “tar.bz2”

tar -zxvf archive.tar.gz
tar -jxvf archive.tarb.z2

Extract to specific directory/folder

tar -zxvf archive.tar.gz --directory /folder/path
tar -jxvf archive.tarb.z2 --directory /folder/path

Parameters

  • x : Extract files
  • f : Tar archive name
  • –directory : Set directory name to extract files
  • -C : Set dir name to extract files
  • -j : Work on .tar.gz file format
  • -z : Work on .tar.bz2 file format
  • -v : Verbose output i.e. show progress on screen

Create Zip file

zip -r archive.zip folder
zip -r archive.zip folder1 folder2
zip archive.zip /path/to/folder1 /path/to/folder2

Extract Zip file

#Extract archive.zip to tmp folder
unzip archive.zip -d /tmp

View and Save output from command at same time, Output to two files at same time

We will use “tee” command

#echo "test" on screen and output to "1.txt" file
echo "test" | tee 1.txt
#echo "test" on screen and output to file "1.txt" and "2.txt"
echo "test" | tee 1.txt > 2.txt

Redirection/Pipeline

System Input/Output

Default Input:Standard InputSTDIN0
Default Output:Standard OutputSTDOUT1
Standard error outputSTDERR22
Standard Input:Keyboard
Standard OutputMonitor/Display
Standard Error OutputMonitor/Display

I/O redirection in Linux

TypeCodeRedirection
stdin0< or <<
stdout1> or >>
stderr22> or 2>>

“>”: Will Overwrite the file

“>>”: Will Append to the file

“set -C”: Disable redirection overwrite. (To force to overwrite use “>|”)

“set +C”: To disable “set -C”

“2>”: Redirect error output

“2>>”: Append mode

Write output and error to different location/file

ls / /aaa1 > /tmp/right1.txt 2> /tmp/err1.txt
ls /aaa1 /tmp/right2.txt 2> /tmp/err2.txt

“/dev/null”: Blackhole, redirect anything to it, nothing will be returned

“&>:”: Redirect standard output and error output to same file (or “2>&1”)

“<“: Use file as input rather than keyboard

echo 'Test message' > output
cat < output
tr 'a-z' 'A-Z' < /etc/fstab

“<<“: End of the file (Here document)

# cat < EOF
> hello
> world
> EOF
'# cat < EOF'
‘# cat < EOF’

“<<<“: The string following the “<<<” becomes the content of the “here” document.

cat <<<'hello world'

“>&2”: Redirect to a specific file descriptor

echo 'Error: Something went wrong' >&2

Pipeline

Use the output from previous command as input for next comand

Command 1 | Command 2 | …

# echo 'Test' | tr 'a-z 'A-Z'
# ls / | tr 'a-z 'A-Z'
# echo 'Test Message!' | tee /tmp/testf.txt

“tee”: Display the content and save to file

Added on 08/02/2020

wget

wget is included in most Linux distros.

We can use wget to download files

wget http://contoso.com/image.jpg

scp

scp – Secure Copy

We can use this command to download/upload files to/from remote server.

e.g. Download file from remote server

scp [email protected]: /path/to/test.txt /path/to/local/directory

e.g. Upload file to remote server

scp /path/to/local/folder [email protected]: /path/to/destination/folder

with “-r” switch, we can download/upload folder

ssh-keygen

Used to create SSH keypair which can be used in Bitbucket, Gitlab etc. to establish secure connection.

ssh-keygen -t ed25519

Combine commands

We can use ; to execute multiple commands without worrying about the previous command succeeded or failed.

ls -l; pwd

Execute second command only if the first command succeeded

mkdir newFolder && cd newFolder

Execute second command only if the first command failed

ping bing.com -c 1 || echo 'Ping bing.com failed'