How to: Search in Linux, How to: Use grep command, How to: Use grep to search

“grep” is very useful when searching text.

On Microsoft Windows, we can use “dngrep” to achieve similar results

grep command

grep is a command-line utility for searching plain-text data sets for lines that match a regular expression. Its name comes from the ed command g/re/p (globally search a regular expression and print), which has the same effect: doing a global search with the regular expression and printing all matching lines. [1]

Some basic grep usage

grep 'test' filename #Search test in file filename
grep 'test' file1 file2 #Search test from file1 and file2
cat filename | grep 'test' #Print out content from filename then search for test
grep --color=never 'test' filename #Search test from filename and don't highlight the results
grep --color=auto 'test' filename #Search test from filename and displays color in the output unless the output is piped to a command, or redirected to a file
grep --color=always 'test' filename #Search test from filename and highlight the matched string/results
grep 'test' test.txt
grep ‘test’ test.txt
grep --color[auto/never/always] 'test' test.txt
grep –color[auto/never/always] ‘test’ test.txt

Simple search

Search www from /etc/passwd file

grep www /etc/passwd
grep www /etc/passwd
grep www /etc/passwd

Search www from /etc/passwd file with case insensitive switch “-i” (Means WWW www WWw wwW WwW wWw will be included in the results as well)

grep -i 'www' /etc/passwd

Search recursively

Search all files from a folder for a text

e.g. Search “configured” from “/etc” folder

grep -r 'disabled' /etc
 
or
 
grep -R 'disabled' /etc
grep -r 'disabled' /etc
grep -r ‘disabled’ /etc

To eliminate file name

grep -hr 'disabled' /etc
 
or
 
grep -h -r 'disabled' /etc
grep -hr 'disabled' /etc
grep -hr ‘disabled’ /etc

Search with “-w” match only whole words switch

grep -w 'word' filename
grep -w test test.txt
grep -w test test.txt

Use egrep to search two different words

egrep -w 'word1|word2' filename
egrep 'test|testaaa' test.txt
egrep ‘test|testaaa’ test.txt
egrep -w 'test|testaaa' test.txt
egrep -w ‘test|testaaa’ test.txt

Show count of matching results & Show number of the row the results at in the file

#Show count of matching results
grep -c -w test test.txt
#Show number of the row the results at in the file
grep -n -w test test.txt
Show count of matching results, Show number of the row the results at in the file
Show count of matching results, Show number of the row the results at in the file

Search with inverse match (Exclude the string)

grep -v excludeWord filename
grep -v text test.txt
grep -v text test.txt
grep -vn text test.txt
grep -vn text test.txt

Use grep with Linux pipeline

grep -i 'searchTerm' command
command | grep -i 'model'

Show hard drive name

dmesg | egrep '(s|h)d[a-z]'

Show cpu model

#With pipeline
cat /proc/cpuinfo | grep -i 'Model'
 
#Without pipeine
grep -i 'Model' /proc/cpuinfo
grep with/without Linux pipeline
grep with/without Linux pipeline

List files which includes the search term

Search all “.txt” files which contains “test”

grep -l 'test' *.txt
grep -l 'test' *.txt
grep -l ‘test’ *.txt

Some switches of grep command

-cCount of occurrence
-hRemove file name and path from results
-iCase insensitive
-lPrint only names of FILEs with selected lines
-nPrint line numbers
-RRecursive search, obey all symbolic links
-rRecursive search all folders
-vReverse match
-wMatch only whole words
–colorApply/Disable color scheme for search results

Help page of grep

Usage: grep [OPTION]... PATTERNS [FILE]...
Search for PATTERNS in each FILE.
Example: grep -i 'hello world' menu.h main.c
PATTERNS can contain multiple patterns separated by newlines.
Pattern selection and interpretation:
  -E, --extended-regexp     PATTERNS are extended regular expressions
  -F, --fixed-strings       PATTERNS are strings
  -G, --basic-regexp        PATTERNS are basic regular expressions
  -P, --perl-regexp         PATTERNS are Perl regular expressions
  -e, --regexp=PATTERNS     use PATTERNS for matching
  -f, --file=FILE           take PATTERNS from FILE
  -i, --ignore-case         ignore case distinctions in patterns and data
      --no-ignore-case      do not ignore case distinctions (default)
  -w, --word-regexp         match only whole words
  -x, --line-regexp         match only whole lines
  -z, --null-data           a data line ends in 0 byte, not newline
Miscellaneous:
  -s, --no-messages         suppress error messages
  -v, --invert-match        select non-matching lines
  -V, --version             display version information and exit
      --help                display this help text and exit
Output control:
  -m, --max-count=NUM       stop after NUM selected lines
  -b, --byte-offset         print the byte offset with output lines
  -n, --line-number         print line number with output lines
      --line-buffered       flush output on every line
  -H, --with-filename       print file name with output lines
  -h, --no-filename         suppress the file name prefix on output
      --label=LABEL         use LABEL as the standard input file name prefix
  -o, --only-matching       show only nonempty parts of lines that match
  -q, --quiet, --silent     suppress all normal output
      --binary-files=TYPE   assume that binary files are TYPE;
                            TYPE is 'binary', 'text', or 'without-match'
  -a, --text                equivalent to --binary-files=text
  -I                        equivalent to --binary-files=without-match
  -d, --directories=ACTION  how to handle directories;
                            ACTION is 'read', 'recurse', or 'skip'
  -D, --devices=ACTION      how to handle devices, FIFOs and sockets;
                            ACTION is 'read' or 'skip'
  -r, --recursive           like --directories=recurse
  -R, --dereference-recursive  likewise, but follow all symlinks
      --include=GLOB        search only files that match GLOB (a file pattern)
      --exclude=GLOB        skip files that match GLOB
      --exclude-from=FILE   skip files that match any file pattern from FILE
      --exclude-dir=GLOB    skip directories that match GLOB
  -L, --files-without-match  print only names of FILEs with no selected lines
  -l, --files-with-matches  print only names of FILEs with selected lines
  -c, --count               print only a count of selected lines per FILE
  -T, --initial-tab         make tabs line up (if needed)
  -Z, --null                print 0 byte after FILE name
Context control:
  -B, --before-context=NUM  print NUM lines of leading context
  -A, --after-context=NUM   print NUM lines of trailing context
  -C, --context=NUM         print NUM lines of output context
  -NUM                      same as --context=NUM
      --color[=WHEN],
      --colour[=WHEN]       use markers to highlight the matching strings;
                            WHEN is 'always', 'never', or 'auto'
  -U, --binary              do not strip CR characters at EOL (MSDOS/Windows)
When FILE is '-', read standard input.  With no FILE, read '.' if
recursive, '-' otherwise.  With fewer than two FILEs, assume -h.
Exit status is 0 if any line (or file if -L) is selected, 1 otherwise;
if any error occurs and -q is not given, the exit status is 2.
Report bugs to: [email protected]
GNU grep home page: <http://www.gnu.org/software/grep/>
General help using GNU software: <https://www.gnu.org/gethelp/>

man page of grep

GREP(1)                                                                                                     User Commands                                                                                                     GREP(1)
NAME
       grep, egrep, fgrep, rgrep - print lines that match patterns
SYNOPSIS
       grep [OPTION...] PATTERNS [FILE...]
       grep [OPTION...] -e PATTERNS ... [FILE...]
       grep [OPTION...] -f PATTERN_FILE ... [FILE...]
DESCRIPTION
       grep  searches  for  PATTERNS  in  each FILE.  PATTERNS is one or more patterns separated by newline characters, and grep prints each line that matches a pattern.  Typically PATTERNS should be quoted when grep is used in a
       shell command.
       A FILE of “-” stands for standard input.  If no FILE is given, recursive searches examine the working directory, and nonrecursive searches read standard input.
       In addition, the variant programs egrep, fgrep and rgrep are the same as grep -E, grep -F, and grep -r, respectively.  These variants are deprecated, but are provided for backward compatibility.
OPTIONS
   Generic Program Information
       --help Output a usage message and exit.
       -V, --version
              Output the version number of grep and exit.
   Pattern Syntax
       -E, --extended-regexp
              Interpret PATTERNS as extended regular expressions (EREs, see below).
       -F, --fixed-strings
              Interpret PATTERNS as fixed strings, not regular expressions.
       -G, --basic-regexp
              Interpret PATTERNS as basic regular expressions (BREs, see below).  This is the default.
       -P, --perl-regexp
              Interpret PATTERNS as Perl-compatible regular expressions (PCREs).  This option is experimental when combined with the -z (--null-data) option, and grep -P may warn of unimplemented features.
   Matching Control
       -e PATTERNS, --regexp=PATTERNS
              Use PATTERNS as the patterns.  If this option is used multiple times or is combined with the -f (--file) option, search for all patterns given.  This option can be used to protect a pattern beginning with “-”.
       -f FILE, --file=FILE
              Obtain patterns from FILE, one per line.  If this option is used multiple times or is combined with the -e (--regexp) option, search for all patterns given.  The empty file  contains  zero  patterns,  and  therefore
              matches nothing.
       -i, --ignore-case
              Ignore case distinctions in patterns and input data, so that characters that differ only in case match each other.
       --no-ignore-case
              Do  not ignore case distinctions in patterns and input data.  This is the default.  This option is useful for passing to shell scripts that already use -i, to cancel its effects because the two options override each
              other.
       -v, --invert-match
              Invert the sense of matching, to select non-matching lines.
       -w, --word-regexp
              Select only those lines containing matches that form whole words.  The test is that the matching substring must either be at the beginning of the line, or preceded by a non-word constituent character.  Similarly, it
              must be either at the end of the line or followed by a non-word constituent character.  Word-constituent characters are letters, digits, and the underscore.  This option has no effect if -x is also specified.
       -x, --line-regexp
              Select only those matches that exactly match the whole line.  For a regular expression pattern, this is like parenthesizing the pattern and then surrounding it with ^ and $.
       -y     Obsolete synonym for -i.
   General Output Control
       -c, --count
              Suppress normal output; instead print a count of matching lines for each input file.  With the -v, --invert-match option (see below), count non-matching lines.
       --color[=WHEN], --colour[=WHEN]
              Surround the matched (non-empty) strings, matching lines, context lines, file names, line numbers, byte offsets, and separators (for fields and groups of context lines) with escape sequences to display them in color
              on the terminal.  The colors are defined by the environment variable GREP_COLORS.  The deprecated environment variable GREP_COLOR is still supported, but its setting does not have priority.  WHEN is  never,  always,
              or auto.
       -L, --files-without-match
              Suppress normal output; instead print the name of each input file from which no output would normally have been printed.  The scanning will stop on the first match.
       -l, --files-with-matches
              Suppress normal output; instead print the name of each input file from which output would normally have been printed.  The scanning will stop on the first match.
       -m NUM, --max-count=NUM
              Stop  reading  a  file  after  NUM  matching  lines.   If the input is standard input from a regular file, and NUM matching lines are output, grep ensures that the standard input is positioned to just after the last
              matching line before exiting, regardless of the presence of trailing context lines.  This enables a calling process to resume a search.  When grep stops after NUM matching lines,  it  outputs  any  trailing  context
              lines.  When the -c or --count option is also used, grep does not output a count greater than NUM.  When the -v or --invert-match option is also used, grep stops after outputting NUM non-matching lines.
       -o, --only-matching
              Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line.
       -q, --quiet, --silent
              Quiet; do not write anything to standard output.  Exit immediately with zero status if any match is found, even if an error was detected.  Also see the -s or --no-messages option.
       -s, --no-messages
              Suppress error messages about nonexistent or unreadable files.
   Output Line Prefix Control
       -b, --byte-offset
              Print the 0-based byte offset within the input file before each line of output.  If -o (--only-matching) is specified, print the offset of the matching part itself.
       -H, --with-filename
              Print the file name for each match.  This is the default when there is more than one file to search.
       -h, --no-filename
              Suppress the prefixing of file names on output.  This is the default when there is only one file (or only standard input) to search.
       --label=LABEL
              Display  input  actually  coming  from  standard input as input coming from file LABEL.  This can be useful for commands that transform a file's contents before searching, e.g., gzip -cd foo.gz | grep --label=foo -H
              'some pattern'.  See also the -H option.
       -n, --line-number
              Prefix each line of output with the 1-based line number within its input file.
       -T, --initial-tab
              Make sure that the first character of actual line content lies on a tab stop, so that the alignment of tabs looks normal.  This is useful with options that prefix their output to the actual content: -H,-n,  and  -b.
              In order to improve the probability that lines from a single file will all start at the same column, this also causes the line number and byte offset (if present) to be printed in a minimum size field width.
       -u, --unix-byte-offsets
              Report Unix-style byte offsets.  This switch causes grep to report byte offsets as if the file were a Unix-style text file, i.e., with CR characters stripped off.  This will produce results identical to running grep
              on a Unix machine.  This option has no effect unless -b option is also used; it has no effect on platforms other than MS-DOS and MS-Windows.
       -Z, --null
              Output a zero byte (the ASCII NUL character) instead of the character that normally follows a file name.  For example, grep -lZ outputs a zero byte after each file name instead of the  usual  newline.   This  option
              makes  the  output  unambiguous,  even  in  the presence of file names containing unusual characters like newlines.  This option can be used with commands like find -print0, perl -0, sort -z, and xargs -0 to process
              arbitrary file names, even those that contain newline characters.
   Context Line Control
       -A NUM, --after-context=NUM
              Print NUM lines of trailing context after matching lines.  Places a line containing a group separator (--) between contiguous groups of matches.  With the -o or --only-matching option,  this  has  no  effect  and  a
              warning is given.
       -B NUM, --before-context=NUM
              Print  NUM  lines  of  leading  context  before matching lines.  Places a line containing a group separator (--) between contiguous groups of matches.  With the -o or --only-matching option, this has no effect and a
              warning is given.
       -C NUM, -NUM, --context=NUM
              Print NUM lines of output context.  Places a line containing a group separator (--) between contiguous groups of matches.  With the -o or --only-matching option, this has no effect and a warning is given.
   File and Directory Selection
       -a, --text
              Process a binary file as if it were text; this is equivalent to the --binary-files=text option.
       --binary-files=TYPE
              If a file's data or metadata indicate that the file contains binary data, assume that the file is of type TYPE.  Non-text bytes indicate binary data; these are either output bytes that are improperly encoded for the
              current locale, or null input bytes when the -z option is not given.
              By  default, TYPE is binary, and grep suppresses output after null input binary data is discovered, and suppresses output lines that contain improperly encoded data.  When some output is suppressed, grep follows any
              output with a one-line message saying that a binary file matches.
              If TYPE is without-match, when grep discovers null input binary data it assumes that the rest of the file does not match; this is equivalent to the -I option.
              If TYPE is text, grep processes a binary file as if it were text; this is equivalent to the -a option.
              When type is binary, grep may treat non-text bytes as line terminators even without the -z option.  This means choosing binary versus text can affect whether a pattern matches a file.   For  example,  when  type  is
              binary the pattern q$ might match q immediately followed by a null byte, even though this is not matched when type is text.  Conversely, when type is binary the pattern . (period) might not match a null byte.
              Warning:  The -a option might output binary garbage, which can have nasty side effects if the output is a terminal and if the terminal driver interprets some of it as commands.  On the other hand, when reading files
              whose text encodings are unknown, it can be helpful to use -a or to set LC_ALL='C' in the environment, in order to find more matches even if the matches are unsafe for direct display.
       -D ACTION, --devices=ACTION
              If an input file is a device, FIFO or socket, use ACTION to process it.  By default, ACTION is read, which means that devices are read just as if they were ordinary files.  If ACTION is skip,  devices  are  silently
              skipped.
       -d ACTION, --directories=ACTION
              If  an  input  file  is  a  directory,  use ACTION to process it.  By default, ACTION is read, i.e., read directories just as if they were ordinary files.  If ACTION is skip, silently skip directories.  If ACTION is
              recurse, read all files under each directory, recursively, following symbolic links only if they are on the command line.  This is equivalent to the -r option.
       --exclude=GLOB
              Skip any command-line file with a name suffix that matches the pattern GLOB, using wildcard matching; a name suffix is either the whole name, or a trailing part that starts with  a  non-slash  character  immediately
              after  a  slash  (/) in the name.  When searching recursively, skip any subfile whose base name matches GLOB; the base name is the part after the last slash.  A pattern can use *, ?, and [...] as wildcards, and \ to
              quote a wildcard or backslash character literally.
       --exclude-from=FILE
              Skip files whose base name matches any of the file-name globs read from FILE (using wildcard matching as described under --exclude).
       --exclude-dir=GLOB
              Skip any command-line directory with a name suffix that matches the pattern GLOB.  When searching recursively, skip any subdirectory whose base name matches GLOB.  Ignore any redundant trailing slashes in GLOB.
       -I     Process a binary file as if it did not contain matching data; this is equivalent to the --binary-files=without-match option.
       --include=GLOB
              Search only files whose base name matches GLOB (using wildcard matching as described under --exclude).
       -r, --recursive
              Read all files under each directory, recursively, following symbolic links only if they are on the command line.  Note that if no file operand is given, grep searches the working directory.  This  is  equivalent  to
              the -d recurse option.
       -R, --dereference-recursive
              Read all files under each directory, recursively.  Follow all symbolic links, unlike -r.
   Other Options
       --line-buffered
              Use line buffering on output.  This can cause a performance penalty.
       -U, --binary
              Treat  the  file(s)  as binary.  By default, under MS-DOS and MS-Windows, grep guesses whether a file is text or binary as described for the --binary-files option.  If grep decides the file is a text file, it strips
              the CR characters from the original file contents (to make regular expressions with ^ and $ work correctly).  Specifying -U overrules this guesswork, causing all files to be read and passed to the matching mechanism
              verbatim; if the file is a text file with CR/LF pairs at the end of each line, this will cause some regular expressions to fail.  This option has no effect on platforms other than MS-DOS and MS-Windows.
       -z, --null-data
              Treat  input and output data as sequences of lines, each terminated by a zero byte (the ASCII NUL character) instead of a newline.  Like the -Z or --null option, this option can be used with commands like sort -z to
              process arbitrary file names.
REGULAR EXPRESSIONS
       A regular expression is a pattern that describes a set of strings.  Regular expressions are constructed analogously to arithmetic expressions, by using various operators to combine smaller expressions.
       grep understands three different versions of regular expression syntax: “basic” (BRE), “extended” (ERE) and “perl” (PCRE).  In GNU grep there is no difference in available functionality between basic and extended syntaxes.
       In  other  implementations,  basic  regular  expressions  are  less  powerful.  The following description applies to extended regular expressions; differences for basic regular expressions are summarized afterwards.  Perl-
       compatible regular expressions give additional functionality, and are documented in pcresyntax(3) and pcrepattern(3), but work only if PCRE is available in the system.
       The fundamental building blocks are the regular expressions that match a single character.  Most characters, including all letters and digits, are regular expressions that match themselves.  Any meta-character with special
       meaning may be quoted by preceding it with a backslash.
       The period . matches any single character.  It is unspecified whether it matches an encoding error.
   Character Classes and Bracket Expressions
       A  bracket  expression  is  a  list  of characters enclosed by [ and ].  It matches any single character in that list.  If the first character of the list is the caret ^ then it matches any character not in the list; it is
       unspecified whether it matches an encoding error.  For example, the regular expression [0123456789] matches any single digit.
       Within a bracket expression, a range expression consists of two characters separated by a hyphen.  It matches any single character that sorts between the two characters, inclusive, using the locale's collating sequence and
       character  set.   For  example,  in  the  default  C  locale,  [a-d] is equivalent to [abcd].  Many locales sort characters in dictionary order, and in these locales [a-d] is typically not equivalent to [abcd]; it might be
       equivalent to [aBbCcDd], for example.  To obtain the traditional interpretation of bracket expressions, you can use the C locale by setting the LC_ALL environment variable to the value C.
       Finally, certain named classes of characters are predefined within bracket expressions, as follows.  Their names are self explanatory,  and  they  are  [:alnum:],  [:alpha:],  [:blank:],  [:cntrl:],  [:digit:],  [:graph:],
       [:lower:],  [:print:],  [:punct:],  [:space:], [:upper:], and [:xdigit:].  For example, [[:alnum:]] means the character class of numbers and letters in the current locale.  In the C locale and ASCII character set encoding,
       this is the same as [0-9A-Za-z].  (Note that the brackets in these class names are part of the symbolic names, and must be included in addition to the brackets delimiting the bracket expression.)  Most meta-characters lose
       their special meaning inside bracket expressions.  To include a literal ] place it first in the list.  Similarly, to include a literal ^ place it anywhere but first.  Finally, to include a literal - place it last.
   Anchoring
       The caret ^ and the dollar sign $ are meta-characters that respectively match the empty string at the beginning and end of a line.
   The Backslash Character and Special Expressions
       The  symbols \< and \> respectively match the empty string at the beginning and end of a word.  The symbol \b matches the empty string at the edge of a word, and \B matches the empty string provided it's not at the edge of
       a word.  The symbol \w is a synonym for [_[:alnum:]] and \W is a synonym for [^_[:alnum:]].
   Repetition
       A regular expression may be followed by one of several repetition operators:
       ?      The preceding item is optional and matched at most once.
       *      The preceding item will be matched zero or more times.
       +      The preceding item will be matched one or more times.
       {n}    The preceding item is matched exactly n times.
       {n,}   The preceding item is matched n or more times.
       {,m}   The preceding item is matched at most m times.  This is a GNU extension.
       {n,m}  The preceding item is matched at least n times, but not more than m times.
   Concatenation
       Two regular expressions may be concatenated; the resulting regular expression matches any string formed by concatenating two substrings that respectively match the concatenated expressions.
   Alternation
       Two regular expressions may be joined by the infix operator |; the resulting regular expression matches any string matching either alternate expression.
   Precedence
       Repetition takes precedence over concatenation, which in turn takes precedence over alternation.  A whole expression may be enclosed in parentheses to override these precedence rules and form a subexpression.
   Back-references and Subexpressions
       The back-reference \n, where n is a single digit, matches the substring previously matched by the nth parenthesized subexpression of the regular expression.
   Basic vs Extended Regular Expressions
       In basic regular expressions the meta-characters ?, +, {, |, (, and ) lose their special meaning; instead use the backslashed versions \?, \+, \{, \|, \(, and \).
EXIT STATUS
       Normally the exit status is 0 if a line is selected, 1 if no lines were selected, and 2 if an error occurred.  However, if the -q or --quiet or --silent is used and a line is selected, the exit status is 0 even if an error
       occurred.
ENVIRONMENT
       The behavior of grep is affected by the following environment variables.
       The  locale  for category LC_foo is specified by examining the three environment variables LC_ALL, LC_foo, LANG, in that order.  The first of these variables that is set specifies the locale.  For example, if LC_ALL is not
       set, but LC_MESSAGES is set to pt_BR, then the Brazilian Portuguese locale is used for the LC_MESSAGES category.  The C locale is used if none of these environment variables are set, if the locale catalog is not installed,
       or if grep was not compiled with national language support (NLS).  The shell command locale -a lists locales that are currently available.
       GREP_OPTIONS
              This variable specifies default options to be placed in front of any explicit options.  As this causes problems when writing portable scripts, this feature will be removed in a future release of grep, and grep warns
              if it is used.  Please use an alias or script instead.
       GREP_COLOR
              This variable specifies the color used to highlight matched (non-empty) text.  It is deprecated in favor of GREP_COLORS, but still supported.  The mt, ms, and mc capabilities of GREP_COLORS have  priority  over  it.
              It  can only specify the color used to highlight the matching non-empty text in any matching line (a selected line when the -v command-line option is omitted, or a context line when -v is specified).  The default is
              01;31, which means a bold red foreground text on the terminal's default background.
       GREP_COLORS
              Specifies the colors and other attributes used to highlight various parts of the output.  Its value is a colon-separated list of capabilities that defaults to  ms=01;31:mc=01;31:sl=:cx=:fn=35:ln=32:bn=32:se=36  with
              the rv and ne boolean capabilities omitted (i.e., false).  Supported capabilities are as follows.
              sl=    SGR  substring for whole selected lines (i.e., matching lines when the -v command-line option is omitted, or non-matching lines when -v is specified).  If however the boolean rv capability and the -v command-
                     line option are both specified, it applies to context matching lines instead.  The default is empty (i.e., the terminal's default color pair).
              cx=    SGR substring for whole context lines (i.e., non-matching lines when the -v command-line option is omitted, or matching lines when -v is specified).  If however the boolean rv capability and the  -v  command-
                     line option are both specified, it applies to selected non-matching lines instead.  The default is empty (i.e., the terminal's default color pair).
              rv     Boolean value that reverses (swaps) the meanings of the sl= and cx= capabilities when the -v command-line option is specified.  The default is false (i.e., the capability is omitted).
              mt=01;31
                     SGR  substring  for  matching  non-empty  text  in  any matching line (i.e., a selected line when the -v command-line option is omitted, or a context line when -v is specified).  Setting this is equivalent to
                     setting both ms= and mc= at once to the same value.  The default is a bold red text foreground over the current line background.
              ms=01;31
                     SGR substring for matching non-empty text in a selected line.  (This is only used when the -v command-line option is omitted.)  The effect of the sl= (or cx= if rv) capability remains active when  this  kicks
                     in.  The default is a bold red text foreground over the current line background.
              mc=01;31
                     SGR  substring for matching non-empty text in a context line.  (This is only used when the -v command-line option is specified.)  The effect of the cx= (or sl= if rv) capability remains active when this kicks
                     in.  The default is a bold red text foreground over the current line background.
              fn=35  SGR substring for file names prefixing any content line.  The default is a magenta text foreground over the terminal's default background.
              ln=32  SGR substring for line numbers prefixing any content line.  The default is a green text foreground over the terminal's default background.
              bn=32  SGR substring for byte offsets prefixing any content line.  The default is a green text foreground over the terminal's default background.
              se=36  SGR substring for separators that are inserted between selected line fields (:), between context line fields, (-), and between groups of adjacent lines when nonzero context is specified (--).  The default  is
                     a cyan text foreground over the terminal's default background.
              ne     Boolean  value  that prevents clearing to the end of line using Erase in Line (EL) to Right (\33[K) each time a colorized item ends.  This is needed on terminals on which EL is not supported.  It is otherwise
                     useful on terminals for which the back_color_erase (bce) boolean terminfo capability does not apply, when the chosen highlight colors do not affect the background, or when EL is too slow or  causes  too  much
                     flicker.  The default is false (i.e., the capability is omitted).
              Note that boolean capabilities have no =... part.  They are omitted (i.e., false) by default and become true when specified.
              See  the  Select Graphic Rendition (SGR) section in the documentation of the text terminal that is used for permitted values and their meaning as character attributes.  These substring values are integers in decimal
              representation and can be concatenated with semicolons.  grep takes care of assembling the result into a complete SGR sequence (\33[...m).  Common values to concatenate include 1 for bold, 4  for  underline,  5  for
              blink,  7  for  inverse,  39  for default foreground color, 30 to 37 for foreground colors, 90 to 97 for 16-color mode foreground colors, 38;5;0 to 38;5;255 for 88-color and 256-color modes foreground colors, 49 for
              default background color, 40 to 47 for background colors, 100 to 107 for 16-color mode background colors, and 48;5;0 to 48;5;255 for 88-color and 256-color modes background colors.
       LC_ALL, LC_COLLATE, LANG
              These variables specify the locale for the LC_COLLATE category, which determines the collating sequence used to interpret range expressions like [a-z].
       LC_ALL, LC_CTYPE, LANG
              These variables specify the locale for the LC_CTYPE category, which determines the type of characters, e.g., which characters are whitespace.  This category also determines the character encoding, that  is,  whether
              text is encoded in UTF-8, ASCII, or some other encoding.  In the C or POSIX locale, all characters are encoded as a single byte and every byte is a valid character.
       LC_ALL, LC_MESSAGES, LANG
              These variables specify the locale for the LC_MESSAGES category, which determines the language that grep uses for messages.  The default C locale uses American English messages.
       POSIXLY_CORRECT
              If  set, grep behaves as POSIX requires; otherwise, grep behaves more like other GNU programs.  POSIX requires that options that follow file names must be treated as file names; by default, such options are permuted
              to the front of the operand list and are treated as options.  Also, POSIX requires that unrecognized options be diagnosed as “illegal”, but since they are not really against the law the default is to  diagnose  them
              as “invalid”.  POSIXLY_CORRECT also disables _N_GNU_nonoption_argv_flags_, described below.
       _N_GNU_nonoption_argv_flags_
              (Here  N  is  grep's numeric process ID.)  If the ith character of this environment variable's value is 1, do not consider the ith operand of grep to be an option, even if it appears to be one.  A shell can put this
              variable in the environment for each command it runs, specifying which operands are the results of file name wildcard expansion and therefore should not be treated as options.  This behavior is available  only  with
              the GNU C library, and only when POSIXLY_CORRECT is not set.
NOTES
       This man page is maintained only fitfully; the full documentation is often more up-to-date.
COPYRIGHT
       Copyright 1998-2000, 2002, 2005-2020 Free Software Foundation, Inc.
       This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
BUGS
   Reporting Bugs
       Email  bug  reports to the bug-reporting address ⟨[email protected]⟩.  An email archive ⟨https://lists.gnu.org/mailman/listinfo/bug-grep⟩ and a bug tracker ⟨https://debbugs.gnu.org/cgi/pkgreport.cgi?package=grep⟩ are avail‐
       able.
   Known Bugs
       Large repetition counts in the {n,m} construct may cause grep to use lots of memory.  In addition, certain other obscure regular expressions require exponential time and space, and may cause grep to run out of memory.
       Back-references are very slow, and may require exponential time.
EXAMPLE
       The following example outputs the location and contents of any line containing “f” and ending in “.c”, within all files in the current directory whose names contain “g” and end in “.h”.  The -n option outputs line numbers,
       the -- argument treats expansions of “*g*.h” starting with “-” as file names not options, and the empty file /dev/null causes file names to be output even if only one file name happens to be of the form “*g*.h”.
         $ grep -n -- 'f.*\.c$' *g*.h /dev/null
         argmatch.h:1:/* definitions and prototypes for argmatch.c
       The only line that matches is line 1 of argmatch.h.  Note that the regular expression syntax used in the pattern differs from the globbing syntax that the shell uses to match file names.
SEE ALSO
   Regular Manual Pages
       awk(1), cmp(1), diff(1), find(1), perl(1), sed(1), sort(1), xargs(1), read(2), pcre(3), pcresyntax(3), pcrepattern(3), terminfo(5), glob(7), regex(7).
   Full Documentation
       A complete manual ⟨https://www.gnu.org/software/grep/manual/⟩ is available.  If the info and grep programs are properly installed at your site, the command
              info grep
       should give you access to the complete manual.
GNU grep 3.4                                                                                                  2019-12-29                                                                                                      GREP(1)

Resources

Wikipedia – grep


How to: Use shortcut keys/Key combinations in Linux Terminal

1 Tab

When entering command, enter beginning of the command, file name or folder name or command option then press “Tab” key, it will complete the rest for you automatically or show all possible results.

2 Ctrl + C

Terminate/Kill the command or process, it will terminate the running process immediately. (signal SIGINT). It can be intercepted by a program, thus the program can clean itself up before exiting or not exit at all.

3 Ctrl + Z

Suspending a process by sending the SIGSTOP signal, it cannot be intercepted by the program.

4 Ctrl + D

Exit the current terminal. If you are using SSH, it will close it. If you are using a terminal directly, it will close the terminal window.

5 Ctrl + L

Clear terminal screen, same effect as “clear” command

6 Ctrl + A

Move the type cursor to the beginning of the line (Same as pressing “Home” key on keyboard)

7 Ctrl + E

Move the type cursor to the end of the line (Same as pressing “End” key on keyboard)

8 Ctrl + U

Wipe the line and move the type cursor to the beginning of the line (Instead of use “Backspace” key to clear the line slowly)

9 Ctrl + K

Wipe the content from the type cursor to the end of the line

10 Ctrl + W

Clear a word

Before Ctrl + W

Before Ctrl + W
Before Ctrl + W

After Ctrl + W

After Ctrl + W
After Ctrl + W

11 Ctrl + Y

It will paste text removed by Ctrl + U, Ctrl + U and Ctrl + K. If you have deleted text by mistake, this will be helpful.

12 Ctrl + P

Review last command, use repetitively to go back further. Many Terminal provides this review function by PageUp key as well. Some provide the review function by using up arrow key as well (↑).

13 Ctrl + N

Similar usage as Ctrl + P but opposite direction, this command navigate to more recent commands. Many Terminal provides this review function by PageDown key as well. Some provide the review function by using down arrow key as well (↓).

14 Ctrl + R

Used for search history commands

Bonus:

Alternatively, we can use “history” command to show all history command

To search from history command, we can use “history | grep searchTerm”


How to: Install Xrdp (Remote Desktop) on CentOS 8 – (How to: Use Windows to remote control CentOS 8)

Xrdp is an open-source implementation of the Microsoft Remote Desktop Protocol (RDP) that allows you to graphically control a remote system

1 If you don’t have GUI/Desktop Environment installed on your CentOS 8, install it by using following command

sudo dnf groupinstall "Server with GUI"

2 Install EPEL repo stands for Extra Packages for Enterprise Linux Repository

sudo dnf -y install epel-release

3 Install xrdp

sudo dnf -y install xrdp

4 (To enable the service on boot and start it now)

sudo systemctl enable xrdp --now

5 (Just start the service without enable it on boot)

sudo systemctl start xrdp

6 Check status of xrdp

sudo systemctl status xrdp

7 Configure Xrdp

Add

exec gnome-session

to the end of the configuration file

/etc/xrdp/xrdp.ini

Or use following command to add “exec gnome-session” to the end of the “/etc/xrdp/xrdp.ini” configuration file

sudo bash -c 'echo "exec gnome-session" >> /etc/xrdp/xrdp.ini'

8 Restart xrdp to load the new configuration

sudo systemctl restart xrdp

9 Configure firewall to allow incoming connection to xrdp

(To allow incoming connection from anywhere to port 3389)

sudo firewall-cmd --add-port=3389/tcp --permanent
sudo firewall-cmd --reload

(To only allow incoming connection from LAN e.g. 10.0.0.0/24 to 3389 port)

sudo firewall-cmd --new-zone=xrdp --permanent
sudo firewall-cmd --zone=xrdp --add-port=3389/tcp --permanent
sudo firewall-cmd --zone=xrdp --add-source=10.0.0.0/24 --permanent
sudo firewall-cmd --reload

Note: More secure way to do is to USE VPN then connect to xrdp server or only accept incoming connection to port 3389 from localhost then setup SSH tunnel to securely forward traffic from your local server on port 3389 to the server on the same port.

Now we use Windows Remote Desktop Connection to connect to the CentOS 8 via xrdp.


Kali Linux – pip/pip3 install -r requirements.txt fail

The Error

When executing following command in Kali Linux

pip install -r requirements.txt
 
OR
 
pip3 install -r requirements.txt

We get error

Command “python setup.py egg_info” failed with error code 1 in ….

Or other errors

We can give following fix a try

The Fix

Use following command instead

pip install --upgrade --force-reinstall -r requirements.txt
 
OR
 
pip3 install --upgrade --force-reinstall -r requirements.txt 

What’s the difference between CentOS Linux vs CentOS Stream

1 If you just want to test CentOS or use it as server: Download CentOS Linux

2 If you want to know what’s difference, read on

In short, CentOS Stream is in-between Fedora and RHEL, it provides clearner vision of what the next version of RHEL will be, give developer time to plan and create next-generation applications, so that they are compatible with future RHEL version. Whereas the CentOS Linux is the normal version we should use if we just want to setup another server.

Resources


Capture The Flag (CTF) – Tools

(Some of the tools are quite old but can still be useful though)

Collection of setup scripts to create an install of various security research tools. Of course, this isn’t a hard problem, but it’s really nice to have them in one place that’s easily deployable to new machines and so forth. The install-scripts for these tools are checked regularly, the results can be found on the build status page.

Installers for the following tools are included:

CategorySourceToolDescription
binaryDirectoryaflState-of-the-art fuzzer.
binaryDirectoryangrNext-generation binary analysis engine from Shellphish.
binaryDirectorybarfBinary Analysis and Reverse-engineering Framework.
binaryDirectorybindeadA static analysis tool for binaries.
binaryLibrarycapstoneMulti-architecture disassembly framework.
binaryDirectorychecksecCheck binary hardening settings.
binaryDirectorycodereasonSemantic Binary Code Analysis Framework.
binaryDirectorycrosstool-ngCross-compilers and cross-architecture tools.
binaryDirectorycross2A set of cross-compilation tools from a Japanese book on C.
binaryDirectoryelfkickersA set of utilities for working with ELF files.
binaryDirectoryelfparserQuickly determine the capabilities of an ELF binary through static analysis.
binaryDirectoryevilizeTool to create MD5 colliding binaries
binaryDirectorygdbUp-to-date gdb with python2 bindings.
binaryDirectorygdb-heapgdb extension for debugging heap issues.
binaryDirectorygefEnhanced environment for gdb.
binaryDirectoryhongfuzzA general-purpose, easy-to-use fuzzer with interesting analysis options.
binaryLibrarykeystoneLightweight multi-architecture assembler framework.
binaryDirectorylibheapgdb python library for examining the glibc heap (ptmalloc)
binaryLibraryliefLibrary to Instrument Executable Formats.
binaryDirectorymiasmReverse engineering framework in Python.
binaryDirectoryone_gadgetMagic gadget search for libc.
binaryDirectorypandaPlatform for Architecture-Neutral Dynamic Analysis.
binaryDirectorypathgrindPath-based, symbolically-assisted fuzzer.
binaryDirectorypedaEnhanced environment for gdb.
binaryDirectorypreenyA collection of helpful preloads (compiled for many architectures!).
binaryDirectorypwndbgEnhanced environment for gdb. Especially for pwning.
binaryDirectorypwntoolsUseful CTF utilities.
binaryDirectorypython-pinPython bindings for pin.
binaryDirectoryqemuLatest version of qemu!
binaryDirectoryqiraParallel, timeless debugger.
binaryDirectoryradare2Some crazy thing crowell likes.
binaryDirectoryrappelA linux-based assembly REPL.
binaryDirectoryropperAnother gadget finder.
binaryDirectoryrp++Another gadget finder.
binaryDirectoryrrRecord and Replay Debugging Framework
binaryDirectoryscratchabitEasily retargetable and hackable interactive disassembler
binaryDirectoryscratchablockYet another crippled decompiler project
binaryDirectoryseccomp-toolsProvides powerful tools for seccomp analysis
binaryDirectoryshellnoobShellcode writing helper.
binaryDirectoryshellsploitShellcode development kit.
binaryDirectorysnowmanCross-architecture decompiler.
binaryDirectorytaintgrindA valgrind taint analysis tool.
binaryLibraryunicornMulti-architecture CPU emulator framework.
binaryDirectoryvalgrindA Dynamic Binary Instrumentation framework with some built-in tools.
binaryDirectoryvillocVisualization of heap operations.
binaryDirectoryvirtualsocketA nice library to interact with binaries.
binaryDirectorywccThe Witchcraft Compiler Collection is a collection of compilation tools to perform binary black magic on the GNU/Linux and other POSIX platforms.
binaryDirectoryxropGadget finder.
binaryDirectorymanticoreManticore is a prototyping tool for dynamic binary analysis, with support for symbolic execution, taint analysis, and binary instrumentation.
forensicsDirectorybinwalkFirmware (and arbitrary file) analysis tool.
forensicsDirectorydislockerTool for reading Bitlocker encrypted partitions.
forensicsDirectoryfirmware-mod-kitTools for firmware packing/unpacking.
forensicsaptforemostFile carver.
forensicsDirectorypdf-parserTool for digging in PDF files
forensicsDirectorypeepdfPowerful Python tool to analyze PDF documents.
forensicsDirectoryscrdecA decoder for encoded Windows Scripts.
forensicsDirectorytestdiskTestdisk and photorec for file recovery.
cryptoDirectorycribdragInteractive crib dragging tool (for crypto).
cryptoDirectoryfastcollAn md5sum collision generator.
cryptoDirectoryforesightA tool for predicting the output of random number generators. To run, launch “foresee”.
cryptoDirectoryfeatherdusterAn automated, modular cryptanalysis tool.
cryptoDirectorygaloisA fast galois field arithmetic library/toolkit.
cryptoDirectoryhashkillHash cracker.
cryptoDirectoryhashpumpA tool for performing hash length extension attaacks.
cryptoDirectoryhashpump-partialhashHashpump, supporting partially-unknown hashes.
cryptoDirectoryhash-identifierSimple hash algorithm identifier.
cryptoDirectorylibc-databaseBuild a database of libc offsets to simplify exploitation.
cryptoDirectorylittleblackboxDatabase of private SSL/SSH keys for embedded devices.
cryptoDirectorymsieveMsieve is a C library implementing a suite of algorithms to factor large integers.
cryptoDirectorynonce-disrespectNonce-Disrespecting Adversaries: Practical Forgery Attacks on GCM in TLS.
cryptoDirectorypemcrackSSL PEM file cracker.
cryptoDirectorypkcrackPkZip encryption cracker.
cryptoDirectorypython-paddingoraclePadding oracle attack automation.
cryptoDirectoryrevengCRC finder.
cryptoDirectoryssh_decoderA tool for decoding ssh traffic. You will need ruby1.8 from https://launchpad.net/~brightbox/+archive/ubuntu/ruby-ng to run this. Run with ssh_decoder --help for help, as running it with no arguments causes it to crash.
cryptoDirectorysslsplitSSL/TLS MITM.
cryptoDirectoryxortoolXOR analysis tool.
cryptoDirectoryyafuAutomated integer factorization.
webDirectoryburpsuiteWeb proxy to do naughty web stuff.
webDirectorycommixCommand injection and exploitation tool.
webDirectorydirbWeb path scanner.
webDirectorydirsearchWeb path scanner.
webDirectorymitmproxyCLI Web proxy and python library.
webDirectorysqlmapSQL injection automation engine.
webDirectorysubbruteA DNS meta-query spider that enumerates DNS records, and subdomains.
stegoaptpngtoolsPNG’s analysis tool.
stegoDirectorysound-visualizerAudio file visualization.
stegoDirectorysteganabaraAnother image stenography solver.
stegoDirectorystegdetectStenography detection/breaking tool.
stegoDockerstego-toolkitA docker image with dozens of steg tools.
stegoDirectorystegsolveImage stenography solver.
stegoDirectorystegosaurusA steganography tool for embedding arbitrary payloads in Python bytecode (pyc or pyo) files.
stegoDirectoryzstegdetect stegano-hidden data in PNG & BMP.
dsniffaptdsniffGrabs passwords and other data from pcaps/network streams.
androidDirectoryapktoolDissect, dis-assemble, and re-pack Android APKs
androidDirectoryandroid-sdkThe android SDK (adb, emulator, etc).
miscDirectoryxspyTiny tool to spy on X sessions.
miscDirectoryz3Theorem prover from Microsoft Research.
miscDirectoryjdguiJava decompiler.
miscDirectoryvelesBinary data analysis and visualization tool.
miscDirectoryyoutube-dlLatest version of the popular youtube downloader.

There are also some installers for non-CTF stuff to break the monotony!

CategoryToolDescription
C magicC-bindA library used to enable function binding in C!
gameDwarf FortressSomething to help you relax after a CTF!
pyvmmonitorpyvmmonitorPyVmMonitor is a profiler with a simple goal: being the best way to profile a Python program.
library collectionsingle_file_libsA large collection of useful single file include libraries written for C/C++
dolphinsudolphinIf your friend ever leaves their laptop unlocked, curl -sSL sh.sudolph.in | sh then wait and see!
tor-browsertor-browserUseful when you need to hit a web challenge from different IPs.

Usage

To use, do:

# set up the path
/path/to/ctf-tools/bin/manage-tools setup
source ~/.bashrc
# list the available tools
manage-tools list
# install gdb, allowing it to try to sudo install dependencies
manage-tools -s install gdb
# install pwntools, but don't let it sudo install dependencies
manage-tools install pwntools
# install qemu, but use "nice" to avoid degrading performance during compilation
manage-tools -n install qemu
# uninstall gdb
manage-tools uninstall gdb
# uninstall all tools
manage-tools uninstall all
# search for a tool
manage-tools search preload

Where possible, the tools keep the installs very self-contained (i.e., in to tool/ directory), and most uninstalls are just calls to git clean (NOTE, this is NOT careful; everything under the tool directory, including whatever you were working on, is blown away during an uninstall). One exception to this are python tools, which are installed using the pip package manager if possible. A ctftools virtualenv is created during the manage-tools setup command and can be accessed using the command workon ctftools.

Help!

Something not working? I didn’t write (almost) any of these tools, but hit up #ctf-tools on freenode if you’re desperate. Maybe some kind soul will help!

Docker (version 1.7+)

By popular demand, a Dockerfile has been included. You can build a docker image with:

git clone https://github.com/zardus/ctf-tools
cd ctf-tools
docker build -t ctf-tools .

And run it with:

docker run -it ctf-tools

The built image will have ctf-tools cloned and ready to go, but you will still need to install the tools themselves (see above).

Alternatively, you can also pull ctf-tools (with some tools preinstalled) from dockerhub:

docker run -it zardus/ctf-tools

Vagrant

You can build a Vagrant VM with:

wget https://raw.githubusercontent.com/zardus/ctf-tools/master/Vagrantfile
vagrant plugin install vagrant-vbguest
vagrant up

And connect to it via:

vagrant ssh

Kali Linux

Kali Linux (Sana and Rolling), due to manually setting certain libraries to not use the latest version available (sometimes being out of date by years) causes some tools to not install at all, or fail in strange ways. AFL and Panda comes to mind, in fact any tool that uses QEMU 2.30 will probably fail during compilation under Kali. Overriding these libraries breaks other tools included in Kali so your only solution is to either live with some of Kali’s tools being broken, or running another distribution separately such as Ubuntu.

Most tools aren’t affected though.

Adding Tools

To add a tool (say, named toolname), do the following:

  1. Create a toolname directory.
  2. Create an install script.
  3. (optional) if special uninstall steps are required, create an uninstall script.

Install Scripts

The install script will be run with $PWD being toolname. It should install the tool into this directory, in as contained a manner as possible. Ideally, full uninstallation should be possible with a git clean.

The install script should create a bin directory and put its executables there. These executables will be automatically linked into the main bin directory for the repo. They could be launched from any directory, so don’t make assumptions about the location of $0!

License

The individual tools are all licensed under their own licenses. As for ctf-tools itself, it is licensed under BSD 2-Clause License. If you find it useful, star it on github (https://github.com/zardus/ctf-tools).

Resource

https://github.com/zardus/ctf-tools

Basics about Network configuration in Linux, IP commands, configuration files etc.

Table of Contents

1 Some useful basic IP commands

1.1 Use network/Interface configuration files to make permanent changes.

For CentOS/RHEL/Fedora etc.

File: /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"
BOOTPROTO=static
ONBOOT=yes TYPE="Ethernet"
IPADDR=10.0.0.10
NAME="System eth0"
HWADDR=00:53:78:2C:7D:9E
GATEWAY=10.0.0.1

For Debian/Ubuntu/Kali Linux etc.

File: /etc/network/interfaces

auto eth0
iface eth0 inet static
address 10.0.0.10
netmask 255.255.255.0
gateway 10.0.0.1

Restart network services to make the changes take effect

sudo /etc/init.d/networking restart
 
OR
 
sudo service restart networking
 
OR
 
systemctl restart networking

1.2 Assign IP address to a specific interface (eth0 in this example) (nonpersistent, will be lost after system reboot)

sudo ip addr add 10.0.0.10 dev eth0

1.3 Remove IP address from a specific interface

sudo ip addr del 10.0.0.10/24 dev eth0

1.4 Check IP address

sudo ip addr
 
OR
 
sudo ip addr show
 
OR
 
sudo ifconfig

1.5 Enable Network interface

sudo ip link set eth0 up

1.6 Disable Network interface

sudo ip link set eth0 down

1.7 Check routing table

sudo ip route show

1.8 Add Static route

sudo ip route del 10.0.0.0/24

1.9 Add persistent static routes

For CentOS/RHEL/Fedora etc.

File: /etc/sysconfig/network-scripts/route-eth0

Add following

10.0.0.0/24 via 192.168.5.20 dev eth0

For Debian/Ubuntu/Kali Linux etc.

File: /etc/network/interfaces

Add following

up ip route add 10.0.0.0/24 via 192.168.5.20 dev eth0

Restart network services to make the changes take effect

sudo /etc/init.d/networking restart
 
OR
 
sudo service restart networking
 
OR
 
systemctl restart networking

1.10 Add default gateway

sudo ip route add default via 10.0.0.1

2 Network configuration file

For CentOS/RHEL/Fedora etc.

File: /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
   #Alias name for the NIC
BOOTPROTO={static|dhcp|none|bootp}
   #Boot protocol, static|none;dhcp
IPADDR=192.168.10.10
   #Set IP address
NETMASK=255.255.255.0
   #Netmask
GATEWAY=192.168.10.1
   #Gateway
ONBOOT=yes|no
   #Activate the network port or not, on boot
HWADDR=00:1E:0B:8F:B0:D0
   #MAC address, if same as the default MAC address of the hardware, this line can be omitted 
DNS1=202.106.0.20
   #Specifiy DNS server
USERCTL=yes|no
   #Users (non-admin/root) allowed to enable/disable this port or not
PEERDNS=yes|no
   #Accept/Reject the DNS server from DHCP while BOOTPROTO is dhcp

For Debian/Ubuntu/Kali Linux etc.

File: /etc/network/interfaces

auto eth1
     #Automatically connect to Ethernet on boot
iface eth1 inet static
     #Assign IP address by static/dhcp
address 192.168.72.8
     #IP address
netmask 255.255.255.0
     #Netmask
gateway 192.168.72.1
     #Default gateway
dns-nameservers 8.8.8.8 4.4.2.2
     #DNS server

3 Hosts configuration

File: /etc/hosts

192.168.0.10 internalserver.mynet

4 Network Interface Controller (NIC) Naming

lo: Localhost loop

ppp#: Point-to-Point Protocol

eth: Ethernet

5 Network management tool

network tool and NetworkManager tool.

network

Restart network

sudo /etc/init.d/network restart

NetworkManager

It can be used to manager network easily, when X Window is not available this GUI tool can be used to manage network without the need to edit configuration file manually.

nmtui
nmtui
nmtui
NetworkManager TUI - nmtui
NetworkManager TUI – mntui

6 NetworkManager cli

NetworkManager provides cli tools as well alongside nmtui

nmcli con show
     #Get UUID table
nmcli dev
     #Check network device status
nmcli r wifi off
     #Turn off wifi
nmcli - commands
nmcli – commands

Start NetworkManager on boot

chkconfig NetworkManager on
 
OR
 
 systemctl enable NetworkManager

Start NetworkManager immediately

service NetworkManager start
 
OR
 
 systemctl start NetworkManager

How to: Find which process is causing high CPU usage Linux/Ubutu/Debian/Kali Linux/CentOS/RHEL

“top” command

top shows CPU usage in real time.

By default, it lists process by their CPU usage, refreshes every 5 seconds.

We can use following command to show top 10 processes with highest CPU usage.

top -b | head -10
[email protected]:~# top -b | head -10
top - 02:05:40 up 20:24,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 165 total,   1 running, 164 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.1 us,  3.1 sy,  0.0 ni, 93.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3913.3 total,   2066.4 free,    617.4 used,   1229.6 buff/cache
MiB Swap:   4094.0 total,   3850.2 free,    243.8 used.   3042.1 avail Mem 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  61649 root      20   0    9144   3560   3164 R   6.7   0.1   0:00.01 top
      1 root      20   0  166420   5908   3936 S   0.0   0.1   0:04.54 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.03 kthreadd
  • -b : Batch mode.
  • head -10: Display first 10 lines in the output.
  • PID : Unique ID of the process.
  • USER : Owner of the process.
  • PR : priority of the process.
  • NI : The NICE value of the process.
  • VIRT : How much virtual memory used by the process.
  • RES : How much physical memory used by the process.
  • SHR : How much shared memory used by the process.
  • S : This indicates the status of the process: S=sleep R=running Z=zombie.
  • %CPU : The percentage of CPU used by the process.
  • %MEM : The percentage of RAM used by the process.
  • TIME+ : How long the process being running.
  • COMMAND : Name of the process.

“ps” command

“ps” stands for “processes status”, it display the information about the active/running processes on the system.

We can use following command to find out high CPU usage processes.

ps -eo pid,ppid,%mem,%cpu,cmd --sort=-%cpu | head
[email protected]:~# ps -eo pid,ppid,%mem,%cpu,cmd --sort=-%cpu | head
    PID    PPID %MEM %CPU CMD
     78       2  0.0  0.4 [kswapd0]
    639     576  4.1  0.4 /usr/lib/xorg/Xorg :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
    262       2  0.0  0.2 [kworker/1:1H-kblockd]
    267       2  0.0  0.1 [kworker/0:1H-kblockd]
    889     823  0.8  0.1 xfwm4 --display :0.0 --sm-client-id 24f144caf-a490-40f0-afc0-fd75665210e1
      1       0  0.1  0.0 /sbin/init splash
      2       0  0.0  0.0 [kthreadd]
      3       2  0.0  0.0 [rcu_gp]
      4       2  0.0  0.0 [rcu_par_gp]

To see the command name instead of full path.

ps -eo pid,ppid,%mem,%cpu,comm --sort=-%cpu | head
[email protected]:~# ps -eo pid,ppid,%mem,%cpu,comm --sort=-%cpu | head
    PID    PPID %MEM %CPU COMMAND
     78       2  0.0  0.4 kswapd0
    639     576  4.1  0.4 Xorg
    262       2  0.0  0.2 kworker/1:1H-kblockd
    267       2  0.0  0.1 kworker/0:1H-kblockd
    889     823  0.8  0.1 xfwm4
      1       0  0.1  0.0 systemd
      2       0  0.0  0.0 kthreadd
      3       2  0.0  0.0 rcu_gp
      4       2  0.0  0.0 rcu_par_gp
  • -e : Select all processes.
  • -o : To customize a output format.
  • –sort=-%cpu : Sort the ouput based on CPU usage.
  • head : To display first 10 lines of the output
  • PID : Unique ID of the process.
  • PPID : Unique ID of the parent process.
  • %MEM : The percentage of RAM used by the process.
  • %CPU : The percentage of CPU used by the process.
  • Command : Name of the process.

htop

htop is a command line utility that allows you to interactively monitor your system’s vital resources or server’s processes in real time

You might need to install htop first

#Debian/Ubuntu/Kali Linux etc.
sudo apt install htop
 
#CentOS/RHEL etc.
sudo yum install htop

To launch htop (Use “q” key to exit)

htop
htop
htop

We can easily sort the processes by their Priority, Nice, Virtual memory usage, Memory, CPU and running time etc. Simply by clicking on the column header.

glances

glances is another utility which can be easily installed and used system resource monitoring tool.

Install

#Debian/Ubuntu/Kali Linux etc.
sudo apt install glances
 
#CentOS/RHEL etc.
sudo yum install glances 

Launch (Use “q” key to exit)

glances

How to: compare Files/Code/Folders/Images/Pictures – List of three Open source Comparing Software/Tools

WinMerge

WinMerge
WinMerge
  • Windows
  • Open source/Free
  • Compare text files/source code files/folders/images
  • Easy to use

WinMerge

Notepad ++

Notepad++ with Compare plugin
Notepad++ with Compare plugin
  • Windows
  • Open source/Free
  • Compare text files/source code files
  • Easy to use

Notepad++

By default Notepad++ doesn’t have compare function.

We can make it possible by easily installing a compare plugin after Notepad++ is installed.

“Plugins -> Plugins Admin…-> Search for “Compare -> Check “Compare” -> Click “Install” button. -> Click on Yes when the pop-up window appears” After it’s done, Notepad++ will restart itself.

Notepad++ -> Plugins Admin
Notepad++ -> Plugins Admin
Notepad++ Install Compare plugin
Notepad++ Install Compare plugin
Click on Yes
Click on Yes
Wait until it's downloaded and installed
Wait until it’s downloaded and installed

Now we can see the Compare plugin is installed. “Plugins -> Compare”

Compare plugin installed
Compare plugin installed

Diffuse

Diffuse
Diffuse
  • Windows/Linux
  • Open source/Free
  • Compare text files/source code files
  • Easy to use

Diffuse

DiffImg

DiffImg
DiffImg
  • Windows/Linux
  • Open source/Free
  • Compare Image files
  • Easy to use

DiffImg

There are many free online text/source code compare and image compare tools as well which doesn’t require any installation, can be handy when it needs to be done quickly, not suitable for huge files or images or any confidential content.


How to: Use “find” command in Linux (Debian, Ubuntu, Kali Linux, CentOS, RHEL/RedHat etc.)

Table of Contents

  • 1 Find command basics
  • 2 Find files and folders by their size
  • 3 Find files by their Owner/Group
  • 4 Find files and folders by date and time
  • 5 Find files by their privileges

1 Find command basics

1.1 Find file from current folder

(“Permission denied” error will appear if the current user doesn’t have permission to access that folder)

Find file named “1.txt” within current working folder

find filename
find 1.txt
find 1.txt
find 1.txt

1.2 Find the file from a folder

Find “1.txt” file under “/tmp” folder

find /path/to/folder -name filename
find /tmp -name 1.txt
find file from a parent folder
find file from a parent folder

1.3 Find file (Ignoring the case/Case insensitive)

Find “1.txt” within “/tmp” folder, ignoring case.

find /path/to/folder -iname filename
find /tmp -iname 1.txt
Find file (Ignoring the case/Case insencitive)
Find file (Ignoring the case/Case insencitive)

1.4 Find folder/directory only

find /path/to/folder -type d -name foldername
find /tmp -type d -name abc
Find folder/directory
Find folder/directory only
Find folder/directory, ignoring the case
Find folder/directory only, ignoring the case

1.5 Find file only

find /path/to/folder -type f -name 1
find /tmp -type f -name 1.txt
Find file only
Find file only

1.6 Find specific file

Find .txt files from “/tmp” folder

find -type f -name "*.extension"
find -type f -name "*.txt"
Find specific file
Find specific file

2 Find files and folders by their size

2.1 Find all 100MB files

find / -size 100M

2.2 Find files which is between 10MB and 100MB

find / -size +50M -size -100M

2.3 Find and delete files which is greater than 500MB in size

find / -size +500M -exec rm -rf {} \;

2.4 Find specific files and delete them

Find all mp3 files which are greater than 5MB, delete them

find / -type f -name *.mp3 -size +5M -exec rm {} \;

3 Find files by their Owner/Group

3.1 Find files by their owner

Find all of the “1.txt” file which belongs to Bob

find / -user Bob -name 1.txt

3.2 Find all files belongs to the user

Find all files belongs to Bob under “/test” folder

find /test -user Bob

3.3 Find files based on their group

Find all files under “/test” which belongs to the group “testgroup”

find /test -group testgroup

3.4 Find all “.txt” files under “/test” which belongs to Bob

find /test -user Bob -iname "*.txt"

4 Find files and folders by date and time

4.1 Find files

4.1 Find files which their data was last modified n*24 hours ago. (3 days in this example)

find / -mtime 3

4.2 Find files being accessed in 3 days

find / -atime 3

4.3 Find files that being modified between last 3-10 days

find / -mtime +3 -time -10

4.4 Find files which their status was last changed 5 minutes ago

find / -cmin -5

4.5 Find files which their data was last modified 5 minutes ago

find / -mmin -5

4.6 Find files which are being access within 5 minutes

find / -amin -5

5 Find files by their privileges

5.1 Find files with “777” permission

find -type f -perm 0777 -print

5.2 Find files without “777” permission

find / -type f ! -perm 777

5.3 Find files with “644” permission (SGID)

find / -perm 2644

5.4 Find files with “551” Sticky Bit permission

find / -perm 1551

5.5 Find SUID files

find / -perm /u=s

5.6 Find read only permission files

find / -perm u=r

5.7 Find executable files

find / -perm /a=x

5.8 Find all files with “777” permission and chmod to 644

find /test -type f -perm 0777 -print -exec chmod 644 {} \;

5.9 Find all folders with “777” permission and chmod to 755

find / -type d -perm 777 -print -exec chmod 755 {} \;

5.10 Find and delete single file (1.txt in this case)

find -type f -name "1.txt" -exec rm -f {} \;

5.11 Find and delete multiple files with same extension (txt files in this case)

find -type f -name "*.txt" -exec rm -f {} \;

5.12 Find all empty files

find / -type f -empty

5.13 Find all empty folders

find / -type d -empty

5.14 Find all hidden files under “/tmp” folder

find /tmp -type f -name ".*"