Useful tools for Python

Python Tutor

Free online Python code visualization, Python learning tool.

Python Tutor
Python Tutor

http://pythontutor.com/

Anaconda

Easy package management

Packed with many useful Python tools

Anaconda
Anaconda
Anaconda - Software
Anaconda – Software

https://www.anaconda.com/distribution/#download-section

Jupyter Notebook

Jupyter notebook is like a magic notebook for Python. It can be used to share Notes, algebra, data analytics, code etc easily.

Jupyter Notebook
Jupyter Notebook

https://jupyter.org/install

(We can use Anaconda to install it easily)

IPython

IPython is a interactive shell for Python.

Supports Automatic indenting, bash shell commands, many built-in functions etc.

https://ipython.org/

Skulpt

Skulpt is a online Python environment built via javascript. Use with CodeMirror, we can do basic Python programming.

Skulpt
Skulpt

http://skulpt.org/


How to: Optimize MySQL, MariaDB with Simple Tools

1 mysqltuner.pl

mysqltuner.pl
mysqltuner.pl

Supports MySQL, MariaDB, Percona Server etc. with over 300

Tuning MySQL performance, checks configuration, including log file settings, storage engine, security. Outline potential issues/fix.

1.1 Download

cd /tmp
 
wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/mysqltuner.pl
 
chmod +x mysqltuner.pl

1.2 Usage

 ./mysqltuner.pl --socket /var/lib/mysql/mysql.sock 

1.3 Output

Items with [!!] are important e.g. Maximum possible memory usage: 10G (300% of installed RAM)

Last section with “Recommendations” tells us where we can look into, which Variables we should adjust and suggested values etc.

mysqltuner.pl: https://github.com/major/MySQLTuner-perl

2 tuning-primer.sh

Similar to mysqltuner.pl.

Currently it handles recomendations for the following:

  • Slow Query Log
  • Max Connections
  • Worker Threads
  • Key Buffer [MyISAM only]
  • Query Cache
  • Sort Buffer
  • Joins
  • Temp Tables
  • Table (Open & Definition) Cache
  • Table Locking
  • Table Scans (read_buffer) [MyISAM only]
  • InnoDB Status

2.1 Download

cd /tmp
 
wget https://launchpad.net/mysql-tuning-primer/trunk/1.6-r1/+download/tuning-primer.sh
 
chmod +x tuning-primer.sh

2.2 Usage

./tuning-primer.sh

tuning-primer.sh: https://github.com/BMDan/tuning-primer.sh

3 pt-variable-advisor

Analyses MySQL variables, output suggestions based on those variables.

3.1 Download

https://www.percona.com/downloads/percona-toolkit/LATEST/

3.2 Usage

pt-variable-advisor localhost --socket /var/lib/mysql/mysql.sock

4 pt-qurey-digest

Analyses log, process list, tcpdump for MySQL queries. Mainly used to analyze slow queries. pt-qurey-digest outputs more details compare to py-query_digest.

4.1 Download

Sames as “3 pt-variable-advisor”

4.2 Usage

pt-query-digest /var/lib/mysql/slowtest-slow.log

4.3 Other usages

# Analyze slow quires
pt-query-digest /var/lib/mysql/slowtest-slow.log > slow_report.log
 
# Quires within 24 hours
pt-query-digest --since=24h /var/lib/mysql/slowtest-slow.log > slow_report.log
 
# Quires within specified time frame
pt-query-digest /var/lib/mysql/slowtest-slow.log --since '2020-01-01 00:00:00' --until '2012-01-10 00:00:00'> > slow_report.log
 
# Slow quires with select
pt-query-digest --filter '$event->{fingerprint} =~ m/^select/i' /var/lib/mysql/slowtest-slow.log> slow_report.log
 
# Query from specific user
pt-query-digest --filter '($event->{user} || "") =~ m/^root/i' /var/lib/mysql/slowtest-slow.log> slow_report.log
 
# All full table scanning, full join slow quires
pt-query-digest --filter '(($event->{Full_scan} || "") eq "yes") ||(($event->{Full_join} || "") eq "yes")' /var/lib/mysql/slowtest-slow.log> slow_report.log

List of 8 useful free software/program for macOS

1 VLC

Free/open source music/video player, can play almost all of audio/video formats

VLC
VLC

Download

2 IINA

Another free/open source music/video player which can play almost all of audio/video formats

IINA
IINA

Download

3 Cheat Sheet

Display shortcut keys, great for learning shortcut keys, eventually you won’t need it once you have them in your mind

Cheat Sheet
Cheat Sheet

Download

4 Spectacle

Move or resize windows easily with keyboard like in Microsoft Windows

Spectacle keys
Spectacle keys
Spectacle - Whole
Spectacle – Whole
Spectacle - Left
Spectacle – Left
Spectacle - Right
Spectacle – Right
Spectacle - Right
Spectacle – Right
Spectacle - Center
Spectacle – Center

Download

5 Aerial

a Mac screensaver based on the new Apple TV screensaver that displays the Aerial movies Apple shot over New York, San Francisco, Hawaii, China, etc. Starting with version 1.6, this also includes the new undersea videos available in tvOS 13!

Aerial screensaver
Aerial screensaver

More information & Download

6 Snip

Take screenshot easily, it can also scroll the web-page and take screenshot of it, so that we can get a full web-page screenshot

Snip
Snip

Downlaod

7 Itsycal

A tiny menu bar calendar.

Itsycal
Itsycal
Itsycal
Itsycal

Download

8 Amphetamine

Simply keep the mac awake…

Amphetamine can keep your Mac, and optionally its display(s), awake through a super simple on/off switch, or automatically through easy-to-configure Triggers. Amphetamine is extremely powerful and includes advanced features for those who need them, yet remains intuitive and easy-to-use for those who don’t need all of the bells and whistles.

(You can also disable sleep function from the systems settings without installing any Apps to achieve the same results)

Amphetamine
Amphetamine

Download

(Can be downloaded directly from App store)


How to: Install VMware Tools in Linux

Install from ISO

1 Launch VMware Workstation

2 Start the Virtual Machine

3 Right click on the Virtual Machine Name on the tab

4 Click on “Install VMware Tools”

Parameter for “./vmware-install.pl”

-d: Use default/suggested answers for questions

-f: Force install

# Debian/Ubuntu etc.
#Mount to /mnt
sudo mount /dev/sr0 /mnt
#Extract, install
tar -zxvf /mnt/VMwareTools-versionNumber.tar.gz -C ~
cd vmware-tools-distrib/
sudo ./vmware-install.pl -f -d

When done, eject the ISO, reboot the operating system

sudo unmout /mnt
eject
sudo reboot

# CentOS/Fedora/RHEL etc.
#Mount to /mnt
mount /dev/sr0 /mnt
#Extract, install
tar -zxvf /mnt/VMwareTools-versoinNumber.tar.gz -C ~
cd vmware-tools-distrib/
./vmware-install.pl -f -d

Install VMware Tools from Online source/Package Manager

# Debian/Ubuntu etc.
#System without desktop environment
sudo apt install open-vm-tools
 
#System with desktop environment
sudo apt install open-vm-tools-desktop
# CentOS/Fedora/RHEL etc.
#System without desktop environment
dnf -y install open-vm-tools
or
yum -y install open-vm-tools
 
#System with desktop environment
dnf -y install open-vm-tools-desktop
or
yum -y install open-vm-tools-desktop

Powerful Linux Interactive shell

fish (friendly interactive shell) is a smart and user-friendly command line shell for Linux, macOS, and the rest of the family.

Autosuggestions

Autosuggestion Thumbnail

fish suggests commands as you type based on history and completions, just like a web browser. Watch out, Netscape Navigator 4.0!

Glorious VGA Color

Colors Thumbnail

fish supports 24 bit true color, the state of the art in terminal technology. Behold the monospaced rainbow.

Sane Scripting

Scripting Thumbnail

fish is fully scriptable, and its syntax is simple, clean, and consistent. You’ll never write esac again.

Web Based configuration

Web Config Thumbnail

For those lucky few with a graphical computer, you can set your colors and view functions, variables, and history all from a web page.

Man Page Completions

Man Page Completions Thumbnail

Other shells support programmable completions, but only fish generates them automatically by parsing your installed man pages.

Works Out Of The Box

Works Out of the Box Thumbnail

fish will delight you with features like tab completions and syntax highlighting that just work, with nothing new to learn or configure.

fish can be installed easily on most Linux distros with their default package manager.

Linux

# Debian/Ubuntu/Kali Linux etc.
sudo apt install fish
 
# RHEL/CentOS/Fedora
sudo dns install fish
or, for older version
sudo yum install fish
 
# Archlinux
pacman -S fish
 
# gentoo Linux
emerge fish
 
# void-Linux
xbps-install fish-shell
 
# NixOS
nix-env -i fish
 
# Guix
guix package -i fish
 
# Solus
eopkg install fish
 
# Hombrew
brew install fish

BSD

# FreeBSD
pkg install fish
 
# OpenBSD
pkg_add fish

Windows

# Cygwin
fish is available in setup, in the Shells category.
 
# Windows Subsystem for Linux
sudo apt install fish
or
depend on the Linux distro you've chose, refer to the above "Linux" part to find correct command to use
 
# MSYS2
pacman -S fish

masOS

# Homebrew
brew install fish
 
# MacPorts
sudo port install fish
 
# Installer
https://github.com/fish-shell/fish-shell/releases/download/3.1.0/fish-3.1.0.pkg
 
10.6+: Installs to /usr/local/

Bonus

  • To use, type fish in the terminal then hit Enter key

To check fish version

echo $FISH_VERSION

HTML version help document

help

To switch default shell to fish

sudo chsh -s /usr/bin/fish

To switch back to default bash shell

sudo chsh -s /bin/bash

(If your default shell is zsh)

sudo chsh -s /usr/zsh

Simple/Quick List of Free Code Editors (Include free, open source)

(There are many commercial editors with trial period, they are not included in this list, only free or open source editors are listed)

  1. Aptana Studio (Windows, Linux, macOS)
  2. Atom.io (Windows, Linux, macOS)
  3. Crimson Editor (Windows)
  4. jEdit (Windows, Linux, macOS)
  5. Notepad++ (Windows)
  6. Programmer’s Notepad (Windows)
  7. PSPad (Windows)
  8. SCREEM (Linux) (HTML/Web)
  9. Visual Studio Code (Windows, Linux, macOS)

How to: Put panorama photo together – Panorama photo stitcher

Sometimes we want to put panorama photos together, so that we can have a complete photo.

It can be time consuming if we do it manually, also it requires skill to be done.

To make it easier, we can use software to achieve same or even better results.

Hugin is an open source and completely free software just does that and it’s easy to use as well.

Hugin
Hugin

Some technical details can be found here: Photometric alignment and vignetting correction

Download


Nikto – Web server scanner

Note: Nikto is included in latest Kali Linux (2020.1)

Nikto is a web server assessment tool. It is designed to find various default and insecure files, configurations and programs on any type of web server.

It can be used to discover potential issues and security vulnerabilities from web servers including:

  • Server and software misconfigurations
  • Default files and programs
  • Insecure files and programs
  • Outdated servers and programs [1]

Some basic usages/Quick start

Scan the IP/Host on TCP port 80

nikto -h 10.0.0.1
 
nikto -h contoso.com

Scan the IP/Host on specified port (443 in this case)

nikto -h 10.0.0.1 -p 443
 
nikto -h https://10.0.0.1:443/

Multiple Ports

nikto -h 10.0.0.1 -p 40,443,3128

Using a proxy

# Using the proxy server specified from configuration file
nikto -h 10.0.0.1 -p 80 -useproxy
 
# Specifying proxy server on the fly
nikto -h 10.0.0.1 -useproxy http://127.0.0.1:3128/

Help

$ nikto -H
   Options:
       -ask+               Whether to ask about submitting updates
                               yes   Ask about each (default)
                               no    Don't ask, don't send
                               auto  Don't ask, just send
       -Cgidirs+           Scan these CGI dirs: "none", "all", or values like "/cgi/ /cgi-a/"
       -config+            Use this config file
       -Display+           Turn on/off display outputs:
                               1     Show redirects
                               2     Show cookies received
                               3     Show all 200/OK responses
                               4     Show URLs which require authentication
                               D     Debug output
                               E     Display all HTTP errors
                               P     Print progress to STDOUT
                               S     Scrub output of IPs and hostnames
                               V     Verbose output
       -dbcheck           Check database and other key files for syntax errors
       -evasion+          Encoding technique:
                               1     Random URI encoding (non-UTF8)
                               2     Directory self-reference (/./)
                               3     Premature URL ending
                               4     Prepend long random string
                               5     Fake parameter
                               6     TAB as request spacer
                               7     Change the case of the URL
                               8     Use Windows directory separator (\)
                               A     Use a carriage return (0x0d) as a request spacer
                               B     Use binary value 0x0b as a request spacer
        -Format+           Save file (-o) format:
                               csv   Comma-separated-value
                               json  JSON Format
                               htm   HTML Format
                               nbe   Nessus NBE format
                               sql   Generic SQL (see docs for schema)
                               txt   Plain text
                               xml   XML Format
                               (if not specified the format will be taken from the file extension passed to -output)
       -Help              Extended help information
       -host+             Target host/URL
       -404code           Ignore these HTTP codes as negative responses (always). Format is "302,301".
       -404string         Ignore this string in response body content as negative response (always). Can be a regular expression.
       -id+               Host authentication to use, format is id:pass or id:pass:realm
       -key+              Client certificate key file
       -list-plugins      List all available plugins, perform no testing
       -maxtime+          Maximum testing time per host (e.g., 1h, 60m, 3600s)
       -mutate+           Guess additional file names:
                               1     Test all files with all root directories
                               2     Guess for password file names
                               3     Enumerate user names via Apache (/~user type requests)
                               4     Enumerate user names via cgiwrap (/cgi-bin/cgiwrap/~user type requests)
                               5     Attempt to brute force sub-domain names, assume that the host name is the parent domain
                               6     Attempt to guess directory names from the supplied dictionary file
       -mutate-options    Provide information for mutates
       -nointeractive     Disables interactive features
       -nolookup          Disables DNS lookups
       -nossl             Disables the use of SSL
       -no404             Disables nikto attempting to guess a 404 page
       -Option            Over-ride an option in nikto.conf, can be issued multiple times
       -output+           Write output to this file ('.' for auto-name)
       -Pause+            Pause between tests (seconds, integer or float)
       -Plugins+          List of plugins to run (default: ALL)
       -port+             Port to use (default 80)
       -RSAcert+          Client certificate file
       -root+             Prepend root value to all requests, format is /directory
       -Save              Save positive responses to this directory ('.' for auto-name)
       -ssl               Force ssl mode on port
       -Tuning+           Scan tuning:
                               1     Interesting File / Seen in logs
                               2     Misconfiguration / Default File
                               3     Information Disclosure
                               4     Injection (XSS/Script/HTML)
                               5     Remote File Retrieval - Inside Web Root
                               6     Denial of Service
                               7     Remote File Retrieval - Server Wide
                               8     Command Execution / Remote Shell
                               9     SQL Injection
                               0     File Upload
                               a     Authentication Bypass
                               b     Software Identification
                               c     Remote Source Inclusion
                               d     WebService
                               e     Administrative Console
                               x     Reverse Tuning Options (i.e., include all except specified)
       -timeout+          Timeout for requests (default 10 seconds)
       -Userdbs           Load only user databases, not the standard databases
                               all   Disable standard dbs and load only user dbs
                               tests Disable only db_tests and load udb_tests
       -useragent         Over-rides the default useragent
       -until             Run until the specified time or duration
       -update            Update databases and plugins from CIRT.net
       -url+              Target host/URL (alias of -host)
       -useproxy          Use the proxy defined in nikto.conf, or argument http://server:port
       -Version           Print plugin and database versions
       -vhost+            Virtual host (for Host header)
                + requires a value

Resources

[1] Nikto v2.1.5 – The Manual
[2] Github