Useful tools for Python

Python Tutor

Free online Python code visualization, Python learning tool.

Python Tutor
Python Tutor

http://pythontutor.com/

Anaconda

Easy package management

Packed with many useful Python tools

Anaconda
Anaconda
Anaconda - Software
Anaconda – Software

https://www.anaconda.com/distribution/#download-section

Jupyter Notebook

Jupyter notebook is like a magic notebook for Python. It can be used to share Notes, algebra, data analytics, code etc easily.

Jupyter Notebook
Jupyter Notebook

https://jupyter.org/install

(We can use Anaconda to install it easily)

IPython

IPython is a interactive shell for Python.

Supports Automatic indenting, bash shell commands, many built-in functions etc.

https://ipython.org/

Skulpt

Skulpt is a online Python environment built via javascript. Use with CodeMirror, we can do basic Python programming.

Skulpt
Skulpt

http://skulpt.org/


Simple ways to improve/optimize Python performance

1 Use built-in function when available

Use built-in function when available
Use built-in function when available

2 Use List comprehensions to Filter a list

Loop
Loop
Filter
Filter
List comprehensions
List comprehensions

3 Use “try” accordingly

# Slow
class Fruit(object):
    name='Apple'
    
fruit=Fruit()
    if hasattr(fruit,'name'):
        fruit.name
 
 
# Quicker
class Fruit(object):
    name='Apple'
    
fruit=Fruit()
    try:
        fruit.name
    except AttributeError:
        pass

4 Use “in” to check list rather than if (Use built-in function)

in vs for loop and if
in vs for loop and if

5 Deduplication

For loop and if vs set
For loop and if vs set

6 Sort

sorted
sorted
.sort()
.sort()

7 Put iterative loop into function

Iterative loop vs iterative loop in function
Iterative loop vs iterative loop in function

8 Check boolean

if var:
 
# Is faster than
 
if var is True:
 
# Is faster than
 
if var==True:

9 Check is the list is empty

if not list:
 
# Is faster than 
 
if list==[]:
 
# Is faster than
 
if len(list)==0:

10 Object vs list

list() and dict() are slower than [] directly returning a list.


How to: Install Anaconda/Anaconda3 in Linux/Debian/Ubuntu/Kali Linux

(For how to launch Anaconda Navigator or create Anaconda Navigator shortcut: How to: Create shortcut icon for Anaconda/Anaconda3 Navigator/Launch Anaconda Navigator in Linux/Debian/Ubuntu/Kali Linux)

1 Download Anaconda3 from official website: https://www.anaconda.com/distribution/

https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh

2 Make it executable

2.1 We can right click on the file, then click on “Properties” -> “Permissions” -> Check “Allow this file to run as a program”

Assign execute permission
Assign execute permission

2.2 We can also use “chomd” command to assign execute permission

Related to chmod command: Unix/Linux (Ubuntu, Debian, Kali Linux etc.) Privilege Management

3 Launch the “Anaconda3-2019.10-Linux-x86_64.sh” to install Anaconda

Anaconda3-2019.10-Linux-x86_64.sh
Anaconda3-2019.10-Linux-x86_64.sh

3.1 Accept the license terms and install to user home directory

Accept the license terms and install to user home directory
Accept the license terms and install to user home directory

3.2 Let the installer to initialize Anaconda3 by running conda init by typing “yes” then hit Enter key

Type Yes, then hit Enter key
Type Yes, then hit Enter key

4 Now it’s installed if there is no error

(Installation on Windows and macOS is very simple, just download the correct binary file (.exe) form official website, execute, follow the steps to install, the shortcut will be create automatically on desktop and in start menu, all installation packages can be found here: https://www.anaconda.com/distribution/)


How to: Convert string/text to ASCII in python & How to: Convert ASCII to string/text in python

Method 1

#String we want to convert to ASCII
my_string = "AAAasdasfd, a[sdfi       94q,p asf "
 
#Convert to ASCII
results_in_ASCII = list(my_string.encode())
 
#Convert back to string
results_in_string = bytes(results_in_ASCII).decode()
 
#Print ASCII
print(results_in_ASCII)
 
print()
 
#Print String
print(results_in_string)
Method 1
Method 1

Method 2

#String we want to convert to ASCII
my_string = "Hello, world!"
 
#Convert to ASCII
results_in_ASCII = [ord(x) for x in my_string]
 
#Convert back to string
results_in_string = ''.join(chr(i) for i in results_in_ASCII)
 
#Print ASCII
print(results_in_ASCII)
 
print()
 
#Print String
print(results_in_string)
Method 2
Method 2

Resource

How do I convert a list of ascii values to a string in python?


PHP MySQL Prepared Statements, data binding Example/Sample (With “CURRENT_TIMESTAMP” etc.)

(If you are not sure what is “->” in php or what is ‘ssi’ in bind_param Scroll to “Some basics” section first, then start from top of the page)

Database connection configuration

(Before starting, we need to create database connection string)

<?php
$servername = "localhost";
$username = "database_username";
$password = "database_password";
$dbname = "database_name";
// Create connection
$db_connect = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($db_connect->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}
//echo "Connected successfully";
?>

Example 1 – Insert with prepared statement, data binding

<?php
//Prepare statement
$stmt = $db_connect->prepare("INSERT INTO Table_Name VALUES (?, ?, CURRENT_TIMESTAMP, ?)");
//Bind data
$stmt->bind_param('ssi', $First_Name, $Last_Name, $Age);
//Populate Data
$First_Name = 'Fred';
$Last_Name = 'Smith';
$Age = 50;
//Execute, 
if(!$stmt->execute()) {
	die('Error: ' . mysqli_error($db_connect));
}
$db_connect->close();
?>

Example 2 – Select with prepared statement, data binding

<?php
$stmt = $db_connect->prepare("SELECT Product FROM Table_Name WHERE ID = ?");
$stmt->bind_param('s',$ID);
$ID = 'ABC123';
$sql->execute();
$result = $stmt->get_result();
if($result->num_rows > 0) {
	while($row = $result->fetch_assoc()) {
		echo $row['Product'];
	}
}
mysqli_free_result($result);
$db_connect->close();
?>

Extra:

if($result->num_rows > 0) {
	while($row = $result->fetch_assoc()) {
		echo $row['Product'];
	}
}

Can be replaced by:

if($result->num_rows > 0) {
	while($row = mysqli_fetch_array($result)) {
		echo $row['Product'];
	}
}

What is fetch_assoc()? [5]

What is mysqli_fetch_array()? [6]

Example 3 – Update with prepared statement, data binding

<?php
$stmt = $db_connect->prepare("UPDATE Table_Name SET First_Name=?, Last_Name=?, Time=CURRENT_TIMESTAMP, Num_Books=? WHERE ID=?");
$stmt->bind_param('ssis', $First_Name, $Last_Name, $Num_Books, $ID);
$First_Name = 'Fred';
$Last_Name = 'Smith';
$Num_Books = 10;
$ID = 'ABC123';
if(!$stmt->execute()) {
	die('Error: ' . mysqli_error($db_connect));
}
$db_connect->close();
?>

Some basics

Object operator “->” [2,3,4]

e.g. “$db_connect->close();”

[2]
 
// Create a new instance of MyObject into $obj
$obj = new MyObject();
// Set a property in the $obj object called thisProperty
$obj->thisProperty = 'Fred';
// Call a method of the $obj object named getProperty
$obj->getProperty();
-> is like
. in Javascript
. in JAVA
. in RUBY
. in CloudFusion
. in Python

Type specification chars “i,d,s,b”

($stmt->bind_param(‘ssi’, $First_Name, $Last_Name, $Age);)

(To learn what is ‘ssi’ in Bind data part, see this “Type specification chars” table)

Type specification chars [1]

CharacterDescription
icorresponding variable has type integer
dcorresponding variable has type double
scorresponding variable has type string
bcorresponding variable is a blob and will be sent in packets

Resources

[1] php official document: mysqli_stmt::bind_param
[2] w3schools: PHP OOP – Classes and Objects
[4] php official document: Objects
[5] php official document: mysqli_result::fetch_assoc
[6] w3schools: PHP mysqli fetch_array() Function


Open source Online Document Management System for Developers – ShowDoc

ShowDoc is a tool greatly applicable for an IT team to share documents online. It can promote communication efficiency among members of the team.

What can it be used for?

  • API Document ( Demo

With the development of mobile Internet, BaaS (Backend as a Service) becomes more and more popular. The Server end provides API, and the APP end or Webpage frontend can invoke data conveniently. Using ShowDoc can compile exquisite API documents in a very fast and convenient way.

  • Data Dictionary ( Demo

A good Data Dictionary can easily exhibit database structure to other people, such as definition of each field and the like.

  • Explanation Document ( Demo

You can absolutely use ShowDoc to compile the explanation documents for some tools, as well as to compile some technical specifications explanation documents for the team to look up.

What functions does it have?

  • Sharing and Exporting

Responsive webpage design can share the project documents to computer or mobile devices for reading. It can also export the project into word document for browsing offline.

  • Permission Management
  • Public Project and Private ProjectProjects on ShowDoc are divided into two categories including Public Project and Private Project. Public Project can be visited by any user no matter he/she logs in or not, while inputting password for verification is needed for visiting the Private Project. The password is set by project creator.
  • Project Transfer

The project creator can transfer the project to other users of the website freely.

  • Project Members

You can easily add or delete project members in the project of ShowDoc. Members of the project can edit the project, but they can not transfer or delete the project (only creator of the project has the permission).

  • Edit Function
    • Markdown Edit
    ShowDoc adopts Markdown Editor, and it is excellent both in editing and reading experience. If you know nothing about Markdown, please search “Learning and Introduction of Markdown” on the search engine.
    • Template Insert
    On the editing page of ShowDoc, a click on the button which is on the top of the Editor can easily insert API interface template and data dictionary template. After inserting the template, altering data is the only thing that need to do and it reduces a lot of work in editing.
    • History Version
    ShowDoc provides a function of History Version on the page, and you can easily restore the page to the former version.

Resource

https://github.com/star7th/showdoc

Configure Squid proxy with php redirector

Squid proxy can be used with different redirectors and rewriters, in this guide we will be using a redirector written with php.

This guide is using Ubuntu 18.04 LTS, your configuration file maybe located differently if using different version or Linux distribution etc.

Note: Before starting following this guide, you have to make sure your squid proxy is configured properly and running without any issue.

Squid configuration

1 Open squid configuration file from

/etc/squid/squid.conf

or

/etc/squid3/squid.conf

For following steps, make sure you are using right path/configuration file (either /squid/ or /squid3/), I will be using /squid/

2 Add following line

url_rewrite_program /usr/bin/php /etc/squid/redirect.php

This indicates that we will be using an redirector/ url write program written in php so that we will need php binary to run the script thus “/usr/bin/php” the redorector file path is “/etc/squid/redirect.php” which means will be putting the “redirect.php” file under “/etc/squid”

PHP url rewrite program/redirector

1 We create a “redirect.php” file under “/etc/squid/”

2 We write necessary script

#!/usr/bin/php
<?php
//By https://dannyda.com
//Modified based on
//http://wiki.squid-cache.org/ConfigExamples/PhpRedirectors
//Using PHP for Redirects
 
//and
 
//https://aacable.wordpress.com/tag/squid-url-redirection/
//------------------------------------------------------------------------------
$temp = array();
// Extend stream timeout to 24 hours
stream_set_timeout(STDIN, 86400);
while ( $input = fgets(STDIN) ) {
  // Split the output (space delimited) from squid into an array.
  $temp = explode(' ', $input);
  // Set the URL from squid to a temporary holder.
  $output = $temp[0] . "\n";
	if(preg_match("/(.*[0-9])\/ABCD\/.*/i", $input)){
		//Replace any url matching xxx.xxx.xxx.xxx/ABCD/aaaaaaaaaaaaaaaaaaaaaa and change to to my.blocked.com/ABCD/aaaaaaaaaaaaaaaaaaaaaa
		$output = "308:".str_replace(parse_url($input,PHP_URL_HOST),'my.blocked.com',strtok($input, ' '))."\n"; //Outout intended modified url (308 Permanent Redirect)
		//file_put_contents("/etc/squid/a.txt", $input,FILE_APPEND); //For debugging we can enable this line, output will be found at /etc/squid/a.txt
		//$output = "302:"."http://www.google.com/"."\n"; //(302 Found, Moved Temporarily)
		
		//We can either output modified url based on ip address, partial url matching, domain matching and modify partialy as well or we can redirect to another url completely
		//Following by more examples
	}
	
	if(preg_match("/(.*[0-9])\/forum\/.*\/sign\=.*\.jpg/i", $input)){
		$output = "308:".str_replace(parse_url($input,PHP_URL_HOST),'mytestforum.com',strtok($input, ' '))."\n";
		// /forum/*/sign
	}
	
	if(preg_match("/(.*[0-9])\/forum\/pic\/item\/.*\.jpg/i", $input)){
		$output = "308:".str_replace(parse_url($input,PHP_URL_HOST),'mytestforum.com',strtok($input, ' '))."\n";
		// /forum/pic/item
	}
	//---------------------------------------------------------------------------
	//Temporarily block URLs based on partial url match with regular expression
	if(preg_match("/ad\.m\.domain\.com(\/.*|$)/i", $input)){
		$output = "308:"."http://255.255.255.255"."\n";
		//adash.m.taobao.com
	}
	if(preg_match("/.*admaster\.com.*/i", $input)){
		$output = "308:"."http://255.255.255.255"."\n";
		//*.admaster.com.cn
	}
	
	if(preg_match("/.*\/AdvertiseInterface\//i", $input)){
		$output = "308:"."http://255.255.255.255"."\n";
		//ad
		//http://albatrosscn.buddylync.com:8080/AdvertiseInterface/
	}
	
	if(preg_match("/.*\/advertise\//i", $input)){
		$output = "308:"."http://255.255.255.255"."\n";
		//ad
		//http://*/advertise/
	}
	
  echo $output;
}

3 We add execute permission

chmod +x /etc/squid/redirect.php

4 We restart squid

~# systemctl restart squid

5 Now we should be able to see the redirector in action

Extended reading

Bonus

Online regular expression testers

Which can help you craft desired regular expression for use within redirector

How to: upgrade/update Anaconda

Keywords: Python, anaconda, conda

Upgrade Anaconda & all packages to latest version:

1 Find and launch Anaconda Prompt

Anaconda from Windows start menu
Anaconda from Windows start menu

2 Using following command to update all packages including Anaconda itself

conda update --all
Anaconda Prompt -> conda update --all
Anaconda Prompt -> conda update –all
Anaconda Prompt -> conda update --all -> finished
Anaconda Prompt -> conda update –all -> finished

(Note: This way, it will drop all the version constraints from the history and tries to make everything as new as it can)

Bonus:

We can also use “conda update anaconda=VersionNumber” to update Anaconda to specified version

e.g.

conda update anaconda=1.9.7