Tips for Let’s Encrypt (certbot etc.)

Ubuntu 18.04 LTS (bionic), Install, Configure “certbot”

https://certbot.eff.org/lets-encrypt/ubuntubionic-apache

Other systems, use following website to find out installation and configuration process

https://certbot.eff.org/

Let Certbot edit your Apache configuration automatically to serve it, turning on HTTPS access in a single step.

sudo certbot --apache
sudo certbot certonly --apache 

Just get a certificate, make the changes to Apache configuration manually

sudo certbot certonly --apache -d contoso.com

With sub-domains

sudo certbot certonly --apache -d contoso.com -d www.contoso.com -d ftp.contoso.com

With multiple domains

sudo certbot certonly --apache -d contoso.com -d www.contoso.com -d ftp.contoso.com -d anotherfakedomain.com -d fakedomain2.com

Test with –dry-run (“–dry-run” switch can be used to Test “renew” or “certonly” without saving any certificates to disk)

sudo certbot certonly --apache -d contoso.com -d www.contoso.com -d ftp.contoso.com -d anotherfakedomain.com -d fakedomain2.com --dry-run

With multiple domains, multiple virtual hosts in different document folders

sudo certbot certonly --apache --webroot -w /var/htdocs/contoso.com/ -d contoso.com -d www.contoso.com -d ftp.contoso.com -w /var/htdocs/anotherfakedomain.com/ -d anotherfakedomain.com -w /var/htdocs/fakedomain2.com/ -d fakedomain2.com --dry-run

Test with Staging server/Environment (higher Rate Limits) (without –dry-run)

Warning: Certificates from Staging server should not be used for production

sudo certbot certonly --server https://acme-staging-v02.api.letsencrypt.org/directory -i apache --webroot -w /var/htdocs/contoso.com/ -d contoso.com -d www.contoso.com

Test with Staging server and with –dry-run

sudo certbot certonly --server https://acme-staging-v02.api.letsencrypt.org/directory -i apache --webroot -w /var/htdocs/contoso.com/ -d contoso.com -d www.contoso.com --dry-run

Test with real server with –dry-run

sudo certbot certonly --server https://acme-v02.api.letsencrypt.org/directory -i apache --webroot -w /var/htdocs/contoso.com/ -d contoso.com -d www.contoso.com --dry-run

Download certificate from real server

sudo certbot certonly --server https://acme-v02.api.letsencrypt.org/directory -i apache --webroot -w /var/htdocs/contoso.com/ -d contoso.com -d www.contoso.com

Use dns as preferred challenge with wild card domain

sudo certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --preferred-challenges dns -d 'contoso.com' -d '*.contoso.com'

For Ubuntu, all certificates, certificate configuration files, renal configuration files, archive, keys etc. are stored in following folder

/etc/letsencrypt

Note

If the certbot complaining about connection issue, it might be that the connection is being blocked by firewall, system firewall or Web Application Firewall (WAF) etc.

Extended Reading

  • Root Certificate for Staging Server/Environment ( https://acme-staging-v02.api.letsencrypt.org/directory )
    • The staging environment intermediate certificate (“Fake LE Intermediate X1”) is issued by a root certificate not present in browser/client trust stores. If you wish to modify a test-only client to trust the staging environment for testing purposes you can do so by adding the “Fake LE Root X1” certificate to your testing trust store. Important: Do not add the staging root or intermediate to a trust store that you use for ordinary browsing or other activities, since they are not audited or held to the same standards as our production roots, and so are not safe to use for anything other than testing.

Linux, Ubuntu etc. How to find where the program is installed

Keywords: Linux, Unix, Ubuntu, Kali Linux, Debian, where the program is installed, where the package is installed, program location, software location, package location, package path, program path, software path, installation path, installation location

whereis command

whereis packagname
 
#e.g.
whereis apache2
whereis apache2
whereis apache2
Usage:
 whereis [options] [-BMS … -f] 
Locate the binary, source, and manual-page files for a command.
Options:
-b         search only for binaries
-B   define binaries lookup path
-m         search only for manuals and infos
-M   define man and info lookup path
-s         search only for sources
-S   define sources lookup path
-f         terminate  argument list
-u         search for unusual entries
-l         output effective lookup paths
-h, --help     display this help
-V, --version  display version

which command

which filename
 
#e.g.
which apache2
which apache2 mysql
which apache2, which apache2 mysql
which apache2, which apache2 mysql
Usage:
 which which [filename1] [filename2] …
which command in Linux is a command which is used to locate the executable file associated with the given command by searching it in the path environment variable. It has 3 return status as follows:
0 : If all specified commands are found and executable.
1 : If one or more specified commands is nonexistent or not executable.
2 : If an invalid option is specified. 

find command

find / -name packagename
 
#e.g.
find / -name apache2
find / -name apache2
find / -name apache2

How to Troubleshoot/Investigate Ubuntu slow boot/startup (systemd)

Keywords: Ubuntu check slow boot, Ubuntu check slow startup, Ubuntu troubleshoot slow boot, Ubuntu check slow startup, systemd, systemd-analyze blame, Kali Linux, boot time, slow boot time, slow startup time


Sometimes, Ubuntu startup takes very long time, we want to find out the cause.

(Following command is not limited to Ubuntu, it can actually be used on any Linux distributions with systemd installed)

When the systems boots up completely, we can use following command to check which process took longest time during boot time.

$ sudo systemd-analyze blame
systemd-analyze blame output from Kali Linux
systemd-analyze blame output from Kali Linux

We can also use head to see top 10 slowest ones

$ sudo systemd-analyze blame | head
sudo systemd-analyze blame | head output from Kali Linux
sudo systemd-analyze blame | head output from Kali Linux

How to: Enable/Disable Apache2 modules and configuration files on Ubuntu (a2enconf, a2disconf, a2enmod, a2dismod)

(If the module is not installed yet use apt to install first or compile from source. Then follow the guide)

Sometimes before enabling apache2 modules, we might need to enable configuration file for the module first, use following command

This can be used to switch php version as well

1 Enable configuration file

#e.g. enable php7.4-fpm configuration file for apache2
sudo a2enconf php7.4-fpm

To disable configuration file for apache2 we can use

#e.g. disable php7.4-fpm configuration file for apache2
sudo a2disconf php7.4-fpm

2 Enable apache2 module

#e.g. enable php7.4-fpm module for apache2
sudo a2enmod php7.4-fpm

To disable module for apache2 we can use

#e.g. disable php7.4-fpm module for apache2
sudo a2dismod php7.4-fpm

Following by a reload or restart for apache2, we should be good to go.

3 Reload or Restart apache2 to make the changes take effect (Ubuntu 15.04+ or above)

#e.g. To reload apache2
sudo systemctl reload apache2
OR
sudo systemctl reload apache2.service
#e.g. To restart apache2
sudo systemctl restart apache2
OR
sudo systemctl reload apache2.service

(For Ubuntu 14.10 or older without systemd use following command to reload/restart apache2)

#e.g. To reload apache2
sudo service apache2 reload
OR
sudo /etc/init.d/apache2 reload
#e.g. To restart apache2
sudo service apache2 restart
OR
sudo /etc/init.d/apache2 restart

Extended reading

a2enconf, a2disconf

a2enconf is a script that enables the specified configuration file within the apache2 configuration. It does this by creating symlinks within /etc/apache2/conf-enabled. Likewise, a2disconf disables a specific configuration part by removing those symlinks. It is not an error to enable a configuration which is already enabled, or to disable one which is already disabled. Note that many configuration file may have a dependency to specific modules. Unlike module dependencies, these are not resolved automatically. Configuration fragments stored in the conf-available directory are considered non-essential or being installed and manged by reverse dependencies (e.g. web scripts). — Ubuntu Manual

a2enmod, a2dismod

a2enmod is a script that enables the specified module within the apache2 configuration. It does this by creating symlinks within /etc/apache2/mods-enabled. Likewise, a2dismod disables a module by removing those symlinks. It is not an error to enable a module which is already enabled, or to disable one which is already disabled. Note that many modules have, in addition to a .load file, an associated .conf file. Enabling the module puts the configuration directives in the .conf file as directives into the main server context of apache2. — Ubuntu Manual

systemctl

systemctl may be used to introspect and control the state of the “systemd” system and service manager. Please refer to systemd(1) for an introduction into the basic concepts and functionality this tool manages. — Ubuntu Manual

How to reset Kali Linux forgotten root password – Reset Kali Linux password with single-user mode

(Single-user mode will not ask for username and password for login, the user has superuser rights)

1 Boot into GRUB menu

1.1 Power on the Kali Linux

1.2 When following boot screen appears, press key to stop the “Booting in x seconds” counter.

Kali Linux 2019.4 - Boot screen
Kali Linux 2019.4 – Boot screen

2 Edit GRUB menu

2.1 Make sure the first option “Kali GNU/Linux” is highlighted.

2.2 Press “e” key to enter GRUB menu edit mode

Kali Linux 2019.4 - GRUB menu edit mode
Kali Linux 2019.4 – GRUB menu edit mode

2.3 Find line start with “linux”

Find line start with "linux"
Find line start with “linux”

2.4 Use arrow keys to navigate, change “ro” to “rw”, append “init=/bin/bash” at the end of the line

Use arrow keys to navigate, change "ro" to "rw", append "init=/bin/bash" at the end of the line
Use arrow keys to navigate, change “ro” to “rw”, append “init=/bin/bash” at the end of the line
Modified line
Modified line

2.5 Use Ctrlx or F10 to boot with modified line, follow screen will appear

Single-user mode
Single-user mode

3 Change password

3.1 Enter following command

# passwd

3.2 Enter new password then retype the new password

Change password
Change password

Now restart the system with physical button (Do not use reboot command this time)

Now you can login with the new password.

Bonus:

With most Linux distributions we can use following routine to reset the forgotten password

1 Enter GRUB menu with editing mode

2 Add single-user boot option to GRUB menu

3 Boot with modified GRUB menu

4 Change password with “passwd” command

5 Reboot with physical button

Configure Squid proxy with php redirector

Squid proxy can be used with different redirectors and rewriters, in this guide we will be using a redirector written with php.

This guide is using Ubuntu 18.04 LTS, your configuration file maybe located differently if using different version or Linux distribution etc.

Note: Before starting following this guide, you have to make sure your squid proxy is configured properly and running without any issue.

Squid configuration

1 Open squid configuration file from

/etc/squid/squid.conf

or

/etc/squid3/squid.conf

For following steps, make sure you are using right path/configuration file (either /squid/ or /squid3/), I will be using /squid/

2 Add following line

url_rewrite_program /usr/bin/php /etc/squid/redirect.php

This indicates that we will be using an redirector/ url write program written in php so that we will need php binary to run the script thus “/usr/bin/php” the redorector file path is “/etc/squid/redirect.php” which means will be putting the “redirect.php” file under “/etc/squid”

PHP url rewrite program/redirector

1 We create a “redirect.php” file under “/etc/squid/”

2 We write necessary script

#!/usr/bin/php
<?php
//By https://dannyda.com
//Modified based on
//http://wiki.squid-cache.org/ConfigExamples/PhpRedirectors
//Using PHP for Redirects
 
//and
 
//https://aacable.wordpress.com/tag/squid-url-redirection/
//------------------------------------------------------------------------------
$temp = array();
// Extend stream timeout to 24 hours
stream_set_timeout(STDIN, 86400);
while ( $input = fgets(STDIN) ) {
  // Split the output (space delimited) from squid into an array.
  $temp = explode(' ', $input);
  // Set the URL from squid to a temporary holder.
  $output = $temp[0] . "\n";
	if(preg_match("/(.*[0-9])\/ABCD\/.*/i", $input)){
		//Replace any url matching xxx.xxx.xxx.xxx/ABCD/aaaaaaaaaaaaaaaaaaaaaa and change to to my.blocked.com/ABCD/aaaaaaaaaaaaaaaaaaaaaa
		$output = "308:".str_replace(parse_url($input,PHP_URL_HOST),'my.blocked.com',strtok($input, ' '))."\n"; //Outout intended modified url (308 Permanent Redirect)
		//file_put_contents("/etc/squid/a.txt", $input,FILE_APPEND); //For debugging we can enable this line, output will be found at /etc/squid/a.txt
		//$output = "302:"."http://www.google.com/"."\n"; //(302 Found, Moved Temporarily)
		
		//We can either output modified url based on ip address, partial url matching, domain matching and modify partialy as well or we can redirect to another url completely
		//Following by more examples
	}
	
	if(preg_match("/(.*[0-9])\/forum\/.*\/sign\=.*\.jpg/i", $input)){
		$output = "308:".str_replace(parse_url($input,PHP_URL_HOST),'mytestforum.com',strtok($input, ' '))."\n";
		// /forum/*/sign
	}
	
	if(preg_match("/(.*[0-9])\/forum\/pic\/item\/.*\.jpg/i", $input)){
		$output = "308:".str_replace(parse_url($input,PHP_URL_HOST),'mytestforum.com',strtok($input, ' '))."\n";
		// /forum/pic/item
	}
	//---------------------------------------------------------------------------
	//Temporarily block URLs based on partial url match with regular expression
	if(preg_match("/ad\.m\.domain\.com(\/.*|$)/i", $input)){
		$output = "308:"."http://255.255.255.255"."\n";
		//adash.m.taobao.com
	}
	if(preg_match("/.*admaster\.com.*/i", $input)){
		$output = "308:"."http://255.255.255.255"."\n";
		//*.admaster.com.cn
	}
	
	if(preg_match("/.*\/AdvertiseInterface\//i", $input)){
		$output = "308:"."http://255.255.255.255"."\n";
		//ad
		//http://albatrosscn.buddylync.com:8080/AdvertiseInterface/
	}
	
	if(preg_match("/.*\/advertise\//i", $input)){
		$output = "308:"."http://255.255.255.255"."\n";
		//ad
		//http://*/advertise/
	}
	
  echo $output;
}

3 We add execute permission

chmod +x /etc/squid/redirect.php

4 We restart squid

~# systemctl restart squid

5 Now we should be able to see the redirector in action

Extended reading

Bonus

Online regular expression testers

Which can help you craft desired regular expression for use within redirector

Ubuntu/Linux Check SSD trim status

Keywords: Linux, Ubuntu, SSD, trim, Status, Solid State Drive, fstrim, fstrim.timer, systemd, systemctl, fstrim, fstrim.timer

1 Launch your terminal

2 Enter following commands to check status

Check trim Timer/Schedule status

# systemctl status fstrim.timer
Ubuntu -  systemctl status fstrim.timer
Ubuntu – systemctl status fstrim.timer

Check Trim status

# systemctl status fstrim
Ubuntu -  systemctl status fstrim
Ubuntu – systemctl status fstrim

How to: Fix Ubuntu apt upgrade -> 404 Not Found

Keywords: Ubuntu apt upgrade Failed to fetch, Ubuntu apt upgrade 404 Not Found, Ubuntu change update source, Ubuntu change update mirror, Ubuntu automatically select mirror, source.list, /etc/apt/source.list

“Failed to fetch http:// 404 Not Found [IP: ]”

Ubuntu apt upgrade - Failed to fetch ... 404 Not Found ...
Ubuntu apt upgrade – Failed to fetch … 404 Not Found …

This can be the issue with the repository/source syncing.

In other words we can try to change repository/source to fix this issue.

1 Login to the Ubuntu

2 Use your favourite text editor to open the file with root permission:

File:

/etc/apt/source.list

Make a copy for backup

$ sudo cp /etc/apt/source.list /etc/apt/source.list_backup

Use your favourite text editor to open it. (Here we using “nano”)

$ sudo nano /etc/apt/source.list

(For nano text editor, when finished editing, Use Ctrl + X to end the editing then press Y to save/overwrite the file)

3 We need to change all lines starting with “deb” and “deb-src” without “#” (Only those including xxxx.ubuntu.com etc. etc.)

3.1 What to change:

Lines similar to following should not touch:

# newer versions of the distribution.
# deb-src http://us.archive.ubuntu.com/ubuntu bionic-security universe
# deb http://us.archive.ubuntu.com/ubuntu bionic main restricted
deb http://download.somerep.com/download/repository sarge contrib

Lines similar to following should be changed:

deb-src http://us.archive.ubuntu.com/ubuntu bionic-security universe
deb http://us.archive.ubuntu.com/ubuntu bionic main restricted

3.2 How to change (Change to what)

There are two ways we can change one is manually specify the server, another is to let the system select mirror automatically.

Manually specifying:

# Using main server/mirror in USA
deb http://us.archive.ubuntu.com/ubuntu bionic main restricted
deb-src http://us.archive.ubuntu.com/ubuntu bionic-security universe
# Using main server/mirror in Singapore
deb http://sg.archive.ubuntu.com/ubuntu bionic main restricted
deb-src http://sg.archive.ubuntu.com/ubuntu bionic-security universe
# Using main server/mirror in China
deb http://cn.archive.ubuntu.com/ubuntu bionic main restricted
deb-src http://cn.archive.ubuntu.com/ubuntu bionic-security universe

Now, the “apt update” and “apt upgrade” etc. will use specified source.

Note and Warning: You have to follow above steps to make the change to entire “source.list” file, not just the two from above example. (Leave the text after “/ubuntu” unchanged, different versions of Ubuntu will have different text, DO NOT CHANGE them!) (For this example they are “bionic main restricted” and “bionic-security”, your system may have different ones)

Let the system select mirror automatically:

# The system will automatically select the best server
deb mirror://mirrors.ubuntu.com/mirrors.txt bionic main restricted
deb-src mirror://mirrors.ubuntu.com/mirrors.txt bionic-security universe

Now, the “apt update” and “apt upgrade” etc. will use best server automatically.

Note and Warning: You have to follow above steps to make the change to entire “source.list” file, not just the two from above example. (Leave the text after “/ubuntu” unchanged, different versions of Ubuntu will have different text, DO NOT CHANGE them!) (For this example they are “bionic main restricted” and “bionic-security”, your system may have different ones)