1
0
mirror of https://github.com/moparisthebest/pi-hole synced 2024-11-13 12:55:05 -05:00

Merge branch 'master' into lighttpd

This commit is contained in:
Jacob Salmela 2015-12-01 15:09:51 -06:00
commit 6c4f4f49bd
7 changed files with 584 additions and 209 deletions

View File

@ -1,3 +1,16 @@
# Automated Install
1. Install Raspbian
2. Run the command below
### ```curl -L install.pi-hole.net | bash```
![Pi-hole automated installation](http://i.imgur.com/Un7lBlj.png)
Once installed, **configure any device to use the Raspberry Pi as its DNS server and the ads will be blocked**. You can also configure your router's DHCP options to assign the Pi as clients DNS server so they do not need to do it manually.
[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif "AdminLTE Presentation")](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=3J2L3Z4DHW9UY "Donate")
# Raspberry Pi Ad Blocker
**A black hole for ads, hence Pi-hole**
@ -6,22 +19,19 @@
The Pi-hole is a DNS/Web server that will **block ads for any device on your network**.
## Coverage
### Security Now! Podcast
Pi-hole is mentioned at 100 minutes and 26 seconds (the link brings you right there)
[![Pi-hole on Security Now!](http://img.youtube.com/vi/p7-osq_y8i8/0.jpg)](http://www.youtube.com/watch?v=p7-osq_y8i8&t=100m26s)
### Tech Blogs
Featured on [MakeUseOf](http://www.makeuseof.com/tag/adblock-everywhere-raspberry-pi-hole-way/) and [Lifehacker](http://lifehacker.com/turn-a-raspberry-pi-into-an-ad-blocker-with-a-single-co-1686093533)!
## Automated Install
1. Install Raspbian
2. Set a **static** IP address
3. Run the command below
```curl -s "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/automated%20install/basic-install.sh" | bash```
Once installed, **configure any device to use the Raspberry Pi as its DNS server and the ads will be blocked**. You can also configure your router's DHCP options to assign the Pi as clients DNS server so they do not need to do it manually.
## Technical Details
A more detailed explanation of the installation can be found [here](http://jacobsalmela.com/block-millions-ads-network-wide-with-a-raspberry-pi-hole-2-0).
[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif "AdminLTE Presentation")](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=3J2L3Z4DHW9UY "Donate")
## Gravity
The [gravity.sh](https://github.com/jacobsalmela/pi-hole/blob/master/gravity.sh) does most of the magic. The script pulls in ad domains from many sources and compiles them into a single list of [over 1.6 million entries](http://jacobsalmela.com/block-millions-ads-network-wide-with-a-raspberry-pi-hole-2-0).
@ -29,7 +39,11 @@ The [gravity.sh](https://github.com/jacobsalmela/pi-hole/blob/master/gravity.sh)
You can add a `whitelist.txt` or `blacklist.txt` in `/etc/pihole/` and the script will apply those files automatically.
## Web Interface
I am also working on a [Web interface](https://github.com/jacobsalmela/AdminLTE#pi-hole-admin-dashboard) so you can view stats and change settings.
The [Web interface](https://github.com/jacobsalmela/AdminLTE#pi-hole-admin-dashboard) will be installed automatically so you can view stats and change settings. You can find it at:
`http://192.168.1.x/admin/index.php`
![Web](http://i.imgur.com/m114SCn.png)
## Custom Config File
If you want to use your own variables for the gravity script (i.e. storing the files in a different location) and don't want to have to change them every time there is an update to the script, create a file called `/etc/pihole/pihole.conf`. In it, you should add your own variables in a similar fashion as shown below:
@ -47,4 +61,8 @@ A technical and detailed description can be found [here](http://jacobsalmela.com
## Other Operating Systems
This script will work for other UNIX-like systems with some slight **modifications**. As long as you can install `dnsmasq` and a Webserver, it should work OK. The automated install only works for a clean install of Raspiban right now since that is how the project originated.
### Examples Of The Pi-hole On Other Operating Systems
- [Sky-Hole](http://dlaa.me/blog/post/skyhole)
- [Pi-hole in the Cloud!](http://blog.codybunch.com/2015/07/28/Pi-Hole-in-the-cloud/)
[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif "AdminLTE Presentation")](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=3J2L3Z4DHW9UY "Donate")

View File

@ -0,0 +1,3 @@
#!/bin/bash
# Flushes /var/log/pihole.log
truncate -s 0 /var/log/pihole.log

View File

@ -1,13 +1,49 @@
#!/bin/bash
whitelist=/etc/pihole/whitelist.txt
adList=/etc/pihole/gravity.list
if [[ ! -f $whitelist ]];then
touch $whitelist
fi
if [[ $# = 0 ]]; then
echo "Immediately whitelists one or more domains."
echo "Usage: whitelist.sh domain1 [domain2 ...]"
fi
combopattern=""
# For each argument passed to this script
for var in "$@"
do
echo "Whitelisting $var..."
# Use sed to search for the domain in /etc/pihole/gravity.list and remove it using an in-place edit
sed -i "/$var/d" /etc/pihole/gravity.list
# Also add the domain to the whitelist.txt in /etc/pihole
echo "$var" >> /etc/pihole/whitelist.txt
echo "Whitelisting $var..."
# Construct basic pattern to match domain name.
basicpattern=$(echo $var | awk -F '[# \t]' 'NF>0&&$1!="" {print ""$1""}' | sed 's/\./\\./g')
if [[ "$basicpattern" != "" ]]; then
# Add to the combination pattern that will be used below
if [[ "$combopattern" != "" ]]; then combopattern="$combopattern|"; fi
combopattern="$combopattern$basicpattern"
# Also add the domain to the whitelist but only if it's not already present
grep -E -q "^$basicpattern$" $whitelist \
|| echo "$var" >> $whitelist
fi
done
echo "** $# domain(s) whitelisted."
# Force dnsmasq to reload /etc/pihole/gravity.list
kill -HUP $(pidof dnsmasq)
# Now report on and remove matched domains
if [[ "$combopattern" != "" ]]; then
echo "Modifying hosts file..."
# Construct pattern to match entry in hosts file.
# This consists of one or more IP addresses followed by the domain name.
pattern=$(echo $combopattern | awk -F '[# \t]' '{printf "%s", "^(([0-9]+\.){3}[0-9]+ +)+("$1")$"}')
# Output what will be removed and then actually remove
sed -r -n 's/'"$pattern"'/ Removed: \3/p' $adList
sed -r -i '/'"$pattern"'/d' $adList
echo "** $# domain(s) whitelisted."
# Force dnsmasq to reload /etc/pihole/gravity.list
kill -HUP $(pidof dnsmasq)
fi

26
advanced/pihole.cron Normal file
View File

@ -0,0 +1,26 @@
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
# Pi-hole: Update the ad sources once a week on Sunday at 01:59
# Download any updates from the ad lists
59 1 * * 7 root /usr/local/bin/gravity.sh
# Pi-hole: Parse the log file before it is flushed and save the stats to a database
# This will be used for a historical view of your Pi-hole's performance
#50 11 * * * root /usr/local/bin/dailyLog.sh
# Pi-hole: Flush the log daily at 11:58 so it doesn't get out of control
# Stats will be viewable in the Web interface thanks to the cron job above
58 11 * * * root /usr/local/bin/piholeLogFlush.sh

View File

@ -1,31 +1,47 @@
#!/bin/bash
# Pi-hole automated install
# Raspberry Pi Ad-blocker
#!/usr/bin/env bash
# Pi-hole: A black hole for Internet advertisements
# by Jacob Salmela
# Network-wide ad blocking via your Raspberry Pi
#
# Install with this command (from the Pi):
# pi-hole.net/donate
#
# curl -s "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/automated%20install/basic-install.sh" | bash
# Install with this command (from your Pi):
#
# Or run the commands below in order
# curl -L install.pi-hole.net | bash
clear
echo " _____ _ _ _ "
echo " | __ (_) | | | | "
echo " | |__) | __ | |__ ___ | | ___ "
echo " | ___/ | |__| | '_ \ / _ \| |/ _ \ "
echo " | | | | | | | | (_) | | __/ "
echo " |_| |_| |_| |_|\___/|_|\___| "
echo " "
echo " Raspberry Pi Ad-blocker "
echo " "
echo "Set a static IP before running this!"
echo " "
echo " Press Enter when ready "
echo " "
read
######## VARIABLES #########
tmpLog=/tmp/pihole-install.log
instalLogLoc=/etc/pihole/install.log
# Get the screen size in case we need a full-screen message and so we can display a dialog that is sized nicely
screenSize=$(stty -a | tr \; \\012 | egrep 'rows|columns' | cut '-d ' -f3)
# Find the rows and columns
rows=$(stty -a | tr \; \\012 | egrep 'rows' | cut -d' ' -f3)
columns=$(stty -a | tr \; \\012 | egrep 'columns' | cut -d' ' -f3)
# Divide by two so the dialogs take up half of the screen, which looks nice.
r=$(( rows / 2 ))
c=$(( columns / 2 ))
# Find IP used to route to outside world
IPv4info=$(ip route get 8.8.8.8)
IPv4dev=$(echo $IPv4info| awk '{print $5}')
IPv4addr=$(ip -o -f inet addr show dev $IPv4dev | awk '{print $4}')
IPv4gw=$(echo $IPv4info | awk '{print $3}')
# IPv6 support to be added later
#IPv6eui64=$(ip addr show | awk '/scope\ global/ && /ff:fe/ {print $2}' | cut -d'/' -f1)
#IPv6linkLocal=$(ip addr show | awk '/inet/ && /scope\ link/ && /fe80/ {print $2}' | cut -d'/' -f1)
availableInterfaces=$(ip link show | awk -F' ' '/[0-9]: [a-z]/ {print $2}' | grep -v "lo" | cut -d':' -f1)
dhcpcdFile=/etc/dhcpcd.conf
####### FUCNTIONS ##########
backupLegacyPihole()
{
if [[ -f /etc/dnsmasq.d/adList.conf ]];then
echo "Original Pi-hole detected. Initiating sub space transport..."
echo "Original Pi-hole detected. Initiating sub space transport"
sudo mkdir -p /etc/pihole/original/
sudo mv /etc/dnsmasq.d/adList.conf /etc/pihole/original/adList.conf.$(date "+%Y-%m-%d")
sudo mv /etc/dnsmasq.conf /etc/pihole/original/dnsmasq.conf.$(date "+%Y-%m-%d")
@ -36,47 +52,226 @@ if [[ -f /etc/dnsmasq.d/adList.conf ]];then
else
:
fi
}
echo "Updating the Pi..."
welcomeDialogs()
{
# Display the welcome dialog
whiptail --msgbox --backtitle "Welcome" --title "Pi-hole automated installer" "This installer will transform your Raspberry Pi into a network-wide ad blocker!" $r $c
# Explain the need for a static address
whiptail --msgbox --backtitle "Initating network interface" --title "Static IP Needed" "The Pi-hole is a SERVER so it needs a STATIC IP ADDRESS to function properly.
In the next section, you can choose to use your current network settings (DHCP) or to manually edit them." $r $c
}
chooseInterface()
{
# Turn the available interfaces into an array so it can be used with a whiptail dialog
interfacesArray=()
while read -r line
do
interfacesArray+=("$line" "available" "ON")
done <<< "$availableInterfaces"
# Find out how many interfaces are available to choose from
interfaceCount=$(echo "$availableInterfaces" | wc -l)
chooseInterfaceCmd=(whiptail --separate-output --radiolist "Choose An Interface" $r $c $interfaceCount)
chooseInterfaceOptions=$("${chooseInterfaceCmd[@]}" "${interfacesArray[@]}" 2>&1 >/dev/tty)
for desiredInterface in $chooseInterfaceOptions
do
piholeInterface=$desiredInterface
echo "Using interface: $piholeInterface"
done
}
use4andor6()
{
# Let use select IPv4 and/or IPv6
cmd=(whiptail --separate-output --checklist "Select Protocols" $r $c 2)
options=(IPv4 "Block ads over IPv4" on
IPv6 "Block ads over IPv4" off)
choices=$("${cmd[@]}" "${options[@]}" 2>&1 >/dev/tty)
clear
for choice in $choices
do
case $choice in
IPv4)
echo "IPv4 selected."
useIPv4=true
;;
IPv6)
echo "IPv6 selected."
useIPv6=true
;;
esac
done
}
useIPv6dialog()
{
whiptail --msgbox --backtitle "Coming soon..." --title "IPv6 not yet supported" "I need your help for IPv6. Consider donating at: http://pi-hole.net/donate" $r $c
}
getStaticIPv4Settings()
{
# Ask if the user wannts to use DHCP settings as their static IP
if (whiptail --backtitle "Calibrating network interface" --title "Static IP Address" --yesno "Do you want to use your current network settings as a static address?
IP address: $IPv4addr
Gateway: $IPv4gw" $r $c) then
# If they choose yes, let the user know that the IP address will not be available via DHCP and may cause a conflict.
whiptail --msgbox --backtitle "IP information" --title "FYI: IP Conflict" "It is possible your router could still try to assign this IP to a device, which would cause a conflict. But in most cases the router is smart enough to not do that.
If you are worried, either manually set the address, or modify the DHCP reservation pool so it does not include the IP you want.
It is also possible to use a DHCP reservation, but if you are going to do that, you might as well set a static address." $r $c
# Nothing else to do since the variables are already set above
else
# Otherwise, we need to ask the user to input their desired settings.
# Start by getting the IPv4 address (pre-filling it with info gathered from DHCP)
# Start a loop to let the user enter their information with the chance to go back and edit it if necessary
until [[ $ipSettingsCorrect = True ]]
do
# Ask for the IPv4 address
IPv4addr=$(whiptail --backtitle "Calibrating network interface" --title "IPv4 address" --inputbox "Enter your desired IPv4 address" $r $c $IPv4addr 3>&1 1>&2 2>&3)
if [[ $? = 0 ]];then
echo "Your static IPv4 address: $IPv4addr"
# Ask for the gateway
IPv4gw=$(whiptail --backtitle "Calibrating network interface" --title "IPv4 gateway (router)" --inputbox "Enter your desired IPv4 default gateway" $r $c $IPv4gw 3>&1 1>&2 2>&3)
if [[ $? = 0 ]];then
echo "Your static IPv4 gateway: $IPv4gw"
# Give the user a chance to review their settings before moving on
if (whiptail --backtitle "Calibrating network interface" --title "Static IP Address" --yesno "Are these settings correct?
IP address: $IPv4addr
Gateway: $IPv4gw" $r $c)then
# If the settings are correct, then we need to set the piholeIP
# Saving it to a temporary file us to retrieve it later when we run the gravity.sh script
echo ${IPv4addr%/*} > /tmp/piholeIP
# After that's done, the loop ends and we move on
ipSettingsCorrect=True
else
# If the settings are wrong, the loop continues
ipSettingsCorrect=False
fi
else
# Cancelling gateway settings window
ipSettingsCorrect=False
echo "User canceled."
exit
fi
else
# Cancelling IPv4 settings window
ipSettingsCorrect=False
echo "User canceled."
exit
fi
done
# End the if statement for DHCP vs. static
fi
}
setStaticIPv4()
{
# Append these lines to /etc/dhcpcd.conf to enable a static IP
echo "interface $piholeInterface
static ip_address=$IPv4addr
static routers=$IPv4gw
static domain_name_servers=$IPv4gw" | sudo tee -a $dhcpcdFile >/dev/null
sudo ip addr replace dev $piholeInterface $IPv4addr
}
installPihole()
{
sudo apt-get update
sudo apt-get -y upgrade
echo "Installing tools..."
sudo apt-get -y install dnsutils
sudo apt-get -y install bc
sudo apt-get -y install toilet
echo "Installing DNS..."
sudo apt-get -y install dnsutils bc toilet
sudo apt-get -y install dnsmasq
sudo update-rc.d dnsmasq enable
echo "Installing a Web server"
sudo apt-get -y install lighttpd
sudo chown www-data:www-data /var/www
sudo chmod 775 /var/www
sudo apt-get -y install lighttpd php5-common php5-cgi php5
sudo mkdir /var/www/html
sudo chown www-data:www-data /var/www/html
sudo chmod 775 /var/www/html
sudo usermod -a -G www-data pi
echo "Stopping services to modify them..."
sudo service dnsmasq stop
sudo service lighttpd stop
echo "Backing up original config files and downloading Pi-hole ones..."
sudo mv /etc/dnsmasq.conf /etc/dnsmasq.conf.orig
sudo mv /etc/lighttpd/lighttpd.conf /etc/lighttpd/lighttpd.conf.orig
sudo mv /var/www/index.lighttpd.html /var/www/index.lighttpd.orig
sudo curl -o /etc/dnsmasq.conf "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/dnsmasq.conf"
sudo curl -o /etc/lighttpd/lighttpd.conf "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/lighttpd.conf"
sudo mkdir /var/www/pihole
sudo curl -o /var/www/pihole/index.html "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/index.html"
echo "Locating the Pi-hole..."
sudo curl -o /usr/local/bin/gravity.sh "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/gravity.sh"
sudo curl -o /usr/local/bin/chronometer.sh "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/Scripts/chronometer.sh"
sudo mv /var/www/html/index.lighttpd.html /var/www/html/index.lighttpd.orig
sudo mv /etc/crontab /etc/crontab.orig
sudo curl -o /etc/dnsmasq.conf https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/dnsmasq.conf
sudo curl -o /etc/lighttpd/lighttpd.conf https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/lighttpd.conf
sudo mv /etc/crontab /etc/crontab.orig
sudo curl -o /etc/crontab https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/pihole.cron
sudo lighty-enable-mod fastcgi fastcgi-php
sudo mkdir /var/www/html/pihole
sudo curl -o /var/www/html/pihole/index.html https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/index.html
sudo wget https://github.com/jacobsalmela/AdminLTE/archive/master.zip -O /var/www/master.zip
sudo unzip -o /var/www/master.zip -d /var/www/html/
sudo mv /var/www/html/AdminLTE-master /var/www/html/admin
sudo rm /var/www/master.zip 2>/dev/null
sudo touch /var/log/pihole.log
sudo chmod 644 /var/log/pihole.log
sudo chown dnsmasq:root /var/log/pihole.log
sudo curl -o /usr/local/bin/gravity.sh https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/gravity.sh
sudo curl -o /usr/local/bin/chronometer.sh https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/Scripts/chronometer.sh
sudo curl -o /usr/local/bin/whitelist.sh https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/Scripts/whitelist.sh
sudo curl -o /usr/local/bin/piholeLogFlush.sh https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/Scripts/piholeLogFlush.sh
sudo chmod 755 /usr/local/bin/gravity.sh
sudo chmod 755 /usr/local/bin/chronometer.sh
echo "Entering the event horizon..."
sudo chmod 755 /usr/local/bin/whitelist.sh
sudo chmod 755 /usr/local/bin/piholeLogFlush.sh
sudo /usr/local/bin/gravity.sh
}
echo "Restarting..."
sudo shutdown -r now
######## SCRIPT ############
# Start the installer
welcomeDialogs
# Just back up the original Pi-hole right away since it won't take long and it gets it out of the way
backupLegacyPihole
# Find interfaces and let the user choose one
chooseInterface
# Let the user decide if they want to block ads over IPv4 and/or IPv6
use4andor6
# Decide is IPv4 will be used
if [[ "$useIPv4" = true ]];then
echo "Using IPv4"
getStaticIPv4Settings
setStaticIPv4
else
useIPv4=false
echo "IPv4 will NOT be used."
fi
# Decide is IPv6 will be used
if [[ "$useIPv6" = true ]];then
# If only IPv6 is selected, exit because it is not supported yet
if [[ "$useIPv6" = true ]] && [[ "$useIPv4" = false ]];then
useIPv6dialog
exit
else
useIPv6dialog
fi
else
useIPv6=false
echo "IPv6 will NOT be used. Consider a donation at pi-hole.net/donate"
fi
# Install and log everything to a file
installPihole | tee $tmpLog
# Move the log file into /etc/pihole for storage
sudo mv $tmpLog $instalLogLoc
whiptail --msgbox --backtitle "Make it so." --title "Installation Complete!" "Configure your devices to use the Pi-hole as their DNS server using this IP: $IPv4addr.
If you set a new IP address, it should work fine, but you may want to reboot the Pi at some point.
The install log is in /etc/pihole." $r $c
sudo service dnsmasq start
sudo service lighttpd start

View File

@ -0,0 +1,19 @@
#!/bin/bash
# Completely uninstalls the Pi-hole
######### SCRIPT ###########
sudo apt-get -y remove --purge dnsutils bc toilet
sudo apt-get -y remove --purge dnsmasq
sudo apt-get -y remove --purge lighttpd php5-common php5-cgi php5
sudo rm -rf /var/www/html
sudo rm /etc/dnsmasq.conf /etc/dnsmasq.conf.orig
sudo rm /etc/crontab
sudo mv /etc/crontab.orig /etc/crontab
sudo rm /etc/dnsmasq.conf
sudo rm -rf /etc/lighttpd/
sudo rm /var/log/pihole.log
sudo rm /usr/local/bin/gravity.sh
sudo rm /usr/local/bin/chronometer.sh
sudo rm /usr/local/bin/whitelist.sh
sudo rm /usr/local/bin/piholeLogFlush.sh
sudo rm -rf /etc/pihole/

View File

@ -1,169 +1,247 @@
#!/bin/bash
#!/usr/bin/env bash
# Pi-hole: A black hole for Internet advertisements
# (c) 2015 by Jacob Salmela GPL 2.0
# Network-wide ad blocking via your Raspberry Pi
# http://pi-hole.net
# Compiles a list of ad-serving domains by downloading them from multiple sources
# This script should only be run after you have a static IP address set on the Pi
piholeIP=$(hostname -I)
piholeIPfile=/tmp/piholeIP
if [[ -f $piholeIPfile ]];then
# If the file exists, it means it was exported from the installation script and we should use that value instead of detecting it in this script
piholeIP=$(cat $piholeIPfile)
rm $piholeIPfile
else
# Otherwise, the IP address can be taken directly from the machine, which will happen when the script is run by the user and not the installation script
IPv4dev=$(ip route get 8.8.8.8 | awk '{print $5}')
piholeIPCIDR=$(ip -o -f inet addr show dev $IPv4dev | awk '{print $4}')
piholeIP=${piholeIPCIDR%/*}
fi
# Ad-list sources--one per line in single quotes
# The mahakala source is commented out due to many users having issues with it blocking legitimate domains.
# Uncomment at your own risk
sources=('https://adaway.org/hosts.txt'
'http://adblock.gjtech.net/?format=unix-hosts'
#'http://adblock.mahakala.is/'
'http://hosts-file.net/.%5Cad_servers.txt'
'http://hosts-file.net/ad_servers.txt'
'http://www.malwaredomainlist.com/hostslist/hosts.txt'
'http://pgl.yoyo.org/adservers/serverlist.php?'
'http://someonewhocares.org/hosts/hosts'
'http://winhelp2002.mvps.org/hosts.txt')
# Variables for various stages of downloading and formatting the list
adList=/etc/pihole/gravity.list
origin=/etc/pihole
piholeDir=/etc/pihole
justDomainsExtension=domains
matter=pihole.0.matter.txt
andLight=pihole.1.andLight.txt
supernova=pihole.2.supernova.txt
eventHorizon=pihole.3.eventHorizon.txt
accretionDisc=pihole.4.accretionDisc.txt
eyeOfTheNeedle=pihole.5.wormhole.txt
basename=pihole
piholeDir=/etc/$basename
adList=$piholeDir/gravity.list
blacklist=$piholeDir/blacklist.txt
latentBlacklist=$origin/latentBlacklist.txt
whitelist=$piholeDir/whitelist.txt
latentWhitelist=$origin/latentWhitelist.txt
latentWhitelist=$piholeDir/latentWhitelist.txt
justDomainsExtension=domains
matter=$basename.0.matter.txt
andLight=$basename.1.andLight.txt
supernova=$basename.2.supernova.txt
eventHorizon=$basename.3.eventHorizon.txt
accretionDisc=$basename.4.accretionDisc.txt
eyeOfTheNeedle=$basename.5.wormhole.txt
# After setting defaults, check if there's local overrides
if [[ -r $piholeDir/pihole.conf ]];then
echo "** Local calibration requested..."
. $piholeDir/pihole.conf
. $piholeDir/pihole.conf
fi
echo "** Neutrino emissions detected..."
# Create the pihole resource directory if it doesn't exist. Future files will be stored here
if [[ -d $piholeDir ]];then
:
else
echo "** Creating pihole directory..."
sudo mkdir $piholeDir
fi
# Add additional swap to prevent the "Error fork: unable to allocate memory" message: https://github.com/jacobsalmela/pi-hole/issues/37
function createSwapFile()
#########################
{
echo "** Creating more swap space to accomodate large solar masses..."
sudo dphys-swapfile swapoff
sudo curl -s -o /etc/dphys-swapfile https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/dphys-swapfile
sudo dphys-swapfile setup
sudo dphys-swapfile swapon
}
if [[ -n "$noSwap" ]]; then
# if $noSwap is set, don't do anything
:
elif [[ -f /etc/dphys-swapfile ]];then
swapSize=$(cat /etc/dphys-swapfile | grep -m1 CONF_SWAPSIZE | cut -d'=' -f2)
if [[ $swapSize != 500 ]];then
mv /etc/dphys-swapfile /etc/dphys-swapfile.orig
echo "** Current swap size is $swapSize"
createSwapFile
else
:
fi
else
echo "** No swap file found. Creating one..."
createSwapFile
fi
# Loop through domain list. Download each one and remove commented lines (lines beginning with '# 'or '/') and blank lines
for ((i = 0; i < "${#sources[@]}"; i++))
do
url=${sources[$i]}
# Get just the domain from the URL
domain=$(echo "$url" | cut -d'/' -f3)
# Save the file as list.#.domain
saveLocation=$origin/list.$i.$domain.$justDomainsExtension
echo -n "Getting $domain list... "
# Use a case statement to download lists that need special cURL commands to complete properly
case "$domain" in
"adblock.mahakala.is") data=$(curl -s -A 'Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0' -e http://forum.xda-developers.com/ -z $saveLocation $url);;
"pgl.yoyo.org") data=$(curl -s -d mimetype=plaintext -d hostformat=hosts -z $saveLocation $url);;
*) data=$(curl -s -z $saveLocation -A "Mozilla/10.0" $url);;
esac
if [[ -n "$data" ]];then
# Remove comments and print only the domain name
# Most of the lists downloaded are already in hosts file format but the spacing/formating is not contigious
# This helps with that and makes it easier to read
# It also helps with debugging so each stage of the script can be researched more in depth
echo "$data" | awk 'NF {if ($1 !~ "#") { if (NF>1) {print $2} else {print $1}}}' | \
sed -e 's/^[. \t]*//' -e 's/\.\.\+/./g' -e 's/[. \t]*$//' | grep "\." > $saveLocation
echo "Done."
else
echo "Skipping list because it does not have any new entries."
fi
done
# Find all files with the .domains extension and compile them into one file and remove CRs
echo "** Aggregating list of domains..."
find $origin/ -type f -name "*.$justDomainsExtension" -exec cat {} \; | tr -d '\r' > $origin/$matter
# Append blacklist entries if they exist
if [[ -f $blacklist ]];then
numberOf=$(cat $blacklist | sed '/^\s*$/d' | wc -l)
echo "** Blacklisting $numberOf domain(s)..."
cat $blacklist >> $origin/$matter
else
:
fi
function gravity_advanced()
###########################
{
numberOf=$(cat $origin/$andLight | sed '/^\s*$/d' | wc -l)
echo "** $numberOf domains being pulled in by gravity..."
# Remove carriage returns and preceding whitespace
cat $origin/$andLight | sed $'s/\r$//' | sed '/^\s*$/d' > $origin/$supernova
# collapse - begin formation of pihole
function gravity_collapse() {
echo "** Neutrino emissions detected..."
# Create the pihole resource directory if it doesn't exist. Future files will be stored here
if [[ -d $piholeDir ]];then
# Temporary hack to allow non-root access to pihole directory
# Will update later, needed for existing installs, new installs should
# create this directory as non-root
sudo chmod 777 $piholeDir
find "$piholeDir" -type f -exec sudo chmod 666 {} \;
else
echo "** Creating pihole directory..."
mkdir $piholeDir
fi
}
# patternCheck - check to see if curl downloaded any new files.
function gravity_patternCheck() {
patternBuffer=$1
# check if the patternbuffer is a non-zero length file
if [[ -s "$patternBuffer" ]];then
# Some of the blocklists are copyright, they need to be downloaded
# and stored as is. They can be processed for content after they
# have been saved.
cp $patternBuffer $saveLocation
echo "List updated, transport successful..."
else
# curl didn't download any host files, probably because of the date check
echo "No changes detected, transport skipped..."
fi
}
# transport - curl the specified url with any needed command extentions
function gravity_transport() {
url=$1
cmd_ext=$2
agent=$3
# tmp file, so we don't have to store the (long!) lists in RAM
patternBuffer=$(mktemp)
heisenbergCompensator=""
if [[ -r $saveLocation ]]; then
# if domain has been saved, add file for date check to only download newer
heisenbergCompensator="-z $saveLocation"
fi
# Silently curl url
curl -s $cmd_ext $heisenbergCompensator -A "$agent" $url > $patternBuffer
# Check for list updates
gravity_patternCheck $patternBuffer
# Cleanup
rm -f $patternBuffer
}
# spinup - main gravity function
function gravity_spinup() {
# Loop through domain list. Download each one and remove commented lines (lines beginning with '# 'or '/') and # blank lines
for ((i = 0; i < "${#sources[@]}"; i++))
do
url=${sources[$i]}
# Get just the domain from the URL
domain=$(echo "$url" | cut -d'/' -f3)
# Save the file as list.#.domain
saveLocation=$piholeDir/list.$i.$domain.$justDomainsExtension
activeDomains[$i]=$saveLocation
agent="Mozilla/10.0"
echo -n " Getting $domain list: "
# Use a case statement to download lists that need special cURL commands
# to complete properly and reset the user agent when required
case "$domain" in
"adblock.mahakala.is")
agent='Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0'
cmd_ext="-e http://forum.xda-developers.com/"
;;
"pgl.yoyo.org")
cmd_ext="-d mimetype=plaintext -d hostformat=hosts"
;;
# Default is a simple request
*) cmd_ext=""
esac
gravity_transport $url $cmd_ext $agent
done
}
# Schwarzchild - aggregate domains to one list and add blacklisted domains
function gravity_Schwarzchild() {
# Find all active domains and compile them into one file and remove CRs
echo "** Aggregating list of domains..."
truncate -s 0 $piholeDir/$matter
for i in "${activeDomains[@]}"
do
cat $i |tr -d '\r' >> $piholeDir/$matter
done
}
# Pulsar - White/blacklist application
function gravity_pulsar() {
# Append blacklist entries if they exist
if [[ -r $blacklist ]];then
numberOf=$(cat $blacklist | sed '/^\s*$/d' | wc -l)
echo "** Blacklisting $numberOf domain(s)..."
cat $blacklist >> $piholeDir/$matter
fi
# Whitelist (if applicable) domains
if [[ -r $whitelist ]];then
# Remove whitelist entries
numberOf=$(cat $whitelist | sed '/^\s*$/d' | wc -l)
plural=; [[ "$numberOf" != "1" ]] && plural=s
echo "** Whitelisting $numberOf domain${plural}..."
# Append a "$" to the end, prepend a "^" to the beginning, and
# replace "." with "\." of each line to turn each entry into a
# regexp so it can be parsed out with grep -x
awk -F '[# \t]' 'NF>0&&$1!="" {print "^"$1"$"}' $whitelist | sed 's/\./\\./g' > $latentWhitelist
else
rm $latentWhitelist
fi
# Prevent our sources from being pulled into the hole
plural=; [[ "${#sources[@]}" != "1" ]] && plural=s
echo "** Whitelisting ${#sources[@]} ad list source${plural}..."
for url in ${sources[@]}
do
echo "$url" | awk -F '/' '{print "^"$3"$"}' | sed 's/\./\\./g' >> $latentWhitelist
done
# Remove whitelist entries from list
grep -vxf $latentWhitelist $piholeDir/$matter > $piholeDir/$andLight
}
function gravity_unique() {
# Sort and remove duplicates
cat $origin/$supernova | sort | uniq > $origin/$eventHorizon
numberOf=$(cat $origin/$eventHorizon | sed '/^\s*$/d' | wc -l)
sort -u $piholeDir/$supernova > $piholeDir/$eventHorizon
numberOf=$(wc -l < $piholeDir/$eventHorizon)
echo "** $numberOf unique domains trapped in the event horizon."
}
function gravity_hostFormat() {
# Format domain list as "192.168.x.x domain.com"
echo "** Formatting domains into a HOSTS file..."
cat $origin/$eventHorizon | awk '{sub(/\r$/,""); print "'"$piholeIP"'" $0}' > $origin/$accretionDisc
cat $piholeDir/$eventHorizon | awk '{sub(/\r$/,""); print "'"$piholeIP"' " $0}' > $piholeDir/$accretionDisc
# Copy the file over as /etc/pihole/gravity.list so dnsmasq can use it
sudo cp $origin/$accretionDisc $adList
kill -HUP $(pidof dnsmasq)
}
cp $piholeDir/$accretionDisc $adList
}
# Whitelist (if applicable) then remove duplicates and format for dnsmasq
if [[ -f $whitelist ]];then
# Remove whitelist entries
numberOf=$(cat $whitelist | sed '/^\s*$/d' | wc -l)
plural=; [[ "$numberOf" != "1" ]] && plural=s
echo "** Whitelisting $numberOf domain${plural}..."
# Append a "$" to the end, prepend a "^" to the beginning, and
# replace "." with "\." of each line to turn each entry into a
# regexp so it can be parsed out with grep -x
awk -F '[# \t]' 'NF>0&&$1!="" {print "^"$1"$"}' $whitelist | sed 's/\./\\./g' > $latentWhitelist
else
rm $latentWhitelist
fi
# blackbody - remove any remnant files from script processes
function gravity_blackbody() {
# Loop through list files
for file in $piholeDir/*.$justDomainsExtension
do
# If list is in active array then leave it (noop) else rm the list
if [[ " ${activeDomains[@]} " =~ " ${file} " ]]; then
:
else
rm -f $file
fi
done
}
# Prevent our sources from being pulled into the hole
plural=; [[ "${#sources[@]}" != "1" ]] && plural=s
echo "** Whitelisting ${#sources[@]} ad list source${plural}..."
for url in ${sources[@]}
do
echo "$url" | awk -F '/' '{print "^"$3"$"}' | sed 's/\./\\./g' >> $latentWhitelist
done
function gravity_advanced() {
# Remove comments and print only the domain name
# Most of the lists downloaded are already in hosts file format but the spacing/formating is not contigious
# This helps with that and makes it easier to read
# It also helps with debugging so each stage of the script can be researched more in depth
awk '($1 !~ /^#/) { if (NF>1) {print $2} else {print $1}}' $piholeDir/$andLight | \
sed -nr -e 's/\.{2,}/./g' -e '/\./p' > $piholeDir/$supernova
grep -vxf $latentWhitelist $origin/$matter > $origin/$andLight
numberOf=$(wc -l < $piholeDir/$supernova)
echo "** $numberOf domains being pulled in by gravity..."
gravity_unique
sudo kill -HUP $(pidof dnsmasq)
}
gravity_collapse
gravity_spinup
gravity_Schwarzchild
gravity_pulsar
gravity_hostFormat
gravity_advanced
gravity_blackbody