Merge pull request #4 from jacobsalmela/master

Rebase

README.md

@@ -1,26 +1,50 @@
 # Raspberry Pi Ad Blocker 
 **A black hole for ads, hence Pi-hole**
 
-![Pi-hole](http://www.hdwallpapersimages.com/wp-content/uploads/2014/03/Black-Hole-Images-540x303.jpg)
+![Pi-hole](http://i.imgur.com/wd5ltCU.png)
 
-The Pi-hole is a DNS/Web server that will **block ads for any device**.
+The Pi-hole is a DNS/Web server that will **block ads for any device on your network**.
 
 ## Coverage
 Featured on [MakeUseOf](http://www.makeuseof.com/tag/adblock-everywhere-raspberry-pi-hole-way/) and [Lifehacker](http://lifehacker.com/turn-a-raspberry-pi-into-an-ad-blocker-with-a-single-co-1686093533)!
 
 ## Automated Install
-On a clean installation of Raspbian, you can run this command to **auto-install the Pi-hole**.  Once installed, configure any device to use the Raspberry Pi as your DNS server and the ads will be blocked.
+
+1. Install Raspbian 
+2. Set a **static** IP address
+3. Run the command below
 
 ```curl -s "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/automated%20install/basic-install.sh" | bash```
 
+Once installed, **configure any device to use the Raspberry Pi as its DNS server and the ads will be blocked**.  You can also configure your router's DHCP options to assign the Pi as clients DNS server so they do not need to do it manually.  
+
+A more detailed explanation of the installation can be found [here](http://jacobsalmela.com/block-millions-ads-network-wide-with-a-raspberry-pi-hole-2-0).
+
+[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif "AdminLTE Presentation")](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=3J2L3Z4DHW9UY "Donate")
+
 ## Gravity
-The [gravity-adv.sh](https://github.com/jacobsalmela/pi-hole/blob/master/gravity-adv.sh) does most of the magic.  The script pulls in ad domains from many sources and compiles them into a single list of [over 120,000 entries](http://jacobsalmela.com/blocking-ads-from-120000-domains/).
+The [gravity.sh](https://github.com/jacobsalmela/pi-hole/blob/master/gravity.sh) does most of the magic.  The script pulls in ad domains from many sources and compiles them into a single list of [over 1.6 million entries](http://jacobsalmela.com/block-millions-ads-network-wide-with-a-raspberry-pi-hole-2-0).
 
 ## Whitelist and blacklist
-You can add a whitelist or blacklist in ```/etc/pihole/``` and the script will apply those files automatically.
+You can add a `whitelist.txt` or `blacklist.txt` in `/etc/pihole/` and the script will apply those files automatically.
+
+## Web Interface
+I am also working on a [Web interface](https://github.com/jacobsalmela/AdminLTE#pi-hole-admin-dashboard) so you can view stats and change settings.
+
+##  Custom Config File
+If you want to use your own variables for the gravity script (i.e. storing the files in a different location) and don't want to have to change them every time there is an update to the script, create a file called `/etc/pihole/pihole.conf`. In it, you should add your own variables in a similar fashion as shown below:
+
+```
+origin=/var/run/pihole
+adList=/etc/dnsmasq.d/adList
+```
+
+See [this PR](https://github.com/jacobsalmela/pi-hole/pull/20) for more details.
+
+### How It Works
+A technical and detailed description can be found [here](http://jacobsalmela.com/block-millions-ads-network-wide-with-a-raspberry-pi-hole-2-0)!
 
 ## Other Operating Systems
-This script will work for other UNIX-like systems with some slight **modifications**.  As long as you can install dnsmasq and a Webserver, it should work OK.  The automated install only works for a clean install of Raspiban right now since that is how the project originated.
+This script will work for other UNIX-like systems with some slight **modifications**.  As long as you can install `dnsmasq` and a Webserver, it should work OK.  The automated install only works for a clean install of Raspiban right now since that is how the project originated.
 
-## Optimizations
+[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif "AdminLTE Presentation")](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=3J2L3Z4DHW9UY "Donate")
-I am working on some great optimizations to allow the script to run much faster.  I also have a bunch of new sources for ad domains but I still need to see if the lists are OK.

advanced/Scripts/chronometer.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+# Displays Pi-hole stats on the Adafruit PiTFT 2.8" touch screen
+# Set the pi user to log in automatically and run this script from /etc/profile
+for (( ; ; ))
+do
+	clear
+	# Displays a colorful Pi-hole logo
+	toilet -f small -F gay Pi-hole
+	echo "        $(ifconfig eth0 | awk '/inet addr/ {print $2}' | cut -d':' -f2)"
+	echo ""
+	uptime | cut -d' ' -f11-
+	echo "-------------------------------"
+	# Uncomment to continually read the log file and display the current domain being blocked
+	#tail -f /var/log/pihole.log | awk '/\/etc\/pihole\/gravity.list/ {if ($7 != "address" && $7 != "name" && $7 != "/etc/pihole/gravity.list") print $7; else;}'
+	
+	today=$(date "+%b %e")
+	todaysQueryCount=$(cat /var/log/pihole.log | grep "$today" | awk '/query/ {print $7}' | wc -l)
+	todaysQueryCountV4=$(cat /var/log/pihole.log | grep "$today" | awk '/query/ && /\[A\]/ {print $7}' | wc -l)
+	todaysQueryCountV6=$(cat /var/log/pihole.log | grep "$today" | awk '/query/ && /\[AAAA\]/ {print $7}' | wc -l)
+	todaysAdsEliminated=$(cat /var/log/pihole.log | grep "$today" | awk '/\/etc\/pihole\/gravity.list/ {print $7}' | wc -l)
+	dividend=$(echo "$todaysAdsEliminated/$todaysQueryCount" | bc -l)
+	fp=$(echo "$dividend*100" | bc -l)
+	percentAds=$(echo ${fp:0:4})
+	
+	echo "Queries:       $todaysQueryCountV4 / $todaysQueryCountV6"
+	echo "Pi-holed:      $todaysAdsEliminated  ($percentAds%)"
+	sleep 5
+done

advanced/Scripts/whitelist.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# For each argument passed to this script
+for var in "$@"
+do
+        echo "Whitelisting $var..."
+        # Use sed to search for the domain in /etc/pihole/gravity.list and remove it using an in-place edit
+        sed -i "/$var/d" /etc/pihole/gravity.list
+        # Also add the domain to the whitelist.txt in /etc/pihole
+        echo "$var" >> /etc/pihole/whitelist.txt
+done
+echo "** $# domain(s) whitelisted."
+# Force dnsmasq to reload /etc/pihole/gravity.list
+kill -HUP $(pidof dnsmasq)

advanced/dnsmasq.conf

@@ -1,9 +1,12 @@
+addn-hosts=/etc/pihole/gravity.list
 domain-needed
 bogus-priv
 no-resolv
-server=127.0.0.1
 server=8.8.8.8
 server=8.8.4.4
 interface=eth0
+listen-address=127.0.0.1
 cache-size=10000
-log-queries
+log-queries
+log-facility=/var/log/pihole.log
+local-ttl=300

advanced/dphys-swapfile

@@ -0,0 +1 @@
+CONF_SWAPSIZE=500

advanced/enable-dns.sh

@@ -1,8 +0,0 @@
-# !/bin/bash
-# Download the ad list
-sudo /usr/local/bin/gravity.sh
-
-# Enable DNS and start blocking ads
-sudo mv /etc/dnsmasq.conf /etc/dnsmasq.conf.orig
-sudo mv /etc/dnsmasq.conf.pihole /etc/dnsmasq.conf
-sudo service dnsmasq start

advanced/gravity.sh

@@ -1,29 +0,0 @@
-#!/bin/bash
-# /usr/local/bin/gravity.sh
-
-# URL to pull list of known ad servers from
-adListURL="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
-
-# Address to send ads to
-piholeIP="127.0.0.1"
-
-# Where the list of ad servers are stored once downloaded
-# Any file in /etc/dnsmasq.d is loaded automatically when the service starts
-adFile="/etc/dnsmasq.d/adList.conf"
-
-# The temporary file for holding
-eventHorizion="/etc/dnsmasq.d/adList.conf.tmp"
- 
-# Parses out the default 127.0.0.1 address and replaces it with the IP where ads will be sent
-curl $adListURL | sed "s/127\.0\.0\.1/$piholeIP/" > $eventHorizion
-
-sleep 10
-
-# If the temporary list of ad servers already exists (the eventHorizion)  
-if [ -f "$eventHorizion" ];then
-	# Then replace it as the new ad file	
-	mv -f $eventHorizion $adFile
-else
-	echo "Error building the ad list, please try again."
-	exit 1
-fi

advanced/index.html

@@ -1,6 +1,4 @@
 <html>
-<head>
-</head>
 <body>
 </body>
 </html>

advanced/lighttpd.conf

@@ -1,12 +1,10 @@
 server.modules = (
-	"mod_access",
+	"mod_expire",
-	"mod_alias",
 	"mod_compress",
  	"mod_redirect",
-	"mod_auth",
 	"mod_rewrite"
 )
-
+ 
 server.document-root        = "/var/www"
 server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
 server.errorlog             = "/var/log/lighttpd/error.log"
@@ -14,22 +12,27 @@ server.pid-file             = "/var/run/lighttpd.pid"
 server.username             = "www-data"
 server.groupname            = "www-data"
 server.port                 = 80
-
+ 
-
+ 
 index-file.names            = ( "index.php", "index.html", "index.lighttpd.html" )
 url.access-deny             = ( "~", ".inc" )
 static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
-
+ 
 compress.cache-dir          = "/var/cache/lighttpd/compress/"
 compress.filetype           = ( "application/javascript", "text/css", "text/html", "text/plain" )
-
+ 
 # default listening port for IPv6 falls back to the IPv4 port
 include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
 include_shell "/usr/share/lighttpd/create-mime.assign.pl"
 include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
-
+ 
+# Set access to 1 day for better query performance when the list gets so large
+# http://jacobsalmela.com/raspberry-pi-block-ads-adtrap/#comment-2013820434
+$HTTP["url"] =~ "^/pihole/" {
+        expire.url = ("" => "access plus 1 days")
+}
+ 
 # Rewrites all URLs to the /var/www/pihole/index.html
-# This should be a blank file (for speed)  
 $HTTP["host"] =~ ".*" {
      url.rewrite = (".*" => "pihole/index.html")
 }

automated install/basic-install.sh

@@ -18,18 +18,32 @@ echo " |_|   |_|      |_| |_|\___/|_|\___| "
 echo "                                    "
 echo "      Raspberry Pi Ad-blocker       "
 echo "									  "
-echo "         Automated install          "
+echo "Set a static IP before running this!"
-echo "			 --Advanced-- 			  "
+echo "			             			  "
+echo "	    Press Enter when ready        "
 echo "									  "
-echo "									  "
+read
-sleep 2
+
+if [[ -f /etc/dnsmasq.d/adList.conf ]];then
+	echo "Original Pi-hole detected.  Initiating sub space transport..."
+	sudo mkdir -p /etc/pihole/original/
+	mv /etc/dnsmasq.d/adList.conf /etc/pihole/original/adList.conf.$(date "+%Y-%m-%d")
+	mv /etc/dnsmasq.conf /etc/pihole/original/dnsmasq.conf.$(date "+%Y-%m-%d")
+	mv /etc/resolv.conf /etc/pihole/original/resolv.conf.$(date "+%Y-%m-%d")
+	mv /etc/lighttpd/lighttpd.conf /etc/pihole/original/lighttpd.conf.$(date "+%Y-%m-%d")
+	mv /var/www/pihole/index.html /etc/pihole/original/index.html.$(date "+%Y-%m-%d")
+	mv /usr/local/bin/gravity.sh /etc/pihole/original/gravity.sh.$(date "+%Y-%m-%d")
+else
+	:
+fi
 
 echo "Updating the Pi..."
 sudo apt-get update
 sudo apt-get -y upgrade
 
 echo "Installing DNS..."
-sudo apt-get -y install dnsutils dnsmasq
+sudo apt-get -y install dnsmasq
+sudo update-rc.d dnsmasq enable
 
 echo "Installing a Web server"
 sudo apt-get -y install lighttpd
@@ -48,19 +62,16 @@ sudo mv /var/www/index.lighttpd.html /var/www/index.lighttpd.orig
 sudo curl -o /etc/dnsmasq.conf "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/dnsmasq.conf"
 sudo curl -o /etc/lighttpd/lighttpd.conf "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/lighttpd.conf"
 sudo mkdir /var/www/pihole
-sudo curl -o /var/www/pihole/index.html "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/index.html"
+sudo curl -o /var/www/pihole/index.html "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/index.html"
-
-echo "Turning services back on..."
-sudo service dnsmasq start
-sudo service lighttpd start
 
 echo "Locating the Pi-hole..."
-sudo curl -o /usr/local/bin/gravity.sh "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/gravity-adv.sh"
+sudo curl -o /usr/local/bin/gravity.sh "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/gravity.sh"
+sudo curl -o /usr/local/bin/chronometer.sh "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/Scripts/chronometer.sh"
 sudo chmod 755 /usr/local/bin/gravity.sh
+sudo chmod 755 /usr/local/bin/chronometer.sh
+
 echo "Entering the event horizon..."
 sudo /usr/local/bin/gravity.sh
 
-echo "Restarting services..."
+echo "Restarting..."
-sudo service dnsmasq restart
+sudo shutdown -r now
-sudo service lighttpd restart
-

block hulu ads/dnsmasq.conf

@@ -1,12 +0,0 @@
-no-resolv
-no-poll
-server=/localnet/192.168.0.1
-server=/localnet/8.8.8.8
-server=/localnet/8.8.4.4
-domain-needed
-interface=eth0
-strict-order
-min-port=4096
-cache-size=10000
-log-queries
-bogus-priv

block hulu ads/hulAdList.conf

@@ -1,2 +0,0 @@
-address=/ads.hulu.com/192.168.1.101
-address=/ads-v-darwin.hulu.com/192.168.1.101

block hulu ads/index.html

@@ -1,8 +0,0 @@
-<html>
-<head>
-<title>Pi-hole blocked an ad</title>
-</head>
-<body>
-<h1>No ads here</h1>
-</body>
-</html>

block hulu ads/lighttpd.conf

@@ -3,7 +3,7 @@ server.modules = (
 	"mod_alias",
 	"mod_compress",
  	"mod_redirect",
-#   "mod_rewrite"
+   "mod_rewrite"
 )
 
 server.document-root        = "/var/www"
@@ -27,6 +27,6 @@ include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
 include_shell "/usr/share/lighttpd/create-mime.assign.pl"
 include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
 
-$HTTP["host"] =~ "ads.hulu.com|ads-v-darwin.hulu.com|" {
+$HTTP["host"] =~ "ads.hulu.com|ads-v-darwin.hulu.com" {
-	url.redirect = ( "^/(.*)" => "http://192.168.1.101:8200/MediaItems/19.mov")
+	url.redirect = ( ".*" => "http://192.168.1.101:8200/MediaItems/19.mov")
 }

block hulu ads/setup-resolv.sh

@@ -1,6 +0,0 @@
-# !/bin/bash
-# DNS config file
-# Run as a local script since modifying it will disconnect the Internet connection
-sudo mv /etc/dnsmasq.conf /etc/dnsmasq.conf.orig
-sudo mv /etc/dnsmasq.conf.pihole /etc/dnsmasq.conf
-sudo service dnsmasq start

block hulu ads/setup.sh

@@ -1,71 +0,0 @@
-#!/bin/bash
-# Block Hulu Plus ads using a Raspberry Pi
-
-# Install with this command (from the Pi):
-#
-# curl -s https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/block%20hulu%20ads/setup.sh | bash
-#
-# Or run the commands below in order
-
-# Update the Pi
-sudo apt-get update
-#sudo apt-get -y upgrade
-
-# Install DNS
-sudo apt-get -y install dnsutils dnsmasq
-sudo service dnsmasq stop
-
-# Install Web server
-sudo apt-get -y install lighttpd
-sudo service lighttpd stop
-
-# Install streaming software
-sudo apt-get -y install minidlna
-sudo service minidlna stop
-
-# Configure Web server
-#sudo lighty-enable-mod fastcgi-php
-sudo chown www-data:www-data /var/www
-sudo chmod 775 /var/www
-sudo usermod -a -G www-data pi
-sudo mv /var/www/index.lighttpd.html /var/www/index.lighttpd.orig
-sudo curl -o /var/www/index.html "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/block%20hulu%20ads/index.html"
-sudo mv /etc/lighttpd/lighttpd.conf /etc/lighttpd/lighttpd.conf.orig
-sudo curl -o /etc/lighttpd/lighttpd.conf "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/block%20hulu%20ads/lighttpd.conf"
-
-# Configure streaming service
-sudo mv /etc/minidlna.conf /etc/minidlna.conf.orig
-sudo mkdir /var/lib/minidlna/videos/
-sudo curl -o /etc/minidlna.conf "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/block%20hulu%20ads/minidlna.conf"
-sudo service minidlna start
-sudo curl -o /var/lib/minidlna/videos/pi-hole.mov "https://dl.dropboxusercontent.com/u/16366947/Documents/Videos/pi-hole.mov"
-sudo service minidlna force-reload
-tail /var/log/minidlna.log
-
-# Configure DNS
-sudo curl -o /etc/dnsmasq.conf.pihole "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/dnsmasq.conf"
-sudo curl -o /tmp/piholedns.sh "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/block%20hulu%20ads/setup-resolv.sh"
-sudo chmod 755 /tmp/piholedns.sh
-
-# Download [advanced] ad-blocking script and then run it
-# http://jacobsalmela.com/raspberry-pi-ad-blocker-advanced-setup/
-sudo curl -o /usr/local/bin/gravity.sh "https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/gravity-adv.sh"
-sudo chmod 755 /usr/local/bin/gravity.sh
-clear
-echo ""
-echo ""
-echo ""
-echo ""
-echo "Go get some coffee--this will take a while"
-echo ""
-echo ""
-echo ""
-echo ""
-sleep 5
-sudo /usr/local/bin/gravity.sh
-sudo service dnsmasq stop
-
-# Restart everything to apply all the changes
-sudo service lighttpd start
-sudo service minidlna start
-sudo /tmp/piholedns.sh

dnsmasq.conf

@@ -1,6 +0,0 @@
-domain-needed
-interface=eth0
-min-port=4096
-cache-size=10000
-log-queries
-bogus-priv

gravity-adv.sh

@@ -1,66 +0,0 @@
-#!/bin/bash
-# The Pi-hole now blocks over 120,000 ad domains
-# Address to send ads to (the RPi)
-piholeIP="127.0.0.1"
-# Optionally, uncomment to automatically detect the local IP address.
-#piholeIP=$(hostname -I)
-
-# Config file to hold URL rules
-eventHorizion="/etc/dnsmasq.d/adList.conf"
-blacklist=/etc/pihole/blacklist.txt
-whitelist=/etc/pihole/whitelist.txt
-
-# Create the pihole resource directory if it doesn't exist.  Future files will be stored here
-if [[ -d /etc/pihole/ ]];then
-	:
-else
-	echo "Forming pihole directory..."
-	sudo mkdir /etc/pihole
-fi
-
-echo "Getting yoyo ad list..." # Approximately 2452 domains at the time of writing
-curl -s -d mimetype=plaintext -d hostformat=unixhosts http://pgl.yoyo.org/adservers/serverlist.php? | sort > /tmp/matter.txt
-echo "Getting winhelp2002 ad list..." # 12985 domains
-curl -s http://winhelp2002.mvps.org/hosts.txt | grep -v "#" | grep -v "127.0.0.1" | sed '/^$/d' | sed 's/\ /\\ /g' | awk '{print $2}' | sort >> /tmp/matter.txt
-echo "Getting adaway ad list..." # 445 domains
-curl -s https://adaway.org/hosts.txt | grep -v "#" | grep -v "::1" | sed '/^$/d' | sed 's/\ /\\ /g' | awk '{print $2}' | grep -v '^\\' | grep -v '\\$' | sort >> /tmp/matter.txt
-echo "Getting hosts-file ad list..." # 28050 domains
-curl -s http://hosts-file.net/.%5Cad_servers.txt | grep -v "#" | grep -v "::1" | sed '/^$/d' | sed 's/\ /\\ /g' | awk '{print $2}' | grep -v '^\\' | grep -v '\\$' | sort >> /tmp/matter.txt
-echo "Getting malwaredomainlist ad list..." # 1352 domains
-curl -s http://www.malwaredomainlist.com/hostslist/hosts.txt | grep -v "#" | sed '/^$/d' | sed 's/\ /\\ /g' | awk '{print $3}' | grep -v '^\\' | grep -v '\\$' | sort >> /tmp/matter.txt
-echo "Getting adblock.gjtech ad list..." # 696 domains
-curl -s http://adblock.gjtech.net/?format=unix-hosts | grep -v "#" | sed '/^$/d' | sed 's/\ /\\ /g' | awk '{print $2}' | grep -v '^\\' | grep -v '\\$' | sort >> /tmp/matter.txt
-echo "Getting someone who cares ad list..." # 10600
-curl -s http://someonewhocares.org/hosts/hosts | grep -v "#" | sed '/^$/d' | sed 's/\ /\\ /g' | grep -v '^\\' | grep -v '\\$' | awk '{print $2}' | grep -v '^\\' | grep -v '\\$' | sort >> /tmp/matter.txt
-echo "Getting Mother of All Ad Blocks list..." # 102168 domains!! Thanks Kacy
-curl -A 'Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0' -e http://forum.xda-developers.com/ http://adblock.mahakala.is/ | grep -v "#" | awk '{print $2}' | sort >> /tmp/matter.txt
-
-# Add entries from the local blacklist file if it exists in /etc/pihole directory
-if [[ -f $blacklist ]];then
-        echo "Getting the local blacklist from /etc/pihole directory"
-        cat $blacklist >> /tmp/matter.txt
-else
-        :
-fi
-
-# Sort the aggregated results and remove any duplicates
-# Remove entries from the whitelist file if it exists at the root of the current user's home folder
-if [[ -f $whitelist ]];then
-	echo "Removing duplicates, whitelisting, and formatting the list of domains..."
-	cat /tmp/matter.txt | sed $'s/\r$//' | sort | uniq | sed '/^$/d' | grep -v -x -f $whitelist | awk -v "IP=$piholeIP" '{sub(/\r$/,""); print "address=/"$0"/"IP}' > /tmp/andLight.txt
-	numberOfSitesWhitelisted=$(cat $whitelist | wc -l | sed 's/^[ \t]*//')
-	echo "$numberOfSitesWhitelisted domains whitelisted."
-else
-	echo "Removing duplicates and formatting the list of domains..."
-	cat /tmp/matter.txt | sed $'s/\r$//' | sort | uniq | sed '/^$/d' | awk -v "IP=$piholeIP" '{sub(/\r$/,""); print "address=/"$0"/"IP}' > /tmp/andLight.txt
-fi
-
-# Count how many domains/whitelists were added so it can be displayed to the user
-numberOfAdsBlocked=$(cat /tmp/andLight.txt | wc -l | sed 's/^[ \t]*//')
-echo "$numberOfAdsBlocked ad domains blocked."
-
-# Turn the file into a dnsmasq config file
-sudo mv /tmp/andLight.txt $eventHorizion
-
-# Restart DNS
-sudo service dnsmasq restart

gravity.sh

@@ -1,28 +1,161 @@
 #!/bin/bash
-# /usr/local/bin/gravity.sh
+# http://pi-hole.net
+# Compiles a list of ad-serving domains by downloading them from multiple sources 
 
-# URL to pull list of known ad servers from
+# This script should only be run after you have a static IP address set on the Pi
-adListURL="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
+piholeIP=$(hostname -I)
 
-# Address to send ads to
+# Ad-list sources--one per line in single quotes
-piholeIP="127.0.0.1"
+sources=('https://adaway.org/hosts.txt'
+'http://adblock.gjtech.net/?format=unix-hosts'
+'http://adblock.mahakala.is/'
+'http://hosts-file.net/.%5Cad_servers.txt'
+'http://www.malwaredomainlist.com/hostslist/hosts.txt'
+'http://pgl.yoyo.org/adservers/serverlist.php?'
+'http://someonewhocares.org/hosts/hosts'
+'http://winhelp2002.mvps.org/hosts.txt')
 
-# Where the list of ad servers are stored once downloaded
+# Variables for various stages of downloading and formatting the list
-# Any file in /etc/dnsmasq.d is loaded automatically when the service starts
+adList=/etc/pihole/gravity.list
-adFile="/etc/dnsmasq.d/adList.conf"
+origin=/etc/pihole
-
+piholeDir=/etc/pihole
-# The temporary file for holding
+if [[ -f $piholeDir/pihole.conf ]];then
-eventHorizion="/etc/dnsmasq.d/adList.conf.tmp"
+	. $piholeDir/pihole.conf
- 
-# Parses out the default 127.0.0.1 address and replaces it with the IP where ads will be sent
-curl $adListURL | sed "s/127\.0\.0\.1/$piholeIP/" > $eventHorizion
-
-# If the temporary list of ad servers already exists (the eventHorizion)  
-if [ -f "$eventHorizion" ];then
-	# Then replace it as the new ad file	
-	mv -f $eventHorizion $adFile
-else
-	echo "Error building the ad list, please try again."
-	exit 1
 fi
-service dnsmasq restart
+justDomainsExtension=domains
+matter=pihole.0.matter.txt
+andLight=pihole.1.andLight.txt
+supernova=pihole.2.supernova.txt
+eventHorizon=pihole.3.eventHorizon.txt
+accretionDisc=pihole.4.accretionDisc.txt
+eyeOfTheNeedle=pihole.5.wormhole.txt
+blacklist=$piholeDir/blacklist.txt
+latentBlacklist=$origin/latentBlacklist.txt
+whitelist=$piholeDir/whitelist.txt
+latentWhitelist=$origin/latentWhitelist.txt
+
+echo "** Neutrino emissions detected..."
+
+# Create the pihole resource directory if it doesn't exist.  Future files will be stored here
+if [[ -d $piholeDir ]];then
+	:
+else
+	echo "** Creating pihole directory..."
+	sudo mkdir $piholeDir
+fi
+
+# Add additional swap to prevent the "Error fork: unable to allocate memory" message:  https://github.com/jacobsalmela/pi-hole/issues/37
+function createSwapFile()
+#########################
+	{
+	echo "** Creating more swap space to accomodate large solar masses..."
+	sudo dphys-swapfile swapoff
+	sudo curl -s -o /etc/dphys-swapfile https://raw.githubusercontent.com/jacobsalmela/pi-hole/master/advanced/dphys-swapfile
+	sudo dphys-swapfile setup
+	sudo dphys-swapfile swapon
+	}
+	
+if [[ -f /etc/dphys-swapfile ]];then
+	swapSize=$(cat /etc/dphys-swapfile | grep -m1 CONF_SWAPSIZE | cut -d'=' -f2)
+	if [[ $swapSize != 500 ]];then
+		mv /etc/dphys-swapfile /etc/dphys-swapfile.orig
+		echo "** Current swap size is $swapSize"
+		createSwapFile
+	else
+		:
+	fi
+else
+	echo "** No swap file found.  Creating one..."
+	createSwapFile
+fi
+
+# Loop through domain list.  Download each one and remove commented lines (lines beginning with '# 'or '/') and blank lines
+for ((i = 0; i < "${#sources[@]}"; i++))
+do
+	url=${sources[$i]}
+	# Get just the domain from the URL
+	domain=$(echo "$url" | cut -d'/' -f3)
+	
+	# Save the file as list.#.domain
+	saveLocation=$origin/list.$i.$domain.$justDomainsExtension
+	
+		echo -n "Getting $domain list... "
+	# Use a case statement to download lists that need special cURL commands to complete properly
+	case "$domain" in
+		"adblock.mahakala.is") data=$(curl -s -A 'Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0' -e http://forum.xda-developers.com/ -z $saveLocation $url);;
+		
+		"pgl.yoyo.org") data=$(curl -s -d mimetype=plaintext -d hostformat=hosts -z $saveLocation $url);;
+
+		*) data=$(curl -s -z $saveLocation -A "Mozilla/10.0" $url);;
+	esac
+	
+	if [[ -n "$data" ]];then
+		# Remove comments and print only the domain name
+		# Most of the lists downloaded are already in hosts file format but the spacing/formating is not contigious
+		# This helps with that and makes it easier to read
+		# It also helps with debugging so each stage of the script can be researched more in depth
+		echo "$data" | awk 'NF {if ($1 !~ "#") { if (NF>1) {print $2} else {print $1}}}' | \
+			sed -e 's/^[. \t]*//' -e 's/\.\.\+/./g' -e 's/[. \t]*$//' | grep "\." > $saveLocation
+		echo "Done."
+	else
+		echo "Skipping list because it does not have any new entries."
+	fi
+done
+
+# Find all files with the .domains extension and compile them into one file and remove CRs
+echo "** Aggregating list of domains..."
+find $origin/ -type f -name "*.$justDomainsExtension" -exec cat {} \; | tr -d '\r' > $origin/$matter
+
+# Append blacklist entries if they exist
+if [[ -f $blacklist ]];then
+	numberOf=$(cat $blacklist | sed '/^\s*$/d' | wc -l)
+	echo "** Blacklisting $numberOf domain(s)..."
+	cat $blacklist >> $origin/$matter
+else
+	:
+fi
+
+function gravity_advanced()
+###########################
+	{
+	numberOf=$(cat $origin/$andLight | sed '/^\s*$/d' | wc -l)
+	echo "** $numberOf domains being pulled in by gravity..."	
+	# Remove carriage returns and preceding whitespace
+	cat $origin/$andLight | sed $'s/\r$//' | sed '/^\s*$/d' > $origin/$supernova
+	# Sort and remove duplicates
+	cat $origin/$supernova | sort | uniq > $origin/$eventHorizon
+	numberOf=$(cat $origin/$eventHorizon | sed '/^\s*$/d' | wc -l)
+	echo "** $numberOf unique domains trapped in the event horizon."
+	# Format domain list as "192.168.x.x domain.com"
+	echo "** Formatting domains into a HOSTS file..."
+	cat $origin/$eventHorizon | awk '{sub(/\r$/,""); print "'"$piholeIP"'" $0}' > $origin/$accretionDisc
+	# Copy the file over as /etc/pihole/gravity.list so dnsmasq can use it
+	sudo cp $origin/$accretionDisc $adList
+	kill -HUP $(pidof dnsmasq)
+	}
+	
+# Whitelist (if applicable) then remove duplicates and format for dnsmasq
+if [[ -f $whitelist ]];then
+	# Remove whitelist entries
+	numberOf=$(cat $whitelist | sed '/^\s*$/d' | wc -l)
+	plural=; [[ "$numberOf" != "1" ]] && plural=s
+	echo "** Whitelisting $numberOf domain${plural}..."
+	# Append a "$" to the end, prepend a "^" to the beginning, and
+	# replace "." with "\." of each line to turn each entry into a
+	# regexp so it can be parsed out with grep -x
+	awk -F '[# \t]' 'NF>0&&$1!="" {print "^"$1"$"}' $whitelist | sed 's/\./\\./g' > $latentWhitelist
+else
+	rm $latentWhitelist
+fi
+
+# Prevent our sources from being pulled into the hole
+plural=; [[ "${#sources[@]}" != "1" ]] && plural=s
+echo "** Whitelisting ${#sources[@]} ad list source${plural}..."
+for url in ${sources[@]}
+do
+	echo "$url" | awk -F '/' '{print "^"$3"$"}' | sed 's/\./\\./g' >> $latentWhitelist
+done
+
+grep -vxf $latentWhitelist $origin/$matter > $origin/$andLight
+
+gravity_advanced

lighttpd.conf

@@ -1,48 +0,0 @@
-# /etc/lighttpd/lighttpd.conf
-server.modules = (
-	"mod_access",
-	"mod_alias",
-	"mod_compress",
- 	"mod_redirect",
-	"mod_auth",
-	"mod_rewrite",
-)
-
-server.document-root        = "/var/www"
-server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
-server.errorlog             = "/var/log/lighttpd/error.log"
-server.pid-file             = "/var/run/lighttpd.pid"
-server.username             = "www-data"
-server.groupname            = "www-data"
-server.port                 = 80
-
-
-index-file.names            = ( "index.php", "index.html", "index.lighttpd.html" )
-url.access-deny             = ( "~", ".inc" )
-static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
-
-compress.cache-dir          = "/var/cache/lighttpd/compress/"
-compress.filetype           = ( "application/javascript", "text/css", "text/html", "text/plain" )
-
-# default listening port for IPv6 falls back to the IPv4 port
-include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
-include_shell "/usr/share/lighttpd/create-mime.assign.pl"
-include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
-
-# Optional if a password protected directory is desired
-# See http://jacobsalmela.com/password-protect-a-lighttpd-web-server-on-a-raspberry-pi-using-mod-auth/
-#auth.backend = "htdigest"                                                          
-#auth.backend.htdigest.userfile = "/etc/lighttpd/.htpasswd/lighttpd-htdigest.user" 
-#auth.require = ( "/stream/" =>
-#    (
-#    "method"  => "digest",
-#    "realm"   => "lcars",
-#    "require" => "valid-user"
-#    ),
-#)
-
-# Rewrites all URLs to the /var/www/pihole/index.html
-# This should be a blank file (for speed)  
-$HTTP["host"] =~ ".*" {
-     url.rewrite = (".*" => "pihole/index.html")
-}

resolv.conf

@@ -1,3 +0,0 @@
-nameserver 192.168.1.101
-nameserver 8.8.8.8
-nameserver 8.8.4.4