Shortly after I learned you could make Pi Holes and block ads with them, I wondered if there was a way to do this on-the-go. I thought for a while: if I was willing to make a second one and keep it on me at all times with a hotspot and a power bank, then yes, I could do that. But its really not very practical. I'd have to always wear a backpack everywhere I went if I wanted to use this for my phone. It never occurred to me that people already thought of this and solved it. Otherwise I might have simply looked it up. Instead I wondered this for years because this was step number three hundred and forty two and I was still on step two with an available Raspberry Pi on my possession.
At school, there's a security guard for our building, stationed right outside our classroom. He's also in my class even though he's busy starring at a monitor and talking to all the girls getting certified in the medical field, who apparently all have crushes on him. One day I told him that I was making a Pi Hole on Thursday, which is project day each week. He asked me if I was going to make a VPN and I was confused, looking strangely at him, "Yeah, another time though. Why?"
"Cus you can tunnel into network from outside to use the Pi Hole on your phone from anywhere."
With shock, I said, "Ah, now I'm making one as soon as I'm done with the Pi Hole!"
Finally got it all set up and plugged into my network and hoped that nothing stood in the way of my making a VPN. Night before Project day, Doug, my instructor, told me to make sure to bring my Pi and my router. So I made sure to set an alarm and brought them. I didn't actually get to be able to start working on the Pi VPN until there was about maybe an hour left of class so I was feeling like this was going to take a couple project days again. I tried looking things up online to understand it. But I had this feeling if I decided to do it one way, Doug would just have me start over to do it some way he thought was better because it'd be more challenging or something, kind of like what happened with the Pi Hole. I lost count how many times I reinstalled Raspbian on my Pi, but it ultimately caused me to brick a memory card, steel another from a Pi Cluster we had in class that no one was using, and then buy two cards in case this happened again.
The night before I thought, wait, I don't have to buy a second Pi to install the VPN on do I? With the way the installation of the Pi Hole was, it kind of seemed to me that the Pi Hole software sort of took up the whole device. I think it was whenever I typed my Pi's IP address into my browser to access the Pi Hole settings and thought, what does it do if I have other things installed? It's just taking me straight to the Pi Hole interface, it's not even letting me select the Pi Hole deal. So what, if I want to use it for more than one thing, is it just tough luck? Do I need to buy a second Pi for the VPN? My instructor said no when I got to class to work on it. In fact I learned that they work like hand and glove, they were made for each other. What else do I not realize here? Plus there was a question in the VPN installation process that asks if the reason for the PiVPN is to be used with a Pi Hole.
I did SSH to get back into my Pi again using the network username and password I had given it when installing the OS for the thirtieth time last week. I had forgotten how to SSH in so I had to look that up. I thought if I simply typed "ssh" followed by the IP address that it would grant me access, then Doug whispered over my shoulder as he passed by while running to help other students, "ssh username@ip-address". Oh. I entered this and had to try it a number of times, it didn't seem to want to let me in until I realized I think twice that I was using the wrong password, which once again, does not display in the terminal so you think the keyboard stopped working or something.
After I got in, we went to https://www.pivpn.io/. And right there on the first page, if I remember right, was two boxes, one right over the other, one was for installation, and the other was for testing (unstable) branch. I figured if I needed to know, I was soon going to do that soon. But we never went back to it. It ended up that I only kept that page open so I could write this.
Then in the top box with the installation command, "curl -L https://install.pivpn.io | bash", I simply copied and pasted it into the Terminal. My memory is a little hazy here. I think we started it, got most of the way through it, and then maybe accidentally cancelled it so I had to start it over again. It stopped part way through multiple times to ask me questions like did I want to turn this on, is this the right thing, do want this or that, whatever, I forgot all the things. If it helps, I answered yes to all of them except for the ones I answered no to or specified otherwise. Doug said you just hit yes every time and there was something I answered no to. Later I will wonder if this was the cause of a problem I'd have later.
There was a point around the end of the installation while it was asking a bunch of questions and it asked what port number I wanted the VPN to use, which I would configure both Roberts and my routers firewalls to forward. I selected it and recorded it.
After it was done installing, I had this fulfilling feeling like "Yeah, I have a VPN, I made it myself!" Then I realized I basically hadn't done anything yet. My instructor then came by and said that I needed to do this, this, and this. I wrote it down after the third time of him listing ten slightly complicated things that I actually didn't understand yet at all. Classmates were looked at me. Well, one of them. But you have to understand that you're an idiot about something until you're not. I finally had my epiphany that allowed me to wrap my head around what he was saying but for good measure, I wanted to write everything down anyway because I knew I'd get home, forget a mundane detail and the whole thing would suddenly make no sense.
But I got home and immediately got to work on it and asked my roommate, Robert, to locate my router on his routers network and set up port forwarding on the port number that I selected during the installation process by the installation process of the PiVPN. But I hit a brick wall. I just keep seeming to run into these things. It's like I am experiencing every conceivable problem that can be had in the making of a Pi Hole and a PiVPN.
So when I got home I thought I understood Doug's instructions, but life happened. And only after life was out of the way again and I could try to work the problem again, everything he said slipped back into obscurity. I took notes on what he wanted to do and why, but I didn't foresee the confusion I would later have over my the notes regarding something that I didn't understand. I had new realizations that then confused me later. Things similar to, wait, if this, then why this? Did I misunderstand Doug?
At least the part I kind of understood that absolutely couldn't be done in class was sort of understandable. Since my roommate has a router and I now have a router, and mine is configured to work with my Pi Hole, now I needed to create this port forward on both my routers or else this VPN will pass through my router and get blocked by his because ingoing and outgoing transmissions that would go over the VPN would be blocked by his routers firewall. We created the port forward for my router in class, and ny router had to be connected to his routers network in order for my router to be found on his network so the port forward could be applied to the specific device that it was meant to accommodate. It would be pretty unsecure to create this port forward and allow any app inside or outside the network to access the network through it. To plug any potential hole, you create a port forward for a specific app so certain traffic goes through that port only and not only doesn't get blocked by either his or my routers firewalls, but also this port forward has to be specified to only be used with my router on his end so that nefarious outside devices don't take advantage of a gaping hole in the firewall.
I went into my Asus router, and for my router, I could go to the WAN selection under Advanced Settings > select Virtual Server / Port Forwarding > Enable Port Forwarding under Basic Config > and then below that under Port Forwarding List, was an Add Profile button so I could select what device it was for, what the external port was, what the internal port was, what IP address this port was going to work with, and select the protocol it would be using, either TCP or UDP or both, in which case I want both. One is Transmission Control Protocol and the other is User Datagram Protocol. TDP ensures delivery of data and messages over a network and UDP allows data to transfer over the network without having to verify delivery. You want UDP to be able to skip verification because this takes time, which would otherwise cause more gaming and video streaming lag than there already is.
These ports refer to the routers firewall. Only certain traffic is allowed through these ports, which can be used to block any number of different kinds of internet traffic from being able to block social media sites, video streaming sites, porn sites, and provides a measure of security from the internet to simply access the network--certain traffic has to be allowed aka whitelisted so that applications will function properly. But firewalls also block many types of attacks because they filter all data that moves through the network and certain ports in a firewall are used for certain things. Imagine a hole in a dam that's a certain size, debris, water and rocks of only such a size can flow through it, anything larger for example would be blocked from passing, although a network firewall doesn't get clogged when something too large tries to go through, and these firewalls don't filter according to file size but rather other attributes and many rules are applied to them to filter certain traffic or even keeping certain users from accessing certain parts of a network.
I had a bonehead problem and it took me all day to solve it. I lost a whole day off to it. I needed to get my roommate to create the Port Forward on his router, and so when he did this, he sent me a list of all the devices on his routers network, even listing the individual devices I had plugged into my personal router. I saw my phone, my old laptop, my new laptop, my desktop, but I didn't see my router anywhere. Nothing that said Asus on it. There were a whole bunch of random devices with names that clearly identified them on the network and they all had an address next to them. I assumed this was an IPv6 address because it had several segments of hexadecimal characters.
So I spent the whole day trying to ascertain what my routers IPv6 address was. I asked google how to look it up, I asked google what it was if it knew, I asked ChatGPT how to find it, and when I did finally figure out how to get it, it didn't match anything on my roommates list. I thought maybe this was because I found the option under the IPv6 button on the side panel to activate it. I did so and this required a reboot from the router and a progress bar that it was completed, at which point the router froze and I had to risk resetting my router if when I unplugged it for a minute and rebooted that it didn't work properly, and I really didn't want to have to reset it because there has been so much configuration that's taken weeks to do and this would wipe it out instantly.
Then my instructor texted saying that I didn't need to activate IPv6 to do this. He said something to the affect that it wasn't the best way to do it or something. But finally around midnight I was about to go to bed when I just happened to see on the main page of my router what its MAC address was, and realized it was quite similar to all the addresses next to the network names of all these devices on my roommates routers network. Everything hinged on being able to select my router as the device this port forward was going to be used for, we simply couldn't move on to the next step without doing so because this is the reason why firewalls work in the first place.
So I matched my routers MAC address with one of the unknown devices addresses on my roommates routers network and I texted him asking if when he finished his nightshift, could he finish the port forward. He texted three minutes later saying it was all done and I was confused. He was at work, it was midnight. He wasn't here to do it. Then I realized he was using an app on his phone to remotely access the configuration settings of his router and he simply applied the changes and used the firewall port my instructor and I decided to use.
When picking a firewall port to do a port forward with, you want to pick a number between 10,000 and 60,000 because any port lower than that is mostly spoken for.
So it was done, I was so excited. I actually thought that I was done with the PiVPN for the second time in this project and that everything should work now. I thought this project was finally over and I could move on with other things I've always wanted to do and didn't have a clue where to start. A few days later I realized I was sorely mistaken. I still needed to add my phone and my laptop to the VPN and the WireGuard VPN app had to be installed on my phone and the program on my laptop, then they had to be linked through the terminal on the Pi hole VPN to make it work so the VPN would be working with the proper devices. Plus the tunnel just isn't complete without something to create a tunnel for. The VPN is ready to work but there's only a tunnel once a device is connected at the other end.
One of the things my instructor wanted me to do was to do the "pivpn -a" command in the terminal to add a device. I spent so much time trying to solve the previous problem that I completely forgot about it.
And nearly a week later when my instructor said that I needed to bring my router into class again to get this port forward and everything finished. Then I had the idea to remote into my desktop at home from my laptop in class, and then SSH into the Pi from there. I thought for sure my instructor would go for it because it adds another layer of difficulty. I would have to use an application like Anydesk or something on both my laptop and desktop, and make sure my desktop stays running while I'm in class because I didn't want to change any sleep settings, I could run a video on loop in VLC player, which almost always overrides any sleep timers or screensavers, except in the case that its time to update VLC player. Then VLC intentionally allows the screensaver to interrupt the video.
I told this to my instructor and he loved it, all configuration could then be done. Alternatively I could come to class for the first hour, go hone and call him on the phone and he could help me that way. He liked both options but I liked my remote desktop idea so I prepared to do this.
I hadn't used any remote access applications such as Anydesk or Remote Desktop in a little bit so I forgot that you have to create connection request on one device and accept it on the other device you're trying to connect to. No one was home when I was in class and it was go time, so I asked my instructor if he would allow me to run home and accept the connection, which he did, I ran home and in about twenty five minutes I was back in class and had started the remote connection successfully.
Doug, came by and helped for about five or ten minutes before being called away again, so meanwhile I was doing research on the next step and also on initiating remote connections without having to accept the connection on the other end. I did find a solution to that. But then the remote connection ended itself automatically due to inactivity. I was so bummed. Next week I guess.
I had a bunch of time to figure out what I was going to do about this auto disconnect due to inactivity and I had looked around on Anydesk to see if there was an option to start a connection from one end of the connection. There was.
My research pointed me what is called unattended access. I found that under the equivalent to googles hamburger menu in Anydesk located in the upper right corner, there's a button in that menu that says "set password". This brings up a window to set a password and confirm it, and after this, you get a window of a number of options including enabling two-factor authentication. I think I also had to check one of the boxes on this page to allow an unattended access remote connection, so when I select my desktop from my laptop remotely, and I enter the password, it will grant me access to my desktop without any further ado. I chose to activate two-factor authentication which also required me to download an app on my phone that offered up time-based one-time codes, which recycles about every ten seconds or something like that. So this would be extremely hard to brute force your way into.
I should add a few things, when you get to this menu to set up the unattended access, don't forget to scroll down because there's quite a few check boxes in this window and you could miss them if you don't realize most of them have to be scrolled to. Plus, I think before my desktop allowed me to create this unattended access, I think I might have had to go into settings and turn on unattended access because I remember opening this window and the new password box was grayed out and couldn't be typed in.
Now I was ready. I told my instructor about this new development with Anydesk and I was hoping we'd finally make some progress after two weeks of stagnation. It was also kind of discouraging me that on top of not finishing this project for over a month and a half now when I thought it'd only take one class or two at the most, I needed to take a test that I was unable to take, I was having problems with insurance, I was having trouble at work, I had so many projects I wanted to do and couldn't seem to ever get to them, but on top of all that, the Pi hole seemed to stop working, so I had a non-completed VPN with its own issues that needed troubleshooting and then the Pi hole itself had problems too.
Apparently it was still blocking ads, I went to a Pi Hole tester website that threw a hundred and fifty ads at my Pi Hole, and six were blocked. My instructor said to update the gravity and that we needed to add to the ad list, so I tried updating gravity and this did nothing. I thought I needed to update my Pi Zero, so I did, I thought maybe nothing takes affect till you reboot so I did, and none of that did anything.
I even tried to solve this on my own. I did research and found a terminal command, "pihole -g --status". I tried it and saw a number of upgrades that errored out. My research indicated that I needed to look into each one individually and I thought at this point, I already don't know what I'm doing and so to dive into a sublevel of things that I was already oblivious to at the surface level would only serve to confuse. I might go back to class to have Doug say I wasted my time. He'll of course say that it's all good stuff to learn but it was unnecessary for this project.
Then Thursday, project day, finally came again.
2023-09-22 - update
I went to class yesterday. It was a bit different, many students didn't come to class. They did tell us at the beginning of the term that we will either love project day or hate it. So Doug was able to focus his attention on my problem way more than usual. I made sure before I left my apartment to open Anydesk and leave something playing on a loop in VLC player. Now in class, I simply opened Anydesk, accessed by device, entered the first password and then the second, and I was in my desktop. I was so thrilled! Compared to last week this was a cinch. I opened PowerShell on my desktop back home and SSH'd from there into my Pi Terminal to complete everything.
Doug was surprised my Pi Hole wasn't working even though it initially worked great. We checked the amount of Domains for adlists. There was something like one hundred thousand and twenty three. I didn't know any better, that could be great for all I knew. He pointed at that and said, oh yeah well we need to still add those other adlists and update your gravity. I told him I already did that and even updated and upgraded the Pi and rebooted it. But that did nothing to fix the problem. I think at this point he probably said, "Well you only have a little over a hundred thousand adlists."
Then we updated gravity again just for good measure. And this is where Doug showed me some other adlists for my Pi Hole to block ads with. We updated the ad list from GitHub using Blocklist Project. I went over this part already in part 1. Doug explained to me that I was only using something like 200,000 blocklists and that wasn't that many, and then after installed the Block List Project, Then I head nearly a million. And that is about how much I want to have. If I do any more than that, then there starts to be a sort of negative effect, not only do all ads get blocked with the more adlists you include on the Pi Hole, but they'll start to block regular internet traffic too. So it won't catch all of them but it will catch a great deal of them, maybe most of them.
It will unfortunately not block any YouTube ads because a few years ago when this got popular enough, YouTube decided they were going to circumvent Pi Holes by internally linking ads to their videos so the ads load as if they're apart of the video and the Pi Hole cannot recognize that. The ads must come from an ad server online that is recognized by the adlists. This isn't quite the end of the story, I have been using AdBlock on my desktop and laptop for YouTube for many years now, and they catch everything on Youtube. I've also started using the SponsorBlock plugin as well, which blocks even those parts of videos you might commonly skip, as though they weren't even there. You can still watch them, the plugin highlights them on the timeline of the video.
But while we were adding adlists from Blocklist Project, I learned you could download lists for not just more ad protection, but for smart tv bloat, porn, abuse, drugs, phishing, torrents, ransomware, malware, etcetera. I took some of this stuff too but Doug recommended against downloading very many because I am only running a Pi Zero, he can run much more because he's using a Pi 4. He showed me his pi hole interface from home.
Now that we added more adlists from GitHub, there are now nearly a million domains on adlists.
When we had initially finished installing PiVPN and were finally ready to add users to the VPN, we typed "pivpn -a". This is where you name the user. I got confused about this and thought that I was actually connecting the device with this command, so I went to find the network name of the computer I wanted to add and learned that this did nothing. Now I just and a really stupid name for a VPN user. But this was a later occasion when I had to do this again and couldn't remember how we did it the first few times. And of course I forgot to wrote it down so this is actually future me beaming into the past to add to my past selves entry.
Doug sort of took the reigns for a bit here and did a whole bunch of stuff really fast so that's probably why I didn't write any of it down because I didn't know what he did and then I was going to address it later but there were too many other things to take care of and it never got done till now. So, clipping my hands and rubbing them together, sighing, and here we go.
Doug typed the pivpn -a command and named the user for my phone and had me install Wireguard on my phone. The command line asked if we wanted a QR code for a phone and he typed the requisite command to do so. We added my phone to the VPN and created the tunnel. And at first it seemed to work. He then created a full tunnel, which I explain later. It was the end of class and for some reason the I didn't get adblocking on my phone or any internet on my phone either. But there was no more class time so we had to leave it for next Thursday.
We did get a lot of stuff done, everything was installed for Pi Hole, PiVPN, Wireguard on my phone and laptop. But for some reason there was an unidentified problem with my Pi Hole and another one with the Pi VPN.
The next week, we logged back in through Anydesk on my laptop to my desktop with unattended access, and it was now a simple effort connecting without driving home to accept connections. Then we would SSH into the Pi through my desktop at home.
Doug thought for sure my roommate didn't create the port forward correctly but I showed him that the port forward was created, on both routers, using the same port, for both the internal outfacing ports and the external inward facing ports, and were configured to use both UDP and TCP. I had proof, and showed him the screen shot my roommate sent of his router configuration.
And then I showed him my routers configuration, which showed the same data but in a different way.
It just didn't make sense that when we turned it on for my phone that my internet was nonexistent. We did research to see what else the problem could be other than just updating gravity and updating and upgrading the Pi Hole, and while I did that, Doug just quickly reinstalled the VPN again for the third time overall since the project began.
So just to recap since this is important for how we connected my laptop to the VPN, it needs to be understood that we were connected from my laptop to my desktop back at home through Anydesk, and then used my desktop at home to SSH into my server there on the same network. This is crucial information for how we connected the laptop. We had reinstalled the VPN and probably Wireguard on my phone as well, and we updated gravity on the Pi Hole but also added the GitHub adlists and this brough the adlist count up to nine hundred thousand something I think. . After this, we did the pivpn -a command. It gave the option of using a configuration file with a key to connect any device, or just a QR code. We did the QR code for my phone again and this time it worked. There was just a simple error while installing and it was corrected y reinstalling it. Now when connected to my VPN, instead of just bytes of data transferring and no real internet connection and no adblocking, it was blocking ads, and it was transferring data in kilobytes and there was internet.
This initial VPN tunnel was a full tunnel, and Doug created a second tunnel known as a half-tunnel. I explain this later. And then for my laptop, we were already connected to my desktop at home on my network through Anydesk, and my desktop was connected to my pi through SSH, so when the command line offered the configuration file with the key for the VPN connection to that new VPN user, we simply opened it and copied it to my laptops clipboard, and then pasted it into Wireguard on my laptop and created a half tunnel and full tunnel with this key. (I write another post on how to do this the more traditional way for any device that isn't a phone and can't use the QR code, but I didn't run into this until late December and had to figure it out on my own so I did and wrote another blogpost on that more traditional method where I had to install a utility to transfer that key from one computer to another using SSH on the same network, but it does the SSH and command line commands for you in the background and you simply use this utility on the GUI to do it the way many would consider the easier way if scared by the command line. I am a technical person so once I had done both, neither of them seemed any harder than the other.
Now to alleviate confusion, we didn't need to create a VPN connection for my desktop at home because the whole reason for using a VPN is if the device you're using isn't in the physical network you want to use it in. So to securely do this, you use a VPN tunnel, a virtual connection to that physical network that is encrypted. This is unnecessary for my desktop which is already there inside the network so no VPN user and no tunnel necessary.
To explain the full tunnel and half tunnel, the full tunnel causes that every last bit of your connection to the internet and your private network to pass over the VPN between the physical network and the devices accessing it. The down side of this is that the internet is slower and you don't need to make queries online from your phone to go through your VPN back home to then be made from your private physical network and then sent back to your phone over the VPN. This is overkill if all you want is adblocking and things like that, or you're using your phones data plan so it's using its own secure network to the internet through the phone company, so you can access your private network through this as well. For that reason, you create a half tunnel and that way only access to the private network and DNS requests and things like that will go over the VPN. This is done by simply changing what IP address ranges the tunnels use. _________________________________________. So if a half tunnel is so much more efficient why would you ever use a full tunnel? The full tunnel is far more secure than the half tunnel. But it's not always necessary. The kinds of times when you definitely want to use a full tunnel are times lie when you're at the airport or some other public network where your private physical network back home could be hacked into by the tunnel you're using on an unsecure network with other devices whos' users may have bad intentions.
2023-10-24 - update
I started to notice today that ads were not being blocked on this one game I play on my phone. I thought, well its been around a month or so and I haven't updated gravity yet so I accessed my Pi Hole interface through my browser over the VPN. I navigated to Tools on the side panel > Update Gravity > I pressed the big, long, blue Update button at the top of the page. This took about a minute or so.
And then at the bottom of the page, I discovered a button flashing white and red that said Pi Hole Update Available or something like that so I inquired about just pressing this button to update and learned that it just leads to the location of an update although I don't know this because I didn't press it.
I decided to SSH into my Pi Hole. This took me some time though because I couldn't remember the IP address for my Pi Hole, which is also my DNS server, so I entered ipconfig into the Power Shell window before entering an SSH session and I didn't see an IP address I recognized. So I did an "ipconfig /all" command. I also didn't recognize it there. Then I checked my WireGuard app on my phone and didn't find it there and then checked the bookmark in my browser for the Pi Hole interface, and then I found it.
So I typed the network name of the Pi Hole with an @ symbol followed by this IP address, all one word. Then I typed "pihole -up", which updates the Pi Hole software. This took several minutes. And there were many times that it seemed to sit for some time and so I thought it was done just to discover that all of the sudden it started updating another thing and added lines to the page and kept scrolling down and down with more processes. And so I started to wonder how I would know that it was actually done. I researched online and found something that when it was done, it would the command line would return to a normal prompt, which in my case suddenly happened. It displayed a green "Pihole1@pihole" which it always shows when ready to accept a new command.
There were two other indicators too though, one would show in the Pi Hole log file, which you would access from the command line, would show a message indicating a successful update. The other was that the Pi Hole website would show the latest version of Pi Hole.
Up next for large projects, a Fileserver running an old motherboard, with Linux Ubuntu Server, using RAID 5.
This has been Truncat3d 00000000111100010100110______________end of line
