My Instructor, Doug, has moved on to take his IT position at BYU full time. The new instructor is a former graduate of the program I am currently doing. So while he is knowledgeable about what to learn for these certifications, he has no experience really troubleshooting anything. I like the guy but he will not be terribly helpful in practical applications, although he is going to try. So I am on my own for the most part.
I think it was something like Monday this week (now Thursday) that I discovered I could not connect to my VPN. Over the past 14 months since the server has gone online, I have experienced several incidents where the VPN just didn't want to load and I had to restart the service I think at one point on the server, or on most occasions, deactivating the tunnel and reactivating it solves the problem, or SMB just didn't want to function because I guess having a persistent SMB server icon in This PC in Windows and using NET USE /DELETE and then reinsertion of credentials can cause errors sometimes. Often times a simple reboot of my windows computer solves this. Other times I have to just use NET USE /DELETE and reboot the Windows computer, and then there was one time where even that wasn't good enough but I now don't remember what was done that did finally fix it. Another time there were several blackouts because during the summer, there always has to be blackouts, and they always have to be in the area I live in, and I obviously leave my server running for good reason so when instead of a blackout, we had a brownout, which according to Doug is worse than a blackout, this caused my server to forget boot order and I couldn't boot my server.
However, this time, I started with deactivating and reactivating the tunnel--nothing happened. I did a NET USE /DELETE, and I was specifically trying to remove the icon that is persistently left behind after you map a drive and the terminal asked if I wanted to turn persistence off. I had to look up what that meant because maybe it was part of the problem and this is where I learned that when I map a drive, this can stop it from leaving a server icon in This PC and consequently, the shortcut icon I have on my desktop of every PC for easy access. I didn't know this was a feature, let alone that it could be turned off. And so since all I was trying to do was remove the mapped drive, I selected no, but this caused an error message to appear saying that there was no drive to delete. I eventually found that I could go to the icon and right-click for context menu and select disconnect. I don't remember if it errored out again but I think it did, but after a reboot, if I recall, it then did finally disappear. I was also hoping a reboot would solve whatever the error was. No.
I kept trying to open my server and it not working, I then started paying attention to whether or not data was both sent and also received over the tunnel, which is shown in the WireGuard Windows application. But time and again no matter what I did, 148 bits would be sent and no data would ever be received. I switched tunnels back and fourth, rebooted the laptop multiple times from class.
I was not at home, I couldn't SSH into the server to troubleshoot on that end because the VPN wasn't working; the thing I needed in order to connect to it to troubleshoot. By this point I had concluded that either the server wasn't up and running, or SMB or the VPN weren't working or the WireGuard app wasn't working. I tried it on my phone, it didn't work either. Just to make sure this wasn't the class WiFi or something like that, I tested it a few times, and made sure I was only using data on my phone. But that didn't work either. My mom reported that her connection was down too.
For this very reason that I may need to fix something without the benefit of SSH, I have Anydesk set up with unattended access on my desktop at home. I entered into it, I pinged the server, 4 packets sent, 4 received, 0 lost. Okay, so the server is running. I opened files on my server through my desktop. Videos played without any problems at all. So the server is up and running and obviously, I could only access files if SMB was working on the desktop. If my phone and my moms laptop were also unable to remotely access the server, then it must be the VPN, WireGuard, or some other thing that I couldn't yet think of.
I accessed Anydesk and got into my desktop, I SSH-ed into the sever just fine, I restarted the VPN service. I entered:
sudo systemctl restart wg-quick@wg0
I checked the WireGuard app' no packets received. I then commanded the server to reboot and then SSH-ed back in again. Checked the WireGuard app. No packets received. I switched tunnels back and fourth and deactivated and so on, rebooted the laptop. Still nothing. I turned to my new instructor and asked him if he knew anything about VPNs, he said no, not much. I texted Doug. I do still have his number. I asked him questions but he either didn't respond or he didn't respond quickly. Either way, talking to him didn't cause me to be able to fix the problem. I had to go home for the night. The next few days I weighed my options. I seriously thought, well, I can just reinstall WireGuard and Pi VPN. If that doesn't fix it then I am really in trouble. The problem with this option though, is that even though it has a high likelihood of resolving the issue, I would have to reconfigure a lot of stuff.
I had recently gone through some troubleshooting stuff with ChatGPT and the problem was something that ChatGPT did not mention. In fact it mentioned a few common things to troubleshoot and then went straight into a number of less likely scenarios. I don't know if it is just a lack of optimization, but I asked ChatGPT if at the beginning of every troubleshooting session, could I tell ChatGPT what I struggle with in troubleshooting so it can optimize hat suggestions it gives for troubleshooting. It did admit that the problem with the last thing I troubleshooted was a very common problem and so I used that addition to springboard into whether or not ChatGPT could optimize somehow to help me and it said yes. I told it what my problem was and asked fi I needed to remind it every time and it said it will just automatically optimize for every time going forward if I desire/. I couldn't believe, you can tell ChatGPT what kinds of answers you want it to give. I don't mean those times where it asks you which response you prefer, I mean you can simply tell it what kinds of answers you want and it will abide by that, for the most part as far as I am aware. I already knew I could tell it to give shorter answers so I don't have to read a book report when I am just looking for yes or no answers or something, and it will do that. So I tried it and it worked.
The problem I was running into around a year ago whenever I tried to use ChatGPT and try not to go for Doug's help was that it would give me all these dense solutions that I didn't understand and so I would just copy and paste commands it would give to try out. Doug said this is a bad idea, that I need to know what it is having me do. I struggled this whole year to do that less. I finally know enough about this kind of thing that I don't have to look up every single thing under the sun in order to know what is going on. If it is a CURL command, I know something is being searched off the internet like an installation file for example. If it is SUDO, it is elevating permissions, if it is APT or APT GET, I am installing something, I know enough of the basic commands that this is less of a problem now. So I really just have to look up specific things that I do not recognize.
Then I told ChatGPT about how I don't know enough about common problems and ChatGPT will often go right into things I don't understand and so I will explore them, but then those things don't solve the problem either so I consult with ChatGPT and it tells me well in that case it could be yet another layer deep of problems which I also don't understand so I have to try to wrap my head around those things too. I will often get three layers deep, not solve the issue, having seriously considered whether it is safe for me to do this, and then lose my nerve and not do it because this is too dense. If in order ot understand a concept or solve a problem you have to completely learn a new thing from scratch, and then that thing requires you to learn a whole other thing completely from scratch too, and then a third layer of something you don't know from scratch, you get overwhelmed very quickly. This is a recipe for solving nothing and feeling out of your depth and like this is just too complicated for you to solve, even if the problem is as simple as just reinstalling something. When I was first installing Pi VPN, this is the kind of thing that happened and I tried to solve the problem without Doug's help, just to have this exact scenario play out. I told this to ChatGPT and it said it would optimize for only the most common things first and only after that it would go deeper. There were other things I asked it to do as well but I don't remember what they were.
I had a conversation with Doug about a month ago now about what do to when this multi layer in-too-deep problem happens when troubleshooting without help. He suggested that I only go so far and then abandon it.
It just so happened that when I initially took to ChatGPT to solve this problem, one of the possible causes of this problem which it mentioned was that my ISP may have changed my IP address. I immediately ignored this because I had never heard of such a thing and thought this was likely a less common scenario.
Over this week, I told another friend of mine who has a lot of IT understanding but is more of a programmer, who he has a VPN of his own which he has troubleshooted, and he said that sometimes his ISP changes his IP address and he ahs to reconfigure everything. And then he said as a matter of fact, they just did it like a week ago. I asked hi who his ISP was. Mine is Google Fiber. It took him a few days but he confirmed that it was Google Fiber. He only lives a few blocks from me as the crow flies. So I immediately started to wonder if this was actually the culprit after all.
I was still weighing my options about reinstalling everything. I had been putting off further troubleshooting because my time is precious and I already don't have much time to study and I am about to take Network + again this next week.
But today is project day in class and I thought, I really do rely on my server quite a bit. And if I keep putting this off, I will just get used to not having a functioning VPN and not remember what I had done and have to start from scratch, which will cause me to then think maybe this was the problem or that, basically I will get further confused over time. I need to fix this while it is still fresh.
I started troubleshooting shortly after arriving in class and kept turning to classmates and asking if they knew anything about VPNs, and none of them do. I am starting to suspect that out of all of us, I am the most seasoned on all the various different things that people are pursuing. Even more than my instructor. This is not very comforting when you are frustrated and stumped and there are options you would far prefer to not explore.
I started to wonder if there was a way to confirm if my P address had changed. I used ChatGPTs help to navigate through the server, while SSH-ed into the server through my desktop at home through Anydesk at school. It took some time. I went to google and asked what my IP address was, and I kept that open for future reference. I wondered what IFCONFIG would tell me, but apparently, it didn't tell me anything I didn't already know. It confirmed the IP address my Google search just gave me. ChatGT had me use CURL IFCONFIG.ME, I don't know what the difference is between this and just plain IFCONFIG, but I tried it because I was hoping maybe it would tell me something useful like that by the way your IP address a week ago was this other thing. It didn't.
It told me to find the WireGuard/wg0.conf file and have a look at it. I believe somewhere in there I may have asked it if there was a way to see what IP address the tunnels on each device were set to use. If it definitively showed a different IP address, then I could just make that one fix and everything would be fine. But ChatGPT told me that to navigate to this file, I needed to type /etc/wireguard/wg0.conf so I typed in all different versions of this without the first forward slash, with SUDO, without wg0.conf, SUDO again, it kept telling me there was no such command or whatever. It returned errors I didn't understand and so I would copy them through Anydesk and paste them into ChatGPT and it told me the translations. One of the translations was simply that I must have typed the command in wrong, and it said this in a very unintuitive way that made it sound like there was no such command and showing proper uses of SUDO and stuff like that. That's when I dropped the file from the path I typed. But it kept telling me I didn't have proper permissions after I had gone to ChatGPT enough for translation. I asked ChatGPT how I can find this file if this command won't retrieve it.
I thought this was a strange problem to suddenly start having, to not be able to reach this WG0.CONF file, because I was able to reach it Monday somehow. So I kept trying to solve it. Finally, ChatGPT suggested trying:
sodu -i
I looked this up to make sure I wasn't uploading my private and preshared keys to the internet or some crap like that. Google explained nothing and gave me links to websites talking about ROOT. Yeah, but I want to know exactly and I don't want to go on a wild goose chase before I discover it. So I opened another ChatGPT window and just asked exactly what this command does. It explained that it raises privileges so I can access this file I explained in a separate instance what I was trying to do. It made it look like it was going to CAT everything to the screen, and I thought, well, I don't need to NANO the file just to see if the IP address is wrong. So after doing this, the color of my user changed and again I entered:
/etc/wireguard/wg0.conf
It worked. ChatGPT said that when I am done be sure to type exit so that i relinquish root privileges.
I opened the file. I looked at it and recognized all the users I had added when typing:
pivpn -a
I found my laptops tunnel name and looked at it for a while. It was still like a half hour that I was looking and thought I found the problem and then being confused because I then asked Doug, hey I need to ask someone who knows something about VPNs and no one here knows anything. I told him I thought I found the problem and I told him I noticed the preshared key looked the same but the other keys looked different when compared to the tunnel info in the WireGuard app on my laptop. I had the .CONF file through CMD in Anydesk and the WireGuard app on my laptop open right next to each other, and he explained that the public keys are supposed to be different. Don't change them. I got confused and spent another half hour resting my brain and getting myself confused because I thought ChatGPT said to edit them when Doug said not to.
I called my other friend who also has a VPN but he didn't answer. I kept asking people if even though they knew nothing about VPNs, could they look at this and tell me if I was crazy. But then someone would tell a joke and they would get sidetracked and I would just give up because I didn't know what I expected them to tell me.
I had a moment where I looked at the IP address and it looked different, it even had a CIDR notation and the one in WireGuard said something like XX.XXX.[wrong number].[wrong number]/32 and the new one said something with two new last octets with a /24..
I kept getting confused about the ENDPOINT. Doug said if I just change the endpoint then it should work fine. But I kept trying to make sure I understood what the endpoint was, because it never said anything about endpoint in the .CONF file and it would in WireGuard and this seemed intuitively that it is the IP address of the computer at the endpoint of the tunnel. But for some reason having something to do with the endpoint not being mentioned at all in the .CONF file and feeling like I read something from ChatGPT saying that you needed to figure out what the endpoint of the device in the .CONF file was by checking this against that and I kept trying to remind myself that I understood what the endpoint was. So this confusion kept slowing me down and then confusing me even more because then I went back to confirm that the IP addresses were not matching and I looked accidentally at allowed IPs and for some reason I only say the first two octets and had not registered in my mind that it was the last two octets and the CIDR notations that were wrong and so when I saw the correct first two octets and then saw that it was allowed IPs and tried to find the endpoint again to confirm, I had a positive feedback loop of confusion until finally I decided to copy the configuration in WireGuard down and then just try what Doug had said, which I would have eventually done anyway if Doug had not texted back.
But I probably would have also changed the public keys and stuff if I didn't decide to just take his advice. I confirmed that the server public key that was different from the client public key was actually located in the edit window of the tunnel configuration, below a dropdown menu showing the name of the tunnel, and it matched the public key of the server and I looked below at the client public key and realized, okay, WireGuard is aware of the servers public key, I don't understand why so may different keys are needed because there's a public key, a private key and a pre shared key and then the public key on the server is different from the public key on the client. So there's 4 keys in total. I had asked ChatGPT more specific questions.
I didn't get around to following the instructions because I just decided to try what Doug said and if I read on, ChatGPT probably would have said basically the same as what Doug said.
I made sure to completely read the IP address for endpoint and compared it to the IP address of the server in .CONF and they were different, the last two octets and the CIDR notations were different. I knew if I checked other tunnels and other devices that they would likely match the wrong IP that was in the laptops WireGuard app. I changed it, I saved it, I activated that tunnel, and it took a second, there was a bit of a hiccup, but it worked. It had then transferred a few bits and received something like 6MB for some reason. I remapped the drive and went on the server on my laptop directly and tried to open a file and it worked. I changed the other tunnel because I use a half tunnel and a full tunnel. It also worked. I Anydesk-ed into my moms laptop and called her and changed the IP address in her app too and it worked. I changed it on my phone for both tunnels and it worked.
There is one thing that I think is very important to mention at this point. I had moments this week where I was contemplating just reinstalling the VPN and the WireGuard apps on all machines which would have also necessitated lots of configuration I did not want to do. And as I learned that my friend, Robert, has a problem with his VPN every year or so with Google Fiber re-assigning IP addresses and I started to really wonder if that could be it and how I would confirm it and having asked him who is ISP was because if it happened this last week, or Monday or whenever, then it is likely what happened to me also, I had moments where I thought that if I did just reinstall everything, I would miss out on the opportunity to learn something new. I would not know what went wrong. I would develop a little more confidence that I could solve problems without help but I wouldn't now what went wrong and certainly not how to fix it or determine if that was the problem in the future.
What I learned from this is that ISPs will occasionally change your IP address without informing you and that it will happen in the background and this really only affects you if you use a VPN or something like that in order to access a secure network remotely and access files. So this is not a thing in regards to using Nord VPN or whatever just to secure your browsing. If you have a self configured VPN then this will likely affect you. And the reason ISPs do this is because occasionally they install new infrastructure throughout cities, MANs, and when they do so, to keep everything optimized, they will reorganize what infrastructure supports which customers such as me. So on the ISPs end, you could be on a whole other network where a new IP is necessary. And this also includes all sorts of things such as load balancing on their end, new hardware being implemented and so they will move you around on the network infrastructure to keep your internet speeds optimized.
However, take that for what it's worth. Doug once told a story about a year ago, where I told the class about my frustration with a certain ISP I dealt with ten years ago, when a family member I lived with died and I had to move, and my ISP wanted to charge me money to move my internet to the new address. I had no money and they were already charging a lot of money for a terrible service. And without warning that the price you were paying for the first three to six months was actually a discounted price, and would skyrocket from 35 to 65 dollars each month even though it wasn't even broadband, it was DSL. And it was substandard. And I was telling the person on the phone that was happening me that I didn't have that kind of money. I remember her specifically telling me that if I asked her to transfer me to the department that deals with customers ending their service, they had more negotiating power than she did. Her department specifically cannot negotiate deals, discounts, or extra fees. But if you decide you're going to leave then the other department that deal with it specifically has the negotiating power to do whatever they can to keep you and she told me to tell them I am leaving because it's too expensive and because they were charging me for moving my service to a new address. They removed the charges and I was stunned that the lady I talked to first was right. SO I did that again a year later when I again moved. And I told another story then about the same internet at the new address, how it was terrible, the guy would set it up and turn it on and leave after verifying that it worked, and it would work less and less well over the course of a half hour before cutting out entirely and I had service men come over to fix it like three times and went through all sorts of trouble to get their updated DSL modem that was supposedly surely the problem and then talking to people on the phone who would say that my area where I live is far away from the main distribution center, whatever they called it and so speeds like this are to be expected. And I didn't have the courage to tell them, so why do you offer service if there is no internet? Why am I paying you for nothing? I got so frustrated that finally called them and said I am leaving. This problem keeps not being solved and I am till paying you. Either it's fixed and there are ni problems going forward or I am gone. You might as well end it now. They came and fixed it and finally the problems went away. So Doug told a similar story where with his ISP, his speed suffered at the house he lives at. And so he complained and they gave all the usual runarounds, but whatever leverage he used, he got them to move him back to the top of the optimization list. I can't remember the specifics but there is a way they set everyone up, where even though everyone is paying for 1Gb internet, not everyone is even getting most of that. And after he confronted them about it, it was fixed. He said he has to contact them once every year or so to get his internet speed adjusted back to what he is paying for and it slowly slides back down until he has to call them again.
Problem solved. I couldn't believe it! Now for the other problem where my moms user account seems to have gone on the fritz. I think I renamed a file on the server through Windows that I forgot had a symbolic link on the Linux server associated with it going to my moms user account. And so I need to finish mapping out my server so I can locate things using a graphical tree like structure to understand my RAID 5, Pi VPN, Pi Hole, RSYNC Cron Job, scripts, Files, Home Assistant and so on. When I SSH into it, I only ever see Home Assistant files and folders. So if I can write a script in Python that will map out the contents of everything with tips on how to reach that directory and edit and execute each file, I think that will make it less confusing.
Problem not solved. A week later, ISP changed the IP address again. I suspect they changed the IP address back to what it was before but I can't be sure because there is no way I can find that shows previous IP addresses. But after only one day of troubleshooting, it came back to the same problem. I thought it was a different issue because every device using my VPN seemed to have access to the file server except my laptop. But then I realized the second time in a week that my desktop does not use the VON at all, it's not even installed and so the only barrier to entry is SMB credentials. Then I checked on my phone and surely if my phone can access the server then there is no issue with the server itself. It took a long time to realize I was on WiFi and with the VPN turned off, once again, SMB is the only barrier to entry. I tried testing my laptop at one point before realizing this by using the hotspot on my phone to see if this was a network adapter problem or something and I still didn't have access. Only later did I realize the phone had access because WiFi was on and the VPN was off, and so eventually tried testing the laptop, to see if I could access the server at all with the VPN and turned the WiFi off and unplugged ethernet, and it didn't work, no bits received, then plugged ethernet back in and VPN off and then the only barrier to entry was SMB, and I had access again. But this finally demonstrated the problem was with all devices, not just the laptop, and that it was once again the VPN. I was bogged down with other problems so it didn't occur to me to check the IP address my ISP had assigned until later, and it was once again different. But by this point I had started to suspect many other things also. It took some time since I was cramming for a test to decide to just try changing the endpoint on the WireGuard app on all devices again. And now ti works again.
Strange how one YouTube video I watched said that you should create a tar ball and that most VPN issues are the result of DNS problems. Why hasn't this been the case for me? Every problem has always been that the VPN needed to be turned off and back on, the computer rebooted, or the public IP address updated. I would include all the SMB problems but this is just a VPN discussion and while SMB works over VPNs a lot, it only has as much relation to the VPN as actual cars have to traffic lights. One uses the other, the other may accommodate the one, but if either of them has a problem, this does not affect the function of that which that does not.
That last sentence reminded me of Bilbo Baggins, "I don't know half of you half as well as I should like; and I like less than half of you half as well as you deserve."
This has been Truncat3d 00000000111100010100110______________end of line
No comments:
Post a Comment