Sunday, November 26, 2023

2023-10-07 - RAID 5 SMB File Server running Ubuntu Server OS

Just so you know from the outset, anything underlined is more technically focused for the project and what I went through to make it work. Anything not underlined is more of the drama involved in allowing me to get the technical stuff to work. I had wanted to do a File server of my own for fifteen years. I have wanted my own Dropbox replacement for ten years. And I have wanted to do a RAID for five years. 

I went into this preferring RAID 10, with striping on at least two drives, which is RAID 0, and mirroring on at least two other drives, which is RAID 1, but they're both put together, making them RAID 10. Suffice it to say that I have a basic understanding at this point, speaking as my future self. Basically what I am saying here is true, but it is more complicated than this. There are other RAID configurations, but I've had little interest in them. I was aware of RAID 5 because of how strange it is, but then when I told Doug, my instructor, about this, he told me that in order to do this affordably, I should do a RAID 5. I hated the idea. 

I only had a few weeks to get the parts for this, some ideas and decisions had to be made. I had one four terabyte HDD which is nearly full, and it has most of my stuff on it, and so it's time for an upgrade. But I don't want to buy another drive that I will then put everything on just to have a bunch of blank drives or multiple drives--I want everything in one place. I have been doing one or the other for fifteen years and it's either a huge waste or it's frustrating to keep jumping from one drive to the other to find things. Of course when I moved away from older drives, they have also aged considerably and are not very trustworthy anymore. So what can you do with that? I need a solution that lets me use all my drives and have a lot more space but keeps everything in one volume so-to-speak. 

I thought I could obtain a computer to turn into a Windows server through my friend and roommate, Nate, who builds old computers and sells them for cheap, ready to go. 

I also thought I had one week to figure out the parts for my file server, and I had an idea. I needed four drives, preferably with more space than I already had available to me now, so two drives for that and another two for backup with RAID 10. I didn't want to put this all on a credit card, so I turned to Nate, and then I had an idea. I have a now old and nearly full 4TB drive, and then I have another 4TB drive to back it up through Windows Backup but it's only a year old. Then I have a bunch of much older drives of various sizes that I don't trust. 

If I buy the next size up from 4TB to 6TB, then I would have to buy four of the next size up and it'd take me a while to get the money saved for that and what I have that I do trust is wasted. So I found another 4TB drive online for about $60, and then had the idea, what if I did a RAID 5 like Doug said, and bought two 4TB drives to go with my one 4TB drive and then also the 4TH backup drive. 

I started putting my plan into affect, had Nate start looking through his usual channels for parts to build a file server, and the only features this computer needed to have was that the motherboard absolutely had to support a hardware RAID. 

When talking to Doug about this project, he absolutely insisted numerous times and in every way that I really didn't need performance out of this computer. 4GB of RAM was plenty...in fact 2GB would be plenty. After all, we're going to do Linux, and Linux is light, free, and works great. He sort of made that decision for me. I wasn't against it, but I am pretty much a Windows guy. 

Apple is proprietary but works however is not very flexible, and is very expensive. Not really meant for gaming, and is mostly geared towards productivity and home use. Windows is not proprietary, very flexible, just works much of the time but not quite as much as Apple, but is more financially accessible, is really geared towards productivity, flexibility, gaming, and can pretty much do most things if not all of them. But you have to pay for Windows. Linux is open source, which means that the masses have access to the source code and can download it and manipulate it if they desire, you cannot do this with MAC or Windows, and it is free. It is also way lighter than Windows or Mac, and is highly flexible and customizable, but it doesn't always just work. In fact if you get into Linux, you will be doing more troubleshooting on average. But as far as I can tell, there isn't exactly anything you can't do with it, except maybe gaming to the extent that you can with Windows. You can still do it, it just requires more work a lot of the time. 

A sort of saying is that with Windows, things just work but you don't know why because you don't have to; Windows does almost everything for you. There are tons of background things happening that you don't have to understand to use it. And then with Linux, you will know everything about how it works because it doesn't do everything for you, and so if you are a real Linux person, you know everything about how and why it works and so you know many things that a Windows guy won't know. 

If you tell me this to tell me that I don't really understand how a computer works, I don't care. My energies are simply focused somewhere else. However, I am not against using it. Give me a compelling reason and I'll foray into it. So foraying into it, I am. I don't have to pay for Windows Server OS because Linux is perfectly well suited for server tasks just like Windows. And it's not like people aren't constantly doing things with Linux such as File Servers, Linux is built with this in mind so no bending is necessary to use Linux for this. 

I went about getting the parts together. Doug, my instructor, said in no small detail that I could even get away with using an old computer. I could use a Core Duo for this. All that really matters is hard drive space and the RAID controller, or the board I purchase that has one built into it. This can be a bit tough to get a hold of. It is common for computers to have RAID built in. Doug made it sound like about half of the motherboards out there have RAID controllers but I quickly learned that just because you have a RAID controller or on the motherboard, doesn't mean that you have the ability to use any RAID option you want. It might only do RAID 0, 1, maybe it'll do RAID 10, maybe JBOD, which means "Just A Bunch Of Disks", and then less common is RAID 5. As of yet, I have not seen RAID controllers in my personal search at all that contain anything other than those five options. There's a RAID 6 for example, but I sure didn't see anything that had it. 

And also, in my search, I was hard pressed to find affordable RAID cards regardless of what RAIDs they supported. 

I should explain RAID 5, but I'll start with what RAID is. RAID 0 is what they call striping. RAID 1 is what's called mirroring. Those are the two most basic types of RAID, from here it gets interesting and uses a lot of tricks. RAID 10 is pretty much just like RAID's 0 and 1 put together and introduces nesting. 0 and 1 both require at least two hard drives to work, both RAID 0 and 1 write to all drives simultaneously, however they differ in what they write and how. So to understand why this matters, you have to understand that data, when written to an empty drive, will write contiguously, or rather in an unbroken, sequential fashion. 

Imagine a round food plater with lips on the edges, and you have peas filling the platter in unbroken circular lines in an outward fashion starting from the center. And the more data written, more peas are lined up in concentric circles outward toward the edges. That's how ones and zeros are arranged on a typical mechanical, magnetic hard drive when it starts out completely empty and fills up for the first time. And lets say one program requires seven peas, they would all be written sequentially in a row with each other.

When you have a RAID 0, picture you have two or more food platters, and they all start out empty and and an equal amount of peas start lining up from the center outwards on each platter. However, the data is broken up. Lets say a that same file that requires seven peas is being added to all the platters at once rather than just one drive. They'll be split up equally among all of the platters, still in concentric lines forming more and more outwards to the edges of the platters. If they were on one drive, the peas that belong together for the program would still be in an unbroken sequence with each other, but you're writing to multiple drives in RAID 0 simultaneously. And because there are more platters, there's more space. Why would you do this? Performance and or extra space. If your drives each write at a max speed of 6Gbps (gigabits per second), and you need them to go faster, you can implement a RAID 0, and all data that gets written is split up equally so every single drives maximum speed is combined with all of the others. 

However, someone thought that this could also be used for another benefit. If you trade the simultaneous operation of two drives for example, and instead you have two platters of peas being filled, lets say you have the seven pea program written in that same old contiguous fashion on a brand new, empty drive on one drive, but every write operation or pea that gets added to one platter is duplicated on the other platter. Instead of related peas being split up between the platters, the peas themselves are being copied for exact duplicates and lined back up in the same unbroken chains of programs on a second platter or drive, called mirroring. This is great for backing up a drive with exact duplication in case of one of the drives failing. 

Also, the spinning disks in magnetic drives are called platters, and instead of peas, ones and zeros are written and read using control arms that have fine points that either use magnetism to write data or sense the magnetic signature that was written so it can be read. And then multiple platters are stacked on a spindle and there are multiple control arms that move across the surfaces of each platter, and on both sides of each platter. 

So that's RAID 0 and 1. Then you get things like RAID 10, which has sort of been my favorite because you get the space benefit of multiple drives being written to simultaneously, striping, and each of those drives also being mirrored at the same time. You have to have an equal number of drives for RAID 10. 

Someone was looking at RAIDs 0 and 1 and thought, why can't we put them together and get the benefits of both in order to have redundancy and space and performance all at the same time? However, this is also where things start to get confusing. The second you introduce multiple digits into a RAID such as RAID 10, you get what's called Nesting

Nesting is where you have a RAID, like RAID 1 for mirroring, which nests a RAID 0 for each of its striping drives or vise versa. So essentially, RAID 10 is very much is a RAID containing a RAID. And then it gets way more confusing because you can have a RAID 50, 60, 100, 150, two digits introduces nesting, and three digits nests a RAID within a RAID within a RAID. This is easy to understand on the surface but to understand the concept of nesting is a bit to wrap your head around. I was aware of these different RAIDs for years and had no idea what nesting was, I didn't even know there was a RAID 50. But there is. And if there's a RAID 7, and a RAID 5, you can have a RAID 75 or even a 750. 

It seems really convoluted and you will wonder why in the world anyone would ever do such a thing, but somewhere out there, there's someone doing things like this for a good reason. Perhaps you want triple redundancy but like me, you don't trust all those layers because they have a potential single point of failure despite the layers of redundancy. But redundancy comes in all shapes and sizes and for different reasons and can be shaped in the case of nesting in particular to suit a specific situation. Consider that 0 is fast, but 1 is no faster than a single non RAID drive by itself. But you're getting a benefit somewhere. If I want the benefit of striping but also of mirroring, then nesting is useful. If I want to use RAID 5 to save money on the number of drives needed, or I'm willing to take a chance on RAID 6 with the ability to lose two drives instead of just one drive with 5, but I then start making more money and want more redundancy but I merely want another RAID 5 or 6 to mirror my existing RAID 5 or 6, then I can do a RAID 51 or 61. You can go as high as triple digit, and the only used example Doug has ever seen is a RAID 100. People can use that if they want lots of speed because you're doing two layers of striping.  

Man I hope I'm not blowing smoke. I already explained a ton of this once and realized I was wrong and threw my whole explanation away, to then do this far more elaborate explanation minus the picture I drew to explain what I was saying the first time. 

But there are several pieces still missing to this explanation. RAID 5 finally, is where you have a minimum of three drives. 

It just occurred to me now, and therefore I had to look it up, that all drives in RAIDs have to be equal size, although you can use different size drives in a RAID, they will simply all be set to the size of the smallest drive. So if you have three drives striping with each other, one of 500MB (Megabytes), another that's 1TB (Terabyte), and the last one is 2TB, then they will all only be used for up to 500MB, and to do otherwise would create all sorts of problems. So if you want that space, you really need to have all drives of the same size. 

Moving on, in a RAID 5, you get what's called parity. This is even harder to explain for me, but to me parity and nesting can be equally as confusing. If you don't want to go into the specifics of how they work, then they aren't so terrible to understand, but that's if you ignore how parity or nesting actually work. So RAID 5 requires at least three drives but can take any number of drives more than three unless you run out of ports to plug them into or something.

Lets say you now have at least three food platters, and you add a pea to the first two drives, those two are data that would be written sequentially to one platter, but they're again split over two drives, although I wouldn't necessarily give RAID 5 the performance badge, when you fully understand RAID 5, you'll see that it can actually be quite slow, but not necessarily in the initial writing process, just in the parity bit. So you have three platters, peas written to the first two platters,, and then the part that takes such a long time, the third pea that is added to the third platter is what's sort of like a fake bit, it's not data that can be used for anything other than the RAID because it isn't like a picture file you can use, and it's also not the backup data which would also be like a copy of the picture file you can use. Instead it's a parity bit or parity pea that uses complicated math to determine what one of the other two peas would be if the first or second platter was lost. 

In this case, lets say you lost the second platter and you replaced it with an empty platter, the RAID will use the information it has based on the one legitimate pea that's in the first platter and the parity pea in the third platter to figure out what kind of pea was in the second platter, and then recreate it from that available data. This works because we're using binary, ones and zeros. Lets say you have a one on the first platter and a 0 on the second platter and a parity bit on the third platter to represent those other two bits on the first two platters. The parity bit that gets written to represent those first two bits is determined by what bits are in those two platters. The last platter only gets one bit to describe what was there. And this may not be how it works, but since it is either a 1 or a 0, what it probably does is if it is a 1 on platter one and a 0 on platter two, the parity bit will probably be a 1 to represent an odd number because one of them was a 0 and the other was a 1, one of the platters was lost and so based on what it has now, if there's no 0 and there's only a 1 on one of the platters and then a parity bit on the third platter that says it was an odd number, then when you add a new drive to replace the lost second drive, the available bit in platter one and the parity bit will have to equal an odd number, so in my example, it'd be 1. and so the RAID controller will know that this missing bit was a 0 and write that bit to the brand new platter in order to rebuild what was lost. 

To further explain other scenarios, if its the first platter that was lost, then it would know to replace it with a 1 because the parity bit demands the rebuilt data be an odd number bit and so if platter two has a 0, then it knows to replace it with a 1. If there are seven platters, this still works because say each bit or digit in this case I list here to represent what bit is on what drive in this example is on a different platter: 101000. If you add up the two ones, that means the parity bit is a 0. So if any one of these platters or drives is lost, the RAID controller adds up what bits it has in the same positions on each drive, adds them up, and assigns a parity bit in this case of 0 because it was an even number, so if one fails and you replace it, how can you know what was missing if seven platters each with one bit have to share only one bit to back each of them up, it knows that if one of the drives for example that was a zero is dead and it still has a 0, then the missing bit must have been a 0, if it was a 1 that was lost, then the parity bit will still demand that there should be an even number so it would be zero and with one drive down and it has the correct even or positive number, then it knows that the lost bit is a 0. So no matter how many drives you lose, this still adds up because RAID 5 will always only ever be able to lose one drive because it requires all drives to vouch for what any one failed drive has in conjunction with the parity bit. 

This method of error checking or bit checking is also called Checksum and is also used in ECC error checking RAM, which again, uses one bit at the end of a series of bits to vouch for what bit is missing by using even or odd number representation. 

RAID 6 does something very similar but the scheme is different, you can afford to lose up to two drives, however the math for RAID 6 is far more complicated than for RAID 5. I'm not done explaining RAID 5. Now we move on to the other part that makes RAID 5 confusing. 

Imagine again that you have three platters with concentric circles of peas written to them. 

So you see that the backup data used for parity is not real data, it's not a copy of the lost data, it is literally just a bit that says that because this bit here is this and I have a representation of the sum total of bits in the same position on each drive adding up to this one bit in binary, it then uses that to recreate the lost data. So even though the recovered data may be a perfect match for what was lost, it was technically created through a very complicated process from scratch and your original data is actually gone, but it was able to be brought back from the brink by surrounding data that could describe the details of what that lost data looked like. 

It's very much like a thief in the night leaving evidence of their presence, the footprints tell us what size shoe they were wearing, possibly what kind of shoe by the tread pattern, where they walked in the house, were they running or walking based on the distance between marks and characteristics indicative of a print made while running, and by examining all this information, a forensics expert can recreate or imitate what the actual thief did in order to get a clear picture of them in the act. They can demonstrate what shoes they had on, how big their feet were, whether they were walking or running, possibly how tall they were by the length of the stride even, the characteristics of their gate as they walked, how many times did they go to that one spot and how many times might they have been standing there because numerous prints were smudged over each other as they shifted their weight and certain prints got deeper and darker than others and they re-stepped in one spot so the new print is slightly off from the original print before it. And so the expert can show you exactly what they did. They are not the thief but you know what you need to know to recreate the data of what the thief did, or in the case of our parity, what bit was lost, so that it can be imitated and written to the new platter or drive even though the computer can't know wat the file was with all the bits put together, it can rebuild the file with the cleverly used trick of pseudo data, bit by bit. 

Just to be clear though, as I realize the flaws in my explanation, RAIDs do not apply to individual platters in a single drive encasing, they apply to multiple drives only and each encasing with its own spindle and platters are treated as one drive. The platter explanation is merely intended to explain how data is written to a disk by using the whole disk, encasing and platters combined, to write on a hard drive in various conditions. 

Moving on!

I had many conversations with Doug about the reliability of software RAID (SRAID). I had learned six years ago that you should stay away from SRAID and stick with hardware RAID (HRAID). There were many stories about how if your OS goes down or something happens to the motherboard or this or that, then you lose everything, plus HRAID is faster. But I had many debates with Doug where he assured me of how great SRAID is and that it can be backed up and recovered and this and that. I just need to save the configuration file, which is apparently only 1 KB. Hard to believe that this is all that stands in the way of great loss, a 1 KB sized config file. But despite multiple debates with Doug, he would explain circles around my argument, but still leaving me feeling unheard. However, I felt the only way to rectify this would be to ask my friend Bo, who has been in IT for twenty plus years. Seems like he always has some point that doesn't jibe with what Doug says, so this has to be ironed out before I can even consider trusting an SRAID. Seems that no one in my class or my instructor Doug has ever heard of this reliability concern with SRAID so maybe Bo knows...ha ha, Bo-Knows. 

He told me once he was thinking of doing a podcast and calling it Bo-Knows, but I guess there was a name copyright thing or whatever so it never happened. 

While this RAID debate was in full swing, Nate discovered the old, used motherboard he had located and I had just purchased had a few bad capacitors. I panicked and Nate offered to buy the old motherboard from me to make it up to me since he missed it in the purchase. I didn't do that because I had more money than him. I was miserably depressed because my project was hanging in the balance and I couldn't wait to get it going. I didn't want to spend more money to get a part I already had. I was already looking at trying to buy a power supply and couldn't find one. 

I ran the power supply issue by Doug and he recommended that since I work at Deseret Industries, a thrift store, they get computers in all the time, and I could buy an old one for five or ten bucks just so I can steal the power supply that would ordinarily be far more expensive all by itself. I looked, I found none at work. I checked another location, no dice. 

Nate had a power supply he bought for $50 or so, and never ended up using it, and it had far more wattage than I needed and I preferred modular or semi-modular power supplies. I prefer having not having the sum total of all power cords that can be connected at the same time, most not being used, get in the way, are eye-sores, complicate cable management, and exit the power supply in a huge bundle. If I just need a bunch of SATA power and no Molex, I'd like the option. 

I found out the hard way that Modular is way more expensive, and then to add insult in injury, you really only start to see modular power supplies around 600 watts and up and I was thinking more like 500 or less, thinking I could save money by searching for a less powerful, lower quality PSU, but I will also have to sacrifice my love for modularity in order to buy a PSU from a trusted brand for no more than $40, and $40 was already as high as I was willing to go, finding only two brands I trust. 

I had this idea after a mostly depressing day at work, feeling down about my capacitor predicament. I suddenly thought, wait, this could just be apart of my project, replacing capacitors in class. I could add this to my list of experience with this project. Not only do I build a server, make it functional and use it regularly, learn something else about Linux, have another computer build under my belt, set up an HRAID 5 on it, and then write about all of it in this blog, but I can now add soldering as well as replacing blown capacitors to my repertoire. 

I ran the idea by Doug. He shot it down nearly instantly by telling me it is likely to be too specialized for this class. I asked, wait, we've got a soldering iron in class, what do you mean it's too specialized? He said it was likely too small. 

So now it's Groundhog Day, reliving my misery at work again. Out of desperation, I asked Nate over the phone how hard it would be to buy another board, when he said he'd start looking, and suddenly I thought to ask, wait, don't you solder occasionally? He said yeah. I was beside myself. Why hasn't he offered to solder this thing? I have a soldering gun if that's the problem. He said he could do it, but he needed the right capacitor and more flux, which I would have to buy. "Okay, how much damage is this?" I asked. He spoke as if lowering the boom, and I thought he was about to say that I might as well just buy a new board when suddenly I registered in my head that he said about $10. Man, that's it, lets do it! 

I thought of the reverse of that scene in Father of the Bride when he gets the news of how much his daughters wedding is going to cost with a wedding coordinator and he misheard the guys quote, he thought for a second there that he said one hundred and fifty dollars per person that attends the wedding, and the guys like, "No, noooo!" "Good I was about to kill myself!" "No, two hundred and fifty a head." 

So Nate was willing to solder a new capacitor. There's a major downside, I get it fixed and don't have to buy another motherboard, but if Nate does the fix, then I will not be involved in the soldering job and get this extra experience. That thing that excited me several days earlier about this really unfortunate occurrence having some sort of upside because it can now just be apart of my project to fix it, was then removed from the equation. Nate hates showing me how to do things or helping me do things. I have a tendency to be particular, and he has a tendency to just get things done even if they're substandard. I asked him if he could at least show me what he was doing while he soldered so I can at least write about it in the blog and he said sure but I'm not going to see anything because his face is going to be right up in there, working really closely and I won't have a view. I knew he wouldn't explain what he's doing. So eventually I gave up and just let him do it without me. What was I going to gain from being there? At least the capacitors would be repaired. 

I thought about this, why did Doug think this was too specialized? Nate was about to do it next to his keyboard with a cheap soldering station while watching the Simpsons. He then moved it to the kitchen table so I could watch until I wasn't able to show up anyway and gave up trying to be there. I took pictures of the capacitors I had just bought and showed Doug the picture and asked him what was so specialized? He said, oh, I thought it was one of those capacitors that are no larger than a few millimeters long, no larger than bread crumbs. 

It didn't dawn on me to move the soldering operation back to class, Nate was already about to do it. 

Then the night before Nate was going to solder the capacitors, I asked Doug on the way out the door after class, wait, you still sounded a little apprehensive about helping me replace the capacitors. You said you still wouldn't help me do anything but remove them, I'd have to solder the new ones on myself or something to that affect. He said, oh yeah, well, your friend's about to do it. I said, wait, yeah, but you knew that when you said you wouldn't do anything more than remove them. I was confused. Why say anything if he knew it was covered? He said, well I don't want to screw up your board. I might mess up and you'll blame me. That seems to imply that this is risky. Should I not have Nate solder my board? He said, sure, if he knows how. Well, you know how. Yeah but its risky, he said. I could accidentally burn the PCB on top of trying to remove the capacitor. Oh. 

Nate started working on the motherboard while I was out and ran into a few problems. He was able to remove the capacitors, but the contact wires for one of the capacitors wouldn't melt and we couldn't get the soldering iron right on them, and we needed to remove them in order to put new ones in. I got home just in time to see this unfolding. Nate and I worked on this for about fifteen minutes when I thought, why don't we just heat it up as much as we can and poke the thing as hard as we can? Do we have anything as thin as these contact wires and strong enough to poke the old, un-melted wires out? I asked for the bag of new capacitors. I grabbed a new capacitor and heated the hole as much as I could and started poking. 

I was able to poke the contact wire out, but only for one of the two blocked holes in the motherboard. I had already been on a rollercoaster ride over this, I got paid that day and I was due to bring this computer in the next day to start working on the file server in class. Not wanting to waste a good Thursday project day because there was only one project day a week to work on this in class, I wanted this problem behind me. I decided to go buy another motherboard so we didn't get hanged up any longer. 

I got another used, older motherboard. But this time it was considerably newer, maybe seven to ten years newer. This was an MSI 970a-G46. 

Turns out I had the g43 for 9 years and had just rebuilt my computer several months earlier and gave the old one to Nate as payment since I suspected many parts of my computer to potentially be at fault for a video cut-out a few seconds long every few seconds. I discovered the irony and found out that my old board had the same raid controller that this other board has. Could have avoided this whole charade if I had just kept my old board. So the second used motherboard was now installed in a case I was borrowing from Nate, along with that PSU I was also borrowing from Nate just so I can get this project rolling in class. 

Day 1, a student named Jacob was helping me and we began working on the file server project with the new motherboard. We set about to figure out what to do first. I looked up a video on Youtube that kind of helped us to figure out that turning RAID on in BIOS was the first step. So to access BIOS, I think I mashed and spammed F1, F2, F12, Delete, and Escape. Which ever one it was, it seemed to work. We scoured all of the options in the BIOS to find it. While in Standard mode in my BIOS, we went Settings > Integrated Peripherals > and changed SATA Mode to RAID Mode. 

Turning on RAID Mode seemed to work totally without a hitch, so we tried to access the RAID controller from there according to that same video I found on Youtube. This started to become very difficult. Other instructions I found online said specifically to use Ctrl+I or Ctrl+M. 

I didn't know it was called the RAID controller yet, or that it was located in the North Bridge. I thought it was just another thing similar to BIOS or to Recovery Mode, just stored in the same place. I also had no idea that the chip in the middle of the board, below the processor, which often had a heat-sync on it, was the North Bridge. Apparently a lot of features are housed there. And apparently there used to be a South Bridge but I think that's not a thing anymore because it's unnecessary or we moved away from that type of architecture of something. 

We entered the raid controller and selected raid 5 and selected all disks and the raid 5 was created. However we were looking over our settings and were confused that the RAID configuration was only recognizing maybe 5 or 6TB of my 16 available TB of hard drive space. It turned out that it was only assigning each 4TB drive a total of 2 usable Terabytes. Doug took the reins and went back to BIOS and saw that it was in legacy mode and he was like ah ha or something like that, switched it to UEFI and then had us re-enter the RAID utility to continue with the HRAID. But now we couldn't access the RAID utility for some reason. We would enter Ctrl+I and Ctrl+M and nothing would happen. 

We tried to figure out what the problem was but we were stumped. We went back through the BIOS, we rebooted, we turned RAID off and back on again. This BIOS has the option of switching at the top of the screen between ECO mode, STANDARD mode, and OC GENIE II mode. I tried going through these different modes just to familiarize myself with them more fully in case there was anything I was missing. That wasn't helpful. After some time I got back to STANDARD mode. We probably tried different SATA ports. 

Some time later, Doug came by and went through the BIOS to see if there was anything he could find. We went through all menus he deemed at all relevant to creating a RAID. We got back to the Settings menu, searched, and then again clicked on Integrated Peripherals and looked around. He spotted the option right below the one to switch between SATA mode and RAID mode, called Board SATA RAID ROM, which was set to Legacy mode. He had us click on it to discover the option to switch to UEFI. That turned out to be important and revealed a far larger problem. 

It turns out that Legacy mode means using MBR. So MBR, or Master Boot record, is a partitioning table and when they first created it in like 1983, they were thinking that they needed a standard that would allow them to partition a drive, or rather, split a single drive into multiple containers. Drives in those days were really small and they never imagined that files and drive sizes would skyrocket, so they said, lets just make the maximum partition size two million megabytes and a maximum of like four or so partitions. Great how that worked out for us!

So I think the problem here is obvious, we're using MBR in legacy mode BIOS and that will limit all four of my 4TB drives to only 2.2TB. So it needs to be GPT. GPT, GUID Partition Table, which was created in the late 1990s, recognizes far larger partitions as well as allowing for far more of them.

As a quick aside, I thought I knew the difference between BIOS and UEFI. I thought UEFI was newer and replaced BIOS, which apparently unless I'm wrong, didn't replace BIOS, it just adds to it or something, they run at the same time. I thought if you had mouse functionality, then it was UEFI because BIOS doesn't have that sort of thing. I guess I was wrong. 

So the way Doug explained it to me...I only seem to briefly understand things when Doug explains them. I seem to know stuff but if I explain it, I am always wrong by a longshot. I suppose I could just never explain anything and people will always think I'm smart. And I'll seem mysterious. lol. 

The way Doug explained it was that depending on how new the motherboard is, what you see, the graphics, all the stuff that makes the BIOS or UEFI look fancy is just a result of the time that the motherboard was built in. They both work differently, the thing you see which we call BIOS or UEFI is just an interface, but it starts to do its stuff when you start booting. And you want this process to be backwards compatible so I assume all new machines have BIOS and UEFI, and can be switched between the two depending on the situation. However Windows 10 and later are UEFI only. They cannot run on BIOS. 

So the beginning of the far larger problem slowly starts to reveal itself here. Doug seemed confident that switching from Legacy to UEFI solved the problem we were having in getting the RAID controller to work. He already knew why Jacob and I needed to use UEFI so that we could do the RAID with GPT. None of this meant anything to me yet. So we went about the process of re-entering the RAID controller. We couldn't get in, and it would just skip on to installing the OS. We were confused. It was working before when in Legacy mode on BIOS, but now it wasn't working in UEFI. We would get to the screen in the boot process where we had to either press CTL+I or M, and it wouldn't respond when we did so. We worked on this for quite a while, Jacob had to go because it was the end of class. But before I started packing up, Doug was actually very curious about this problem and continued to work on with with, wanting for me to be able to make some sort of progress since we only get to do this once a week. 

Basically we spent the next few hours trying to do a firmware update. We found something online for this motherboard that said that there were RAID set up files for this motherboard. 


So we tried to do a firmware update. We talked about updating the BIOS, but this made it look like it was specific software that was supposed to come with the motherboard that would allow us to do a hardware RAID or HRAID. So the RAID controller only worked in legacy mode. This really annoyed Doug. As he would later put it, "Why would you put out a motherboard with all these features, just for none of them to work when you switch to UEFI mode!?

Doug recommended I do software RAID, which I didn't want to do because my understanding was that SRAID was unreliable. 

I wasn't ready to give up on HRAID. My roommate, Nate, was now also helping out in looking for the RAID setup files online, and he found a few places, so we downloaded those files which Doug and I had tried before we finally went home, but they were the wrong format from what the motherboard was requesting. 

At this point, Owen, the security guard, walked in the room about an hour or so after class to ask how things were going and Doug said oh yeah, we need to get out of here before the alarm automatically activates in the building. Owen said, no actually that activates at such and such time. So Doug and I kept going. 

Doug got an idea to try something, since the internet was telling us the files we downloaded were the files we needed, but the motherboard kept not accepting them and saying that we needed this other weird format Doug had never heard of. He got an idea to just change the extension to the one the motherboard wanted us to use, but he was concerned that this could brick my motherboard. We decided that we could try a BIOS update to see if that would change anything and then we could finally install the files or if perhaps the update would already have the files included.

Nate took it upon himself to update the BIOS for me while I then decided to give the older motherboard another try since we just needed to clear those soldering holes. Doug then instructed me to not go and blow a bunch more money on yet another motherboard or other parts that I was not already planning on buying to make this work.

So back to the old motherboard. Nate and I had stalled on fixing the capacitor on the old motherboard because I just ran and bought a new motherboard, which is the board that Jacob and I were struggling to get working and then Doug and I spent hours after class working on. 

Doug said I could bring the motherboard in Monday the next week and he'd allow me to do a project day kind of thing since he was going to have some time. And I think I was caught up on my work so he would see if he could remove those stuck leads.

I brought in the old motherboard so he could try and fix it himself, he held the soldering iron on there for much longer than I had, and repeatedly, Doug managed to get one of the two blocked holes free but the last one was proving very stubborn. He did the soldering iron on both sides of the motherboard, poked it more, and he turned it up as hot as the iron would go, and used the de-soldering suction gun we had in class. But the blockage would resolidify before we could get the suction on there and neither of us could get a good seal on the board with the suction anyway. We ended up using the sim tray pin tool that came with the iFix-it kit we had in class, and while pushing on it with the sim tray pin from one side of the motherboard and melting from the other with the soldering iron, Doug eventually got the stuck piece out.

The motherboard now had burn marks where the two holes were cleared, but it was fixed. Before I went home for the night, he said to go ahead and bring in both motherboards. Oh and he did also say that there was a good chance that he friend the board and it wouldn't work, however I think he also said that this might not have mattered anyway because for all we know, these capacitors really didn't factor in what we were going to be doing with the board. So this could have all just been academic. 

I bought a new computer case I thought emulated what I liked about cases but also this particular project. It was an open-frame case for less than a hundred dollars. ______________






Project day Thursday, it was before I was going to head to class and I had switched the new motherboard that couldn't recognize the whole RAID out of the computer case and put the old motherboard back in. And then I ran into class with the old motherboard in the new case, ready to hopefully get the RAID configured finally.


The next number of classes was just about doing assignments and discussing with Doug what we could do to rectify the situation. I told him the old motherboard I bought first for $25 was still in play because I was having my roommate, Nate, solder new capacitors to the motherboard. So he did. I was going to watch but had too much going on and so last minute told Nate to go ahead without me. I got home before he finished, just in time for him to tell me he removed the bad capacitors, but part of the soldered parts of the leads from the bad capacitors broke off and were now stuck in the motherboard.




________________________________

  • in class we discovered that the new motherboard had a fake raid controller that had hardware built in for the raid, but it relied on software, I had to decide if I was going to try to make an SRAID and doug kept on trying to make the raid himself to see if he could do it, so he could then delete it and let us do it for the learning experience. He failed. But we had a few options. I couyld bring in the mpold motherboard monday to see if he could clear the soldering hole. 
  • that next monday, i brought in the old motherboard to see if we could clear the soldering hole for both contacts for the capacitor. doug managed to clear it using a plunger tool, some solution emant for difficult solderings, and then he just resorted to using melted solder wire to mix with the broken capacitor wire and we managed after some time to melt it and clear the hole.
  • i ordered the new case. I wasnt going to buy this case because the old motherboard from the HP xw or wx 4400 had a different form factor and I didnt think any new case would take it. But since we bought the new motherboard, I went ahead and bought it. In putting it together nate and i discovered that I couldnt have bought the more perfect case if for whatever reeason I did end up needing to switch back to the old motherboard because all the connection points were totally adjustable sideways and up and down.
  • doug said that it may not be necessary to fix the capacitor because theres a chance that it didnt actually use that capacitor so this could be a waste of our time. 
  • i got the computer case and installed the old motherboard, nate soldered the new capacitor on and we changed the new motherboard out for the old one that we thought had a real raid controller instead of a fake raid like the new one had. 
  • doug did warn me that he was not willing to make the new solder himself because he didnt want to be responsible if it didnt work. I told him nate was helping me do it. Then he pointed out that it may not work. 
  • doug told me i should bring both motherboards just in case. 
  • second thirsday, the 12th, i brought the computer with the new case and the old motherboard in with all the capacitor repairs done to it, nate and i tested it and ot booted just fine. I worked on it to creaet the raid, did so easiuly. but when selecting the disks for the raid on the old motherboard this time, i disicovered that the partitioning scheme this computer used was mbr instead gpt. unfortunately mbr only recognizes disks up to 2TB in size. I asked if we could just partition the disks so they were only 2 TB per partition on each 4TB disk and he said thats the problem, it will only recognize up to 2 TB on a disk anyway so we wouldnt even reach that point.
  • we thought about installing win8 or 10 where gpt is recognized, but we determined that this motherboard was too old in order to recognize anything newer than win vista, which used mbr. 
  • I switched the motherboards in 15 minutes and got it tightened in and ready to use. we went back to trying to solve the problem with the old board. good thing i broguht it. it had gpt. but i did some research on the raid controller the new motherboard had and discovered that it was not a fake raid after all, it was a full raid controller that did not rely partially on software to function. its just that it was integrated into the south bridge and thats why it wasnt immediately distinguishable on the board.
  • i had activated raid on sata mode juts like before and immediatley after that, changed the legacy bios to uefi mode since i saw doug do that last time. but we were totally unable to access the raid cinfiguration utility. we couldnt figure out why. it wouldnt even acknowledge that there were disks connected to the board to install anything on. all the computer would boot to was the bios. we played around with it for a while and turned uefi mode off. all of the sudden we were able to create a raid 5 with all the disks being recognized just like last time. but doug said we had to then go back and change it to uefi mode again because ubuntu server couldnt be isntalled on legacy mode. a little while later we discovered that th e reason this wasnt working was because the way legacy bios and uefi read disks is completely different. So what you  do in one mode, is unrecognizable in another. create the raid five with all disks in legacy and switch to uefi and all of the sudden it couldnt find any drives to boot to or to install an os to. 
  • i was frustrated that no matter what we did, we couldnt get this thing to work. in the middle of this, a game sort of came about organically where every time a group of students working on a project said, "Hey doug" because they needed help, he had to do ten pushups. He was allowed a grace period of 5 minutes so if fve minutes had gone by and someone said it, he had to do the pushups and then come help. I got a littke impatient but never sdaid anything because i needed his help at this point being completley stuck and time was being wasted because he was doing push ups every few minutes before actually coming to help. Then he sat at his desk for a minute while I was waiting and I said "hey D..." and realized what I was doing half way through and then completed my sentence, "...oug" and he gave me an evil look and then did ten pushups. Everyone said I was playing with fire even though  this was my first time making him do pushups. I guess it didnt really help that 20 minutes earlier I made a deal with him that I would just say "Hey Mr. Douglas" so he didnt have to do the pshups. So having made this deal, I made him do pushups and he gave me an evil look. 
  • Class was over, everyone but me and Jack had packed up and left but he was still trying to figure out the problem this other group was having, which was strikingly similar to ours. They too were trying to create a partition and install linux, they just werent making a raid for a file server, but I think they were trying to make a server. They had failed and all but jack left, so doug was trying for just a little longer to figgure out what the problem was. Then jack left when doug still failed and said theyd try again later, and then he came to me and tried again. We were there for another nearly 2 hours before we both went home and Owen the security guard came in and said that just so we know, the building automatically activates certain security features at 11, and it was about 10:30. 
  • we had tried several times to get the raid to be recognized or to do the raid in legacy but he knew that ubuntu wouldnt be recognized, so we looked at firmware. we tried to flash the bios. doug found a page online that looked like it was saying that with this firmware update, it opened up the feature to do raid in uefi mode rather than just legacy bios. we tried to flash the uefi from the bios but it wouldnt recognize the file format the update came in, which was an exe. I called nate and he did this sort fo thing all the time, he found an update that used another format and the vboard still didnt recognize that. doug tried a few more times. I asked what if we changed the format of the update to a .txt file and looked at the ocde and then doug had the idea of just changing the format from the format we had to the format the board regognized which was .ob1 i thin,. yeah, star wars, haha. I didnt realize that till a few days later when explaining it to nate and asking f he could figure out how to flash the bios. 
  • finally doug concluded that we needed to isntall win10 on this motherboard to flash the bios through win10, which would then accept the .exe format the first update he found was programmed in. but we couldnt figure out why it wouldnt isntall win10. then as we were packing up, doug said, wait, lets take the sata ports out of raid mode and see if it installed win10 then. I said my roommate should be able to do that since we werre leaving.
  • nate did it. win10 was installed, the bios was flashed, and it was at this point that I asked nate if that means the uefi was also flashed and he said he didnt know but maybe. i also asked if maybe the uefi raid update could possibly be a standalone update or something and we didnt find anything that indicated any such thing. 
  • talked to My friend Bo Landsem on phone. been trying to get him for over a week and a half to ask him to explain the reliability factors of HRAID vs SRAID. 
    • this turned out to potentially throw a monkey wrench into my whole project because when I explained any aspect of the project to hi, he began to question every aspect, asking why would I use Linux. He has points. Windows does make things easier. As far as the productivity factor because of how much time you spend solving problem's and much of the reasoning to use Linux is because of the money you save because Linux is free / open source. However with Windows you just part with the cash even if it is exorbitant depending on your use case or you just would rather save the money, the fact that it is done so much more simply in Windows saves you so much time and frustration that you have the ability to take care of other issues for that client or other clients, or you can simply work less for just a little less profit. There is merit in the argument. I am personally a Windows guy, but that is only because I grew up using it and understand how it works in a lot of ways. I know what it can do which lines up a lot of the time with what I want it to do. Each OS has its own benefits and disadvantages. But I am trying to be objective here. Linux isnt my first choice but in looking at my project, I do want to keep copsts down and I am willing to learn how to use Linux in order to complete my project and get what I want. Bo made the point that things just work better, they break down less, theyre easier to maintaina nd set up, most of the time they just work, but you have to pay for it however the cost is worth it. His friend Scott and other friends of Bo's make the point about cost, they talk about other Linux features vs Windows features I dont know about I am sure because they have these debates over the course of decades, and Bo mentioned that many of these friends have switched to Windows finally because it is easier and worth it in the long run. Okay. But the point here is to learn and keep costs down. So Bo talked about that. He actually did convince me to switch to windows, but since I want to be a network engineer like Bo, I wanted to do Windows Server, he recommended the latest one and those were easily multiple hundreds of dollars. I thought, well surely I could buy a used windows key for like $30 like you can with windows 10 and 11. He even recommended it which I thought wasd weird because he listed a bunch of reasons why used keys were actually not a great diea back in 2019, such as that you dont know whaen they will expire and so you buy one just to find out a month later that it expired and now you have to buy another one and you just dont know the condition under which they were used, what package were they apart of. Businesses end usage or go out of business all the time and so to make back some of the money from the failure, they will sell everything they had such as computers and equipment, if they owned the building, and in this case, windows keys. But now he was telling me different and so I thought id do that but then he turned me to a site where you could buy keys and the keys were for hundreds of dollars, and may of the keys were just for server cals, which is the purchase of access for more computers to your server. You can buy a server OS with no CALS, or just one CAL, which would be ideal for me because I would be the only administrator and this doesnt necessarily affect how many computers I would share connections to the server with. In my personal case, they could all just use the one password or whatever. I'm not sharing this with more than maybe one or maybe at the absolute most, two other people. But just the CALS were at least 30 dollars just for one. So I thought, okay. back to Linux. I want to keep costs down. I was willing to go the windows server route since I want to work with windows server anyway, it started to make sense, like wait, I am missing a prime oppertunity here to get into the neiche I want to be in anyway, why wouldnt i do windows server OS? And then I saw how expensive they were. Then Bo started attacking why use a server OS at all, just use win10. He explained he could create a server on his computer right now in just a minute or so. I thought, okay, thats interresting, I would like to know that just for achedemic reaosns, but I still want a server. Then he attacked that for a while, no really, you dont need a dedicated compyter for it at all. And I thought I had finally developed the ultimate end argument comment because this is not the first time I have been badgered over the way I want to do things juts because someone else, who is smart and in general, smarter than others around them, think that ebcause they have so many things figured out, it makes no sense why people would conduct their lives or a particular thing any other way than these smart people have learned to and actively do condict themselves. 
      • I had a friend that I often got help with problems with my house or truck, and I explained that I was trying to make more space in the cabin of my truck for stuff so I could clear out the bed of the truck. It was all stuff I want to keep in the truck in case I needed it. But this other friend, Adam, was like, i see you have 12" subs under your back seat. If you just get rid of the subs then you have tons fo space for all that stuff thats in the bed. And I said no, I use the subs and I like using the subs. In fact, I use the subs every time i get int eh car. And I was beside myself because this reasoning meant nothing to Adam, he kept harping on why I dont just get rid of my subs and its like, dude, I want to use my subs more than I want to use the tire pump in my bed. Id be more likely to lose the tire pump than the subs. The punp is there for emergencies or occasional convenience. Okay, so get rid of the subs. Not happening. This just went on for fifteen minutes, and then he escalated his argument, in fact how often do you use your back seat? Maybe once a year I said. Okay, so why use your back seat at all, just hinge it upwards into the storage configuration against the back of the cab and put all the stuff thats in the bed here in the back of your cabin. No, I want to use the seat when I need to. He then made it personal and compared my reasoning to my moms reassoning because he saw her apartment where shew clearly had way too much stuff for such a small apartment, he asked if she conducts banquits because she has enough cookware for it and I said no, she hardly cooks at all. I explained that she lost a bunch of her cookware in a devorce and so shes like someone from the great depression where they can just never have ennough of that thing saved up for the possibility that they go through another situation like that again. If it was food, then you stockpile food, if it was money then you could never have too muc j money saved up and itll be one of your highest prioroties. For my mom, one of those things is cookware. Im not justifying it, but that is the fact of the matter and I have talked with her about it, others have, then they ask me to talk with her and I tell them Ive been talking to her about it for many years and its just not something I have been able to get her to get through to her about. And now adam was using this as an example why I shouldnt keep my back seat and should get rid of my subs. Ok, lets put this in perspective, you have your way, I have my way, I am dissatisfied with my waybbut that doesnt mean your way is THE SOLUTION TO EVERYTHING. There are many ways, and often it takes me a decade to solve a seemingly stupid problem. Like how do you get suspenders to stop slipping off your sweat pants, well, it took me ten years to finaly just put a safety pin in my beltless pants so the teeth of the suspenders have something to bite into. Who gave me this solution? It wasnt Knows-Everything Adam. And guess what, it works. It was simple, cheap and very effective. That is what I look for a lot of the time. In an unrelated note, what did Doug say in class like a month ago when troubleshooting something for an employer, he said always be simple, fast and cheap. ????????????????????????. Then Adam escilated his argument again saying that you should never keep anything in your car because people break into cars, and if they breka in and take nothing, you still have to replace your window or door or whatever. But if you leave it open, and theres nothing to take, nothing gets damaged and nothign gets stolen and if they dont see anything they might want to steal then theyre less likely to try and enter your car. My question is what does he do to keep his family safe in their house? he argued that you can buy lock picking kits at the local store. ge gave an example of this time when his neighbor had locked himself out of his house and adam went to the store to buy a lock picking it while the neighbor decided to climb into his attick and get down to his fron door and unlock it from the isnide. And adam got back from the store and had picked his front door locks before the guy got down from the attick ato unlock his door. So when I was driving long ut of Vegas last year and there was wind storm coming up behind me, driving 80 and the lid on my truck was getting violently blown open from the rear wind, and I didnt want to lose my stuff or have the lid broken off, I had to stop to see if I could fix it and there was nothing I could do and realized all i could do was put something heavy over my stuff so it couldnt blow out and then felt stuck because I had to keep on arm on the lid the whole time while moving 40 pound salt bags with one arm up and over other items in the bed. I concluded that the only soltion was to one, buy new locks to keep it closed and two, drive as fast as I dare to keep the lid from blowing off in teh short term. I got adams help to install the locks and we argued for a half hour about using locks and he began to tell his negighbore lock picking story again and I told him I knew, and explained the wind thing that happened and because hes smart, he pointe dout thats like a one in a thousand scenario. And I said, greta, so going forward when it happens again that one time, I will have lost a mere fifteen dollars on locks rather than hundreds becasue the lid bloew off. And he kept arguing and when all of my points overrode his, he stopped arguing. He even said that if the point is to stop the lid from getting blown off, why have the locks set so the lid stays all the way down, Im already wasting money and time on this and I said, well I will also have the ebnefit of locking if I want and he mentioned his point again, yes but adam, not everyone knows there s alock picking kit every 20 feet on this planet and how to use it. ignorance and the immediate block from robbing me stops a lot of crimes from taking place, let me just have that advantage. most potential robberies are stopped merely by one instance of protection, so a front door with a lock is one insatnce, then getting caught by the police is another, and then the resident having a gun is yet another. But most robberies are stopped becasue someone would do something on a whim and cant because there was a lock in the way or someone near them might see them steal the item. Let me have the one insatnce and all I lose is 15 bucks to protect me against most scenarios. I seriously dont consider myself a brilliant person but seriously, smart people say these things and I know what it is, they have seemingly great ideas really quickly because they have a high IQ, I dont necessarily, my reasoning comes from really pondering over long periods of time, and I have found that with time, I usually can out argue most points high IQ people make who are not willing to take the time that I musttake. My ideas are stupid in the short term, but get really fleshed out, whereas theirs are instantaneous and look really sharp. But its like a pencil thats sharp compared to all the other pencils but thats because it hasnt been used for any length of time. Not to say quick and sharp isnt increadibly useful, but its also not the only thing that matters and not all quick and sharp solutions stick. I also told adam I wanted the lid as low as it goes with teh locks so that wind cant slip inder the lid and blow it up, he saida thats a small gap and I said, yeah but at 80 miles an hour, thats all wind needs to rip off my locks and now my lid too. For the first time ever, I whitnessed a smarter person than me say "Oh, that'sd a good point." I had time to think about things like this so I was prepared. So for once I was ready with a great argument that couldnt be disregarded. I was about to say blown off but...I'm not a huge pun guy eveb if occasionally...
    • Bo then attacked using windows server and then said why not just use win10 and I was like, yeah but now I need a system that can run win10, it wont be low power or low heat or low footprint on the processor or ram. And now I have to still buy something for it. After my experience arguing with adam I was like, hey Bo, look, even if its a bad idea, I see where youre going, but this is not only a solution I would really like to use, its also a great learning oppertunity, which I dont want to spend much money on. He was trying to tear down those points already before I finished speaking and I said, oh and when I write about this in my blog, its going to help me get a job, and then he lost a lot of steam there and said, yeah, I see your point. Yeah thats good. And then I went on, later after he had continued to tear my project apart, and I said yeah I see your point, however, I need to be alowed to make my own mistakes no matter how unwise you think it is to do this. Then he went on. I really thought I lowered the boom on him but no. He skipped a beat and then continued. And then he attacked the very essence of even having a file server at all. why not just use google remote desktop. its like google drive but you install it on all computers incvolved and then you can retrieve any file you want from any computer at any time. (I didnt say this but, yeah well this means I need to leave my desktop running 24/7. I dont want to do that. My computer actually really heats my room up. Id like to have a break from that. Yes, I know, the server would also heat my room, up, but you see, its so low performance, and its just running Linux, so it wont heat my room up nearly as much. We're talking about solutions like one person owning two cars because one is huge and goes right through the gas and is great for special use cases that its used for all the time, and then owning a sall sedan that barely uses the gas by comparrison and you can do most of your regular driving in that. Sometimes these things just make sense. If you only have money for one car, okay fine. Thats not the situatiob im dealing with here, used compuefrs running linux are cheap. And then Bo went on, in fact, you really should just store everuythign on the cloud. I told him my same argument I gave him 4 years ago, what if I come up against a situation where I have to explain my expenses because I need finaincial help. Like relatives for example, they will only help you if youre doing everything you can to help yourself first. I dont want to have to explain a cloud service expense. I buy the parts one time and it mine. Four years ago I said this and he recoiled immediately and said, well, yeah thats a good point I would see why you would do that. You buy the parts one time and they are yours and you dont have to keep spending money on them for eons. its like having a stroage unit, after enough time, that storage unit costs more to keep over the long run than what the worth of the items within. I would never have to explain the power consumption costs to anyone hlping me with bills for example. it could have been my AC running just as easily as my computer and not many people are going to scrutinize a power bill much unless its just really exorbitant. But lets be honest here, computers dont require that much power over the corse of a month, especially not the one I use. My portable AC easily uses more power and it barely raises the price by ten or fifteen dollars a month. 
  • I concluded that after talking to brother landsem, okay look, is SRAID more reliable than HRAID and he said yes absolutely no question. Hes had all raids fail on him but he was always able to recover hardware raids and he had a dssoftware raid he wqas simply unable to recover. then he told a story about sitting with a microsoft guy and he showed him how great SRAID was and Bo was like, and to recover it? And the guy saud, oh, well theres no way to recover it, but other than that its amazing. And i told him that doug said you can recover it, with both SRAID and HRAID, you simply back up the 1KB configuration file that goes with the RAID. And then Bo made the point that maybe he just doesnt know, its been a bit, which are his words, since he talked to this microsoft guy and maybe things are different now. I thought, yep, hes probably saying 20 years since that conversation, and doug is absolutely all over me about how SRAID is recoverable and he uses then all the time and I want to know how reliable vs HRAID and he says hes never had an SRAID fail on him. Bo questioned, okay, well is he just your instructor or does he really know what hes talking about. I feel like qauetions such as these really question my intelligence. Im being really thoghtful about this, is there any part of this that demonstrates me to be simple minded on the matter? I said, okay, Doug is not only an instructor at MTEC at night, but he is a system admin at Brigam Young Universtiy full time and also does admin work for some other company too. Oh, yeah well he probably knows what hes talking about. but i have had almos every raid fail on me or something like that, he said soemthing just like that I just cant remember the exact wording. and owen the MTEC security guard whos also in our class, he uses software raid too and it hasnt failed him. I asked hi how long its been implemented and he said soenthing like nine months and I was like, oh, and kept it to myself that that's not exactly a reliable example when I was talking to Owen. 
    • Since then I have concluded, hold on, Bo isnt considering privacy issues, the fact that prices for cloud services can rise at any time, what if I cant make the paymet that month, do I juts lose all my stuff? I want for the only person that has access to my stuff to be me and those I specifially give access to, not some company that can access it and says they dont just for some revelation in the news that they do spy on your stuff...oh but theyre not specifically pilfering your fiules, they just need to have access for diagnotic reaosns or to monitor your usage hapbits or something. Uh, no. I am satisfied with my usage habits and dont need some company deciding for me anything. Totally, this is what I should have said. I am not paying one cent I dont have to pay. I have religiously used Dropbox for ten years, at first you could get 3GB for free and you were incentivised to connect as many devices to it as possible and they would give you more space for free. I did that. They didnt give me as much space for all the devices I connected as they promised but a little over 5GB has been very useful. But its not anything like what I would do if there were o limits, like there would be if it was my server, my hard drives, my cloud service that I buy one time and use until it breaks or outlives its usefulness. and then about a year ago now, dropbox up and decided that you have to pay more for any features, and they limited the amount of devices you could access it with to three if you wanted to continue to use the free version. You know something thats great about owning my own truck, no one gets to control how many tools I am allowed to stroe in it. If I want to store three hammers for no apparent reason, I can do it if I want. That is freedom. My freedom is being limited because dropbox isnt having as much success making money and so they are tightening your purse strings in order to get the same servies out of you. Oh yeah, its only five bucks, or ten, or twenty. But you know, most regular things now cost at least five bucks at the grocery store things that used to be less than ten are now fifteen or more and so its remarkable how little you can buy before you are just making a quick run to the store and walk out paying 50 dollars for tonight's dinner for one. Hey, Raman every single night for the rest of your life anybody? I know a guy that does this, and I would never be able to do that. to finish up with Bos argument that the amount of time and energy and money you save if you just spend more money for windows for example over linux, I agree, it makes sense, but not in every single scenario. Guess what, if youre trying to get things done and you simply dont have money or you only have it because youre extreemly careful with it like me, then you can just let me have my raid file server running linux that isnt my desktop running 24/7 and heating my room up to the temperature of the sun. 
  • there is a solutuion to my cognitive dissonance regarding why i should use raid 5 at all if i cannot totally rely on the raid to not fail beccause it does after all have one point of failure. The whole point of multiple drives backing multiple drives up like in raid 10 is because you have multiple drives so you have lots of space and even speed is increased because of striping, but then the whole process is repeated a second time identically across an equal number of drives, like having two raid 0's right next to each other but in the same system, and they mirror each other like in raid 1. But unfortunately, any raid controller is going to represent a single point of failure. now when i first heard about JBOD, i seariously thought this had to be a joke or something becuse its literally Just-a-Bunch-Of-Disks, and I was like, why wouldnt you just forget raid altogether at that point and create a storage space in windows control panel? 
  • my idea is to create a JBOD with 3 of the 4TB disks instead of all four, then buy a 12TB ironwolf to back the JBOD up. It'll sort of be cold backup storage, but itll be one thing that doesnt require a ton of configuration that will give me redundancy. And some people say that ironwolf drives have more reliability that WD blue drives, which is what I am used to using. Well, I have purchased a couple of Seagate drives in the past, one is a 3.5" 2TB Barrcuda drive and the other was a Free Agent 500GB 2.5" external drive, the second drive I ever purchased. I dont knwo what happened to the free agent but the 2TB Barracuda is still in use, and I have had it nearly 15 years. But I dont trust it anymore even though it still works. I have had more problems with WD, specifically my 3TB  3.5" Blue drive six years back. I have had other drives fail me but I dont know if they were WD. This option of doing the JDBOD with three 4TB disks instead of four allows me to get away with buying just a 12TB ironwolf instead of buying the 16TB for much more money and with extra space I wouldnt need to have backed up. It would make sense to do all four 4TB drives and get a 16TB ironwolf to back them up but I dont need that much space and 16TB is way more than I need and it is more expensive. I dont want to this to be taken as scripture, but it seems to me that one drive with even more memory might potentially be more prone to failure than the smaller drive too. Dont quote me on that. I would only refer to ironwolf in this case anyway because it is merely cold staorge or level 1 redundancy and not my main storgae solution. 
  • explain storage spaces control panel)
  • started the computer up and switched from ahci to raid mode, then from legacy bios to uefi mode, and the omputer would only load to the EFI Shell. So it still has the same problem it had before we flashed the bios.
  • this was the point that doug suggested we try installing ubuntu server over legacy bios, which is supposed to be installed on uefi rather than legacy bios. But he recalled that there are versions that do work with legacy bios. I did some research and everything i found suggested things like using Rufus, and then using the MBR partitioning scheme.  I immediately stopped and thought, yeah we're doing SRAID. I cant use MBR because none of my 4TB drives would ever be recognised as more than 2TB. I did research however if there is any way to use MBR but not have it affect your storage, and basically what it cmae down to is that if you use legacy bios, youre using MBR, if you use UEFI, then you have the option of using either MBR or GPT. So I dont see any way around using a SRAID. 
  • Talked to my instructor about my JBOD plus an Iron Wolf drive. He kept trying to explain to me that I still have no redundancy. I was confused because I'm like, no i have reundancy because of the ironwolf and hes like but if its apart of the JBOD and I stopped him and said no, theres the JBOD and then theres the ironwolf just using something similar to windows backup to constantly backup the JBOD. Then he explained again that I still dont have redundancy in a way I didnt understand. He then said that the raid itself would have no redundancy and I said, yeah I know. My problem is with RAID itself having a single point of failure because the raid can fail even if the drives dont. Then he said yeah but you have the config file that can rebuild the RAID. And I said I dont trust that. Thats why I am going with the ironwolf. Long story short, he said something that I cant repeat because I was tired and couldnt think when he said it, but somehow he convinced me to go back to using the RAID 5. And then he said that what we can do is after we create the raid, we can then break it on purpose after i transfer all my data to it, and try to bring it back so i can see how it works and whether or not itd work. And this will be okay because all my stuff would still be on the old drives, i wouldnt delete them until after we run this test. 
  • I explained the problem with BIOS and MBR being tied at the hip to doug and he said something interresting. "Yeah, but why when you switch to legacy bios mode on the new motherboard, why then does the board recogize all 4TB on all four drives? I asked if he recalled the old motherboard recognized the drives being 4TB each and he said no. I thought, well, that is a greta questuon. This is where he suggested we try to update the raid controller itself rather than just the BIOS. So now I'm looking for an AMD SB950 driver to then flash the motherboard RAID controller with. I asked Nate earlier about this and he said there was nothing. Oh well. Here we go. I have a feeling this wont solve it either, haha. I am totally willing to do SRAID at this point because Im going to also buy an IronWolf to back the RAID up, so this really is just achademic at this point. Cracking my knuckles, here we go. 
  • in the process of looking for this, i made a discovery that was immediately crucial for SRAID, so we just sort of forgot about what I was doing because I found a website where someone was having similar issues to me except they were doing an SRAID, and while reading it they said that they couldn't get the SRAID to work and another user commented saying that they needed to install ubuntu server on a non-raid drive in order to make it work. 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!on unix.stackexchange "

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Why is uEFI firmware unable to access a software RAID 1 /boot/efi partition?
Ask Question


BIOS firmware can boot a BIOS formatted /boot partition installed on a software RAID 1 pair no problem. It can even boot from a /boot installed on LVM volume that lives on a software RAID 1 pair.

But with a uEFI install, /boot/efi has to be on a non md partition or the firmware can not access it.

Is this a flaw with uEFI firmware? Or is the problem with how Ubuntu sets up /boot/efi on software RAID devices? Could it be a flaw with how GPT partition tables present software RAID to the firmware?

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!For reference, I’m using: Ubuntu Server 14.04.3 64 bit mdadm RAID setup from the 'Manual' option in the partitioner.

So we immediately went to strategizing how we were going to overcome this hurtle because since the beginning Doug has been saying that we'll just install the OS on the RAID because you can do that in Linux, but not with Windows. However, the problems with the UEFI firmware has caused this need for another drive for the OS to re-emerge.
So I'm going to buy a 2490GB Kingston drive for 22 bucks as soon as I get paid, was going to do 125GB but apparently that ones 23 bucks, so...okay, Ill spend less and get more, thanks Kingston.
I want to get going on the project and was going to grab a magnetic laptop drive from home that was out of use and I had three, one of them is being used to back up all my stuff from my 4TB drive I decided to use for this RAID, so Id rather not add that and risk losing the data to a random drive reformat. The other two unused laptop drives turned out to both be out of commission completely. So I clered off a jump drive I use for situations like these and am using that for the OS until I get the kingston drive, and somehow we're going to mirrir the data or something.
Came back to class tuesday night, and here we go. Then I spent the whole class being stuck by the boot utility. Since ate installed win10 on one of the drives meant for the RAID, it kept trying to boot into that the second I swtched out of SATA RAID mode and back into AHCI mode. I kept turning all the other boot options back to disabled, and even though I pressed f10 to save and exit immediately, it wasnt recognizing the boot order settings. It recognized that I switched to AHCI, so I dont know what was going on there. I did it a second time and then escaped and formally selected save and reboot, and it stuck that time. I was still stuck on the boot problem for some reason but then we spammed all of the potential keys, we dont know which one allowed us to access One-Time Boot Menu but with this board, the MSI 970a-G46, we spammed F10, F11, and F12, and it worked. Once we got there we selected the second USB that had the Ubuntu Server OS files on it.
I ended up sharing this second USB drive with Jack in my class because he and his group last thgrusday were doing a sort of similar project that they were having terrific difficulties with and were not able to solve, and even Doug looked at it and couldnt figure it out so both yesterday night and tonight Jack put the computer back together to try and create a dule boot with win10 and ubunutu server. They were having a lot of similar problems to us. They could install windows but not linux, or this happened or that, I think he ended up reseating the RAM at one point but he also took off the heatsync and re-pasted it too just in case. And last ngith he got ubuntu server installed and then installed windows or the other way around and the installation fo one corrupted the other. Tonight, he spent hours troubleshooting this before doug discovered that the hard drive is in MBR partitioning scheme and windows wants to create all these partitions, and MBR will only recognize up to four and with both linux and windows installed, it wanted five or six, I dont remember. So doug told him to reformat for gpt and he thought he did. But he installed both windows and linux, they both worked, he could switch betweent eh two and discovered from windows that the dosk was still in mbr. So, what gives? He spent about forty five minutes trying to figure out what the deal was with that even though it was working before doug came around and said that actually what happened is that it actually is in mbr but the reason why its working is that he created three partitions for windows 10 to install on and tricked it into just putting those partitions together even though the system still saw them as seperate partitions, and then when he did linux, it created another partition for linux and then a left over space was left blank and unable to be accessed because MBR only recognizes so much disk space. However, some of it could still be recovered and added to the linux partition from withi linux or else if you did it in windows, the linux partition would go nuts.
SO I used a USB that belonged to a friend and put ubuntu server on that so jack and I didnt have to share
We started loading ubuntu server, selected the language, the keyboard layout, skipped the network setup, I only stopped at the page for Guided Storage Configuration to use entire blank USB disk I cleared my stuff off of until I get the kingston drive, and then changed the disk to have ubuntu server installed on from one of the raid drives to my USB since after all I want that drive for the raid. ;I dont want to lose a whole 4TB to a 500MB OS. Then i turned LVM group with LUKS off because im not doing any virtual machine stuff with this system. And then at custom storage, turned it on with the space bar.
On this next page, Storage configuration, one of my 4TB drives needed to be reformatted for the RAID to work, and so we selected that drive and a small drop menu had the reformat feature. You have to rpess okay at the bottom of the screen before it officiallly reformats. We also switched my USB drive from exfat to ext4, we selected RAID 5. Under Format and/or mount md0, which is how we tell it where we're mounting the RAID, in Linux you have to tell it where to mount the drive because windows does C drive and D drive and so on, but you have to specify that in linux.
we changed the boot order back to UEFI ubuntu which now shows up since its been installed, as well as USB  install.

We skipped all the defaults so it could install really quick.

On Wednesday after plugging in at home, connect to network log in and then do !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"ip a s" does the same as ipconfig, see if there's an ip address, can write it down so we can SSH into it tomorrow. leave server running

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Fix explanation of RAID 5 on parity, explain how parity works and get rid of compression

Fun point to learn, parity uses checksum. what it does is it puts all the data in columns of 0s and 1s and however many total bits you have, there's a last column at the end that counts as one more bit, but what this last bit is representing isnt actual data, but a key that is used to resurrect lost data or to check data that could be in error. So this checksum uses this last bit on the end to represent even or odd numbers of bits. If you count a whole row of bits across all the columns, and you add the them up, is that number even or odd? If its odd, then you write a one, if its even then you write a zero for that last bit, so if you lose one for example, by losing a whole drive in your raid array, it can use all the data across the drives plus the parity bit at the end that represents a whole bunch of data for each bit and it can do math to basically add up what the number for the data would have to be in order for the parity bit to make sense in the resurrection of the lost variables or data on the bad drive, or in the case of ECC memory, there's a 50% chance that the EEC checksum bit will catch the errored data and be able to correct it and that 50% chance for catching errors is way better than 0% chance.

!!!!!!!!!!!!!!!!!!!!!!!!!1go back to raid 10 and explain nested raids???

FYI JBOD offers way more features than storage spaces does in Windows control panel, because you can incorporate either striping or mirroring or both>need to look that up to make sure.
and there are raid 50's, 60's, even 100's. i think that's something like a raid 5 with a total raid 5 stripe-i think, or raid 6 with a mirror, or a 10 with an extra mirror-again, i think, I'm probably wrong somehow. And there's a 51 raid too, a raid 5 with a mirror attached I think. actually nesting introduces tiers of raids. so a raid 100 means there's three tiers because each digit of a raid number means that there's two raids on top of each other or more. so raid 10 is a raid 0 stripe of two raid 1's, and a raid 100 is literally a raid 0 that's divided up into two, which is divided up into two more each, then of a raid 1.

Now on Wednesday night, I left my desktop running that had all the files saved on various drives connected to it, and I left it with Anydesk running with an Unattended Access password st up on it so I can activate the remote connection from one side, and I left the server running and connected to my router via ethernet.
I'm in class now and we don't have much time left, we were going to do a project day today because there's a vacation thing at the end of this week, so we don't have to come in Thursday this week. But one of us was struggling understanding RAID and so our class went over it for anyone interested and I sort of got sucked into it even though I already knew most of what he taught us today. So now I'm spending the last 50 minutes of class doing my project, starting with SSH-ing into my file server that's at home.
(Don't know if I mentioned that i installed the minimized version of of ubuntu server, which only does command line so there's no desktop or GUI. its just a UI, no graphic to go with the user interface.
I SSH'd into the server through my VPN, but as I used PowerShell before to do the Pi Hole Pi VPN project, I thought id just open PowerShell. Nope. There's like five different versions apparently, I selected PowerShell x86, which is an older version of PowerShell, and does not recognize the SSH command. And there's also PowerShell ISE and SIE x86. ISE is for writing scripts apparently.
I'm connected now, and I used Anydesk to connect to my desktop so I could at least start backing up my data from drives I don't trust very much.
Doug explained that I don't have a file server yet. Right now I just have a server OS with a RAID. I need to set up the file server before I can set up its connection to other computers on a network.
I selected the file containing all my backup data, right clicked, went to properties, sharing tab...but then Doug stopped me and said there was a better way. We then tried to install SSH on my desktop at home from class over my laptop through the VPN. This was unsuccessful.
Then we tried transferring files from the desktop to the server by_________________, which seemed to work for a little bit until Linux stopped us because it didn't like the way I had named many of my folders, which Windows was fine with. If I had to guess I would say that it might have had a problem with special characters I might have used like underscores and dashes but not necessarily those.
I discovered I was unable to access my router though my VPN. as long as I'm in my Wi-Fi network there's no problem but through the VPN is no good. it will have to be repaired but for now i can remote into my desktop anytime i want as long as its running and Anydesk wasn't closed and I can then access my desktop remotely and then my router from there.
I got my IronWolf in the mail, as well as my Kingston SSD for the Ubuntu Server OS and for installing Samba which will turn my server into a file server which can serve files to network clients.
I attached the hard drives to my desktop, and then (picture case and talk about my concerns before buying it), and cable managed and then since I was concerned about my router access and other things???????, I brought the server into class yesterday Wednesday 2023-10-25. Jack worked with me. He's probably the smartest one in the class. He wanted to understand what I was doing too so he was interested. Since I had trouble remembering the username and password we have Ubuntu server on the jump drive it was installed on, we decided to just go ahead and reinstall it from scratch on the Kingston SSD. I always write down usernames and passwords for this very reason, but we did something like user and password for the credentials. And then I forgot what it was. And we were in a hurry because class was already over when we configured this. I came in class and Doug reminded me what he set it to, and it was the same username and password we give student use network devices in class. Whatever, starting over so it doesn't matter. This time I'll write everything down.
I thought for sure starting over meant we'd have to remake the RAID 5 too but to my surprise and to Doug's delight so he could shove it in my face that Software RAIDS really are reliable, we had wiped everything and even plugged the drives back in to different SATA ports and the system still recognized the RAID and so we didn't have to remake it at all. We simple continued passed creating the RAID and selected it for use on my server.
We reinstalled Ubuntu server, I updated it before installed Samba just in case, we installed Samba and there was a bunch of configuration involved.
Installing Samba is a file server software that makes any server into a file server, and it sort of fetches files and serves them to clients.
I went home and went into my router settings and gave my server a static IP address in my router settings. Then on both my desktop and laptop, I went to a File Explorer folder and right-clicked on This PC, and selected Map to Network Drive. This brings up a window to select the drive letter that you want your computer to identify this drive as, then you type "\\" and IP address of the file server, then another backslash and the folder name you want to map directly to at that IP address, no spaces. reconnect on sign in will already be selected. But make sure to select to connect using different credentials. This makes it so whenever you try to access this file server, windows doesnt try to log in using your windows credentials. Press finish and then a window for entering your network credentials for the server will appear. This is the login for Samba. Its important to keep the login for the server and the login for samba seperated. When you want to mess with the server itself, such as installing samba for exampl,e, you want the server credentials. If you want to access the files, then you want the credentials for samba that you created when you install samba. I was confused by this until just now. I wrote these credentials down and didnt know which scenario I would use either so this is what theyre for. After you input the credentials for mapping the network drive, a new folder will open for the file server whcih you can add stuff to. Start transferring files.
Now i had some particulars I wanted for my server. I am sharing it with my mom and I often want to send her stuff and absolutely hate having to load things on jump drives and go over there and make sure I grab my jump drive before I leave and all that. I can send it and she gets it when she wants to use it. And thats it. Its like using Venmo instead of going to the store or bank and taking out cash just to pay a friend back for something. So much less trouble. So I also created a user account for my mom, but her user doesnt have access to any of my stuff because she often fumbles with what she does on the computer and I dont want all my photography randomely deleted one day and not even she knows how it happened. However I will only give her access to her folder and I gave myself access to both her folder and mine, so its a simple matter to transfer stuff back and forth.
I also mapped her folder to both my desktop and laptop and then on my iPhone I simply went to the Files app, and under the browse tab, I selected the botton at the top right of the screen, which displays a circle with three dots in it. then I selected the connect to server option, I didnt have to worry about backslashes here, which is a good thing because IOS simply doesnt provide them here anyway. So I typed in the server IP address, on the next page it simply showed both mine and my moms folders. So now I can move everything off dropbox and just use my server.
this isnt done quite yet. Now I need to set the server up to automatically back the raid up to my 12TB IronWolf. Doug wanted me to look up "rsync" with "cron".
I went to start studying it and thought i should SSH into my server to get started, and I actually ran into this problem with the Pi Hole and I completely forgot about it so I dont know if I write about it but I am going to write it here for good measure. Its an SSH fingerprint mismatch. What happened is that because we used SSH on my laptop to access the file server once already but it was for the installation that was on the jump drive which is no longer in use, my laptop was expecting to connect to that installation and the fingerprint was different. So it have me the warning "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed." To fix this, on my laptop I simply went to my C drive, users, clicked on the user, at the top of the folder was a subfolder called ".ssh", I entered that and inside were two files, one was Old Hosts and the other was Known Hosts. You delete that Known Hosts file and then retry the SSH connection and it should work this time.
I tried again and this time it asked me if I wanted to create a fingerprint. I said yes.
Now moving on with RSYNC with CRON. Actually, RSYNC stands for Remote Sync and is not really a backup tool but can be used as part of a solution to for backup. This is the loophole I gather that I'll be using as according to a google search, when you combine it with other tools such as CRON, it can be used for backup. So it's not actually a backup tool, instead it actually transfers files and synchronizes them. And CRON stands for Chronos, derived from Greek referring to time. But in its full expansion, CRON is like Chronicle without the 'h'. Some will say that CRON stands for 'command run on notice', but that's not it. So put together, RSYNC with CRON will give me a backup solution. And so when you hear someone say CRON job, you know they're referring to a command line utility time based job that you can create for various tools including RSYNC. And what's nice about RSYNC is that it won't just transfer over a bunch of files every time there's a change, it'll sync them so any small changes on one side reflected over to the backup side, which requires less bandwidth.
Another   good thing to know is that are different types of backups called GFS or Grandfather backups, Father Backups, or Son Backups. This can seem a little confusing because these backups are--I think, more commonly known by their more direct names. A Grandfather Backup is the same as a Full Backup taken once a month or on some regular basis that isn't everyday, usually more like once a moth. A Father Backup are usually Incremental or Differential Backups that happen in a more common interval such as weekly. Incremental Backups are just making regular changes or additions on a regular basis and are small and frequent while Differential Backups are very similar to Incremental Backups except they are much larger and less frequent, maybe like every week or something by comparison. And Son Backups are Incremental Backups which happen every day, treated like additions or changes to the previous backup whether it was another Incremental Backup or a full Backup. 

In cron we set minute, hour, day of the month, day of the week, and month of the year, and I did wednesday at 3am every week so we typed 0 for the first minute of the hour, space, 3, space, * because we arent setting for one day of the month, we actually want a certain day of every week, however wednesday every week could land on any day of the month in the future so you put asterisk to mean 'everything'. And youll notice that all the text above this command is in blue, that is because theres a hash mark at the beginning of each of those lines which tells the program to ignore those lines. So when you type the minute hour day etc., do not include a hash mark or else cron will ignore it and wonder why you didnt set the min hour etc.
cron opens in a text editor of your choice so when you enter the configurations you want, you have to save them rather than press enter. You save by pressing control + S if editing in the Nano text editor.
also, class is ending so we want to start cron backing things up right now, and if we get the file tranferring going and then I sleep my laptop and leave, my ssh will end and because it ends, the file transfer will end as well. So doug showed me that if you type "screen" and that'll give you another screen where you can carry out tasks in ubuntu started from an ssh that will not end the task when the ssh ends. When you first use screen, press eneter to switch to the new screen, to detatch from that screen press ctr + A and then ctr + D. To then return to it again, do another "screen -r" command and itll bring me back to where it is.
"sudo rsync -aP /mnt/raid /mnt/ironwolfe" starts the manual backup instead of waiting for it to run the command later. Itll do it again at 3am but...whatever. Captial P in command means progress bar within the command line

In a cron job, the day of the week can be represented as either 0-6 or 1-7, depending on the system and the cron implementation. Here's the typical mapping:0 or 7: Sunday
1: Monday
2: Tuesday
3: Wednesday
4: Thursday
5: Friday
6: Saturday

So, both representations are commonly used. You can use either 0-6 or 1-7 to specify the day of the week in your cron job, and it should work as expected on most systems.

I conclude that at the end of this, I had spent a total of nearly $500 on my file server. The DIY case was between $60 and $90, I bought two 4TB hard drives for about $100 or $120ish, the first computer to be used for the project costed $25 but had no power supply, and then when that didn't work I spent between $40 and $50 on a new one, which also didn't have a power supply but also didn't have a CPU fan and might also not have had any RAM either. So Nate added that to his costs for building the computer for me. Then I bought the IronWolf drive to back up the RAID for about $200, or maybe it was a bit less because of a nearly perfectly timed sale.

This has been Truncat3d 00000000111100010100110______________end of line

Thursday, November 9, 2023

2023-11-09 - Turn Off System Sounds In Windows 11

 This is a quicky, but one that has been annoying me for years. Every time Microsoft gets it's greasy fingers on your computer environment, they change a whole bunch of your preferred settings back to what they want instead of what you want. Not very customer friendly. I'm not a Linux guy, even if I can appreciate Mac, Linux and Windows different approaches, we have to pay for Windows to give us things we specifically don't want, and so I am sort of rooting for Linux or some other OS to take Windows' place. And as soon as that happens you'll get PC's that cannot take any other installation other than Windows.  

Anyway, so changing our system sounds up from a mute or a very low, non-heart-attack inducing volume is one of those things Windows resets to their liking about every month or something like that. I did find something online that could help with this. I have to wait and see if it stands the test of a Windows update reconfiguration. 

But if you go to C:\Windows\System32 > click on SndVol.exe > click on the System > go to the dropdown menu and select "No Sounds" Sounds icon > then click apply. This worked on my Windows 11 laptop. Alternatively, when I clicked on the same file in the system32 folder, the Volume Mixer just came up as can be expected, so just click on the icon of the device you want to mute, in this case the System Sounds icon, this will bring up the Sounds window where you'll find the same dropdown menu to select No Sounds. Click Apply. 

This stops any un-muting or volume change for system sounds. It does of course do more than stop the volume from changing away from a low volume and instead disables the function altogether. However, after about ten years, I finally realized I had zero needs or desires to hear system sounds. 



And being an audio guy, I have all my electronics connected to a mixer, funneling to my audio system, and I have the knobs and faders all set to the setting I prefer according to the devices that use those channels on my mixer, so only moderate volume changes from the devices themselves are needed beyond that. But when Windows reconfigures the volume to full and my knobs are set a certain way, I'll be sitting pretty in my room, while quietly working on my computer most times, when all of the sudden a blaring bell ringing noise bursts from all directions of my my surround sound system and scares the crap out of me. 

So congratulations Microsoft, the beginning of me losing interest in your product is finding the need to download all sorts of tools and utilities that change Windows to function the way I want rather than what others decide for me.

This has been Truncat3d 00000000111100010100110______________end of line

Tuesday, October 31, 2023

2023-10-30 - Writing A Script/Batch File for instant sleep mode on Windows 11

 So I like using hibernation primarily for all my laptop needs because it saves the most power without totally shutting down and also recovering all of my work, saved or not. The problem with this is that sometimes I just want to use the bathroom for a few minutes and either want temporary security or some power savings but still get the benefit of recovering within a second or so, which even if hibernation is fast, it still takes like fifteen or twenty seconds. Sometimes I want to attend to something and I don't have much time or there's a fleeting idea I want to explore before it's gone. Maybe I'm keeping notes during a lecture. Occasionally speed is irreplaceable and loss likely. 

I had this idea, not having written any scripts successfully in my life but having been highly interested, I decided I wanted something that I could put my laptop into sleep mode instantly with, without a bunch of button clicks, without having to navigate through tedium to reach it. I don't want to just use Windows + L because that just logs you out and only after a few minutes does it go to sleep if you have that feature set to do so. 

My interest in scripts came from back in 2016, I read some tech article somewhere that talked about the crazy stuff you could do with scripts, and it told this story about this one dude in an office that had moved on to a new job and his former co-workers had the job of cleaning up his computer and desk for a new hire. So they were going through his computer and were stunned at the discoveries they made. Apparently there wasn't anything about his job that he didn't find some way to automate using scripts. 

Basically he'd come to work in the morning, and he'd turn on his computer, log in and click on the first script which in turn then activated a number of other scripts to run, and one of the first ones to start was to tell the coffee machine in the break room to start brewing. You se, the coffee machine in this break room had a LAN port because it was a smart coffee maker, an IoT device (internet of things). And this coffee machine ran Linux. So he had a script that communicated with that coffee machine and within the time that it took this guy to start his computer and get the scripts running to the time that he got to the break room, a cup of coffee will have been poured. 

When he got back to his desk, the scripts will have already automatically filtered through all of his work emails to check for any key words in those emails so that whatever the request in those emails was, he had keywords and phrases programmed into the script so they would recognize them and automatically reply back with the necessary information. These are just a few examples of the things this guy had running on scripts. It seemed to me when I first read this article, the guy basically didn't have to do anything because he had scripts doing it all. I also thought I probably won't take it to that level, but it'd sure be cool to learn how to do that because there are plenty of things that I would like to do with a simple double-click or keyboard stroke from the desktop. 


https://www.stuff.co.nz/technology/digital-living/99164024/a-programmer-wrote-scripts-to-secretly-automate-his-job
These articles also had a GitHub link for some of the scripts this guy wrote: Click Here

Anyway, I found instructions online for how to put my laptop to sleep at the click of an icon. They said to right-click on the desktop > go New > Create Shortcut > in the shortcut window as the location of the item, type "powershell.exe -Command "Add-Type -AssemblyName System.Windows.Forms; [System.Windows.Forms.Application]::SetSuspendState('Suspend', $false, $false)".

This uses the suspend state method to put your computer to sleep. Click Next > name the shortcut something like Sleep Computer > click Finish.

So now I had a shortcut icon on my desktop for putting my laptop to sleep. I double-clicked it and it did nothing. I don't know why this didn't work, I did some research and it pointed to the possibility of administrator permissions getting in the way. It suggested to right-click on it and go Run As Administrator. I did that and it still did nothing. Every time I clicked on it, the CMD or PowerShell window would appear for about a half second and then nothing would happen. 

The next attempt from my brief online research was to open a text document > enter "@echo offrundll32.exe powrprof.dll,SetSuspendState 0,1,0" > save the file and change the file extension to a ".bat" > and windows will ask for confirmation when doing this because randomly changing any file extension can cause a file to become unstable. But this is a common thing and a .bat file can be created from a text document from something like notepad. Then my instructions told me to right-click on this .bat file > go Create Shortcut > put this on your desktop > double-click on it, and it should put my computer to sleep. 

This time when I clicked on it, a command prompt window even more briefly appeared and still did nothing. However, with more research I found something I had forgotten about shortcut icons. 

Shortcut Icon Keyboard Command

In my further research I sort of accidentally re-discovered that when you create a shortcut icon, and then right-click on it > go properties > select the shortcut tab, you can then go down to the Start In section of the window and in the Shortcut Key field type the keyboard shortcut you want to use in order to activate whatever that shortcut leads to. So you don't type what keyboard buttons you  want to use, instead you decide what keyboard shortcut you want to use, which can be tricky because you don't want to override another keyboard shortcut, especially if you commonly use it, but if not then it doesn't matter, jsut keep in mind when maybe you suddenly need it, it's used for something else other than it's default use. And when you create the kayboard shortcut, you hold the keys in the order that they'll be typed. So for example, "Command + Alt + S" which is what my research suggested for my sleep mode command, you'll simply hold the command key, and then also the alt key and then also the S key and let go. The field will indicate the key combination recorded and you press Apply >OK, and you're finished. 

This didn't really matter all that much since I hadn't actually gotten this command to put my laptop to sleep. I asked for some hel from Jack, one of my classmates who's been coding for years and is well versed and is one of the smartest in the class. He looked at the code and wondered if perhaps it would activate when clicked, but the mouse moves subtly after clicking so it never actually suspends. So he looked up the code for activating the command for a few seconds after double-clicking on the .bat icon for the command. This also didn't work. He couldn't help me for very long though because he was busy with an assignment.

Then

Doug came at the end of class, after everybody was gone, as usual, to try and help me figure this out. He looked at the code and discovered that I had made a mistake when typing the code. I had mistaken the "runll32.exe" command for "run1132.exe". They look exactly the same except that in the first one, there are two lower-case L's and I typed two number 1's (one's) instead. 

This still didn't fix anything. Then Doug did some research. He found this: A file that puts the computer to sleep when clicked? - Microsoft Community

"

A file that puts the computer to sleep when clicked?

I am trying to make a desktop icon which when clicked puts the computer to sleep. I tried searching a lot but didn't find anything. Maybe a batch file would work by changing the icon of a shortcut. But I can't find a command that puts windows 10 to sleep. The code I found hibernates it. 

(@echo off

echo %time% timeout /T 1 && rundll32.exe powrprof.dll,SetSuspendState 0,1,0
echo %time%
)


I need something like this on the desktop

A way to pin this to the taskbar would also be helpful!

You'll need a third-party tool to put your computer to sleep using command-line.

Try one of the many options discussed here:

Put Windows Into Sleep or Hibernate Mode Using Command-line

How can I put the computer to sleep from Command Prompt/Run menu?

You can use the PsShutdown tool [download] from Microsoft Sysinternals.

The command-line to put your computer to sleep is below:
"

Using PsShutdown

PsShutdown is part of the PsTools suite from Microsoft Windows SysInternals which every sysadmin would have in their toolkit. Here is the command-line to place the system into sleep mode.

psshutdown -d

When you use this tool for the first time, you’ll see a EULA screen which you need to accept to proceed."

So we installed the PsShutdown tool. I didn't really understand this next part. Shouldn't the tool then go in a file directory where a bunch of shutdown tools are? According to Doug, no. It just needs to be dropped in a common file path. So he went to This PC > C drive > Windows > System 32, grabbed it and dropped it there. 

Then we edited my previous .bat file to simply say "psshutdown -d -t 0", saved it, I put it in my Downloads folder just to put it somewhere out of the way. 

There was no reboot. We tested it. Upon clicking the file the command prompt appeared very briefly and the laptop went to sleep. I created a shortcut now this time so that I could then click on the shortcut >hold Alt and press Enter > go shortcut tab > enter the Control + Alt + S key combination again > pressed Apply and Ok and then tried the shortcut and it worked. Then I tried the keyboard combination while the desktop was selected and it worked. 

I would like to say that this is it. And it should be. But the next day it stopped working. I moved the shortcut I left next to the main .bat file in the Downloads folder back to the desktop, it still didn't work. What am I doing wrong?

I don't know if what I did fixed anything or if it did, what it fixed. So I deleted the shortcut and re-copied the .bat file, pasted it to the desktop and re-configured the keyboard combination. When clicking the shortcut, it works. And then when I did the keyboard combination while the desktop is selected, it did nothing...for about 3-5 seconds and then it went to sleep. What's the deal? I don't know. But it's working again for now. 

This has been Truncat3d 00000000111100010100110______________end of line

Tuesday, October 24, 2023

2023-10-23 - NTFS Folder Mounting...In Windows 10

 This is a problem I have had for several years now, and had no answer for it--until now. Many games these days take up 80 GB or even 140 GB and there's no apparent sign that this is going to stop climbing anytime soon. A lot of the time the C drive isn't a terribly large drive. 

My C drive on my desktop is a 256 GB SSD. Nine years ago when I was only just thinking about how to spend some money I had, and I had always wanted to break into the SSD world, I decided I was going to switch the C drives in my laptop and desktop to SSD's. But I also had this idea that to stretch my money further, I wouldn't buy anything of any extravagance that I didn't have to. 

I also bought a lightweight, compactly broken down tripod for my camera for about $70 around this same time, and there was a slightly taller one when fully extended for ten dollars more. The salesman at the Photography store debated with me why I wouldn't just get the bigger one, it's only ten dollars. And I was not going to just arbitrarily spent money I didn't have to spend. They guy was like, but it carries a half pound more than the smaller one and this and that. I said, well, I only need it to carry the camera I have. So this doesn't matter to me. He just didn't seem to get it. 

So to save some more money, I decided that since there was only room for one drive in my laptop that I would buy a bigger drive for it and keep to the maximum size that I thought I would ever actually need for my desktop. Since it had many TB of hard drives plugged into it, there was no reason to spend tons of money on another 500 GB SSD for it, so I did one 500 GB SSD for the laptop and one 256 GB drive for the desktop. I have however come to regret this a few times. But I now have the capacity to switch to an M.2 NVME so it doesn't matter that much anymore. 

So when I decided I wanted to play World of Warships a few years ago, I tried to install it and it simply wouldn't run the game. The Wargaming application opened but the actual game itself would never open. I had installed it on an external drive because I simply didn't have enough space on my C drive for it. So I have become a big fan of World of Warships but have never once played it. I have been watching The Mighty Jingles, Flambass, and Flamu for years now with regularity, but never even opened the game. 

I can only assume that installing the game on any other drive but the C drive is the reason it never worked. However, now I know what would have likely solved it. In Windows, there is a feature for NTFS Folder Mounting. 

So imagine you have a new drive, and for gaming in particular, and other reasons, when you mount the drive in Disk Manager, you right-click on the unpartitioned space disk > select New Simple Volume, next > enter size, next > and instead of selecting drive letter, there's a bullet selection for assigning a letter, another for simply leaving the drive without an assigned drive letter, and there's also an option for "Mount in the following empty  NTFS folder. Select this option.  Click Browse > click the plus under the C directory, and then you can expand Users > then (user name) > Desktop > and click New Folder > name this folder whatever you would like to name it for the purpose it will be fulfilling > click OK. 

This will trick the computer into allowing you to save anything meant for the C drive on another drive, such as programs and applications including your excessively large games. 

This has been Truncat3d 00000000111100010100110______________end of line

Monday, September 11, 2023

2023-09-11 - Pi Hole VPN + Pi Hole Project pt2

 Shortly after I learned you could make Pi Holes and block ads with them, I wondered if there was a way to do this on-the-go. I thought for a while: if I was willing to make a second one and keep it on me at all times with a hotspot and a power bank, then yes, I could do that. But its really not very practical. I'd have to always wear a backpack everywhere I went if I wanted to use this for my phone. It never occurred to me that people already thought of this and solved it. Otherwise I might have simply looked it up. Instead I wondered this for years because this was step number three hundred and forty two and I was still on step two with an available Raspberry Pi on my possession.  

At school, there's a security guard for our building, stationed right outside our classroom. He's also in my class even though he's busy starring at a monitor and talking to all the girls getting certified in the medical field, who apparently all have crushes on him. One day I told him that I was making a Pi Hole on Thursday, which is project day each week. He asked me if I was going to make a VPN and I was confused, looking strangely at him, "Yeah, another time though. Why?"

"Cus you can tunnel into network from outside to use the Pi Hole on your phone from anywhere." 

With shock, I said, "Ah, now I'm making one as soon as I'm done with the Pi Hole!" 

Finally got it all set up and plugged into my network and hoped that nothing stood in the way of my making a VPN. Night before Project day, Doug, my instructor, told me to make sure to bring my Pi and my router. So I made sure to set an alarm and brought them. I didn't actually get to be able to start working on the Pi VPN until there was about maybe an hour left of class so I was feeling like this was going to take a couple project days again. I tried looking things up online to understand it. But I had this feeling if I decided to do it one way, Doug would just have me start over to do it some way he thought was better because it'd be more challenging or something, kind of like what happened with the Pi Hole. I lost count how many times I reinstalled Raspbian on my Pi, but it ultimately caused me to brick a memory card, steel another from a Pi Cluster we had in class that no one was using, and then buy two cards in case this happened again. 

The night before I thought, wait, I don't have to buy a second Pi to install the VPN on do I? With the way the installation of the Pi Hole was, it kind of seemed to me that the Pi Hole software sort of took up the whole device. I think it was whenever I typed my Pi's IP address into my browser to access the Pi Hole settings and thought, what does it do if I have other things installed? It's just taking me straight to the Pi Hole interface, it's not even letting me select the Pi Hole deal. So what, if I want to use it for more than one thing, is it just tough luck? Do I need to buy a second Pi for the VPN? My instructor said no when I got to class to work on it. In fact I learned that they work like hand and glove, they were made for each other. What else do I not realize here? Plus there was a question in the VPN installation process that asks if the reason for the PiVPN is to be used with a Pi Hole. 

I did SSH to get back into my Pi again using the network username and password I had given it when installing the OS for the thirtieth time last week. I had forgotten how to SSH in so I had to look that up. I thought if I simply typed "ssh" followed by the IP address that it would grant me access, then Doug whispered over my shoulder as he passed by while running to help other students, "ssh username@ip-address". Oh. I entered this and had to try it a number of times, it didn't seem to want to let me in until I realized I think twice that I was using the wrong password, which once again, does not display in the terminal so you think the keyboard stopped working or something. 

After I got in, we went to https://www.pivpn.io/. And right there on the first page, if I remember right, was two boxes, one right over the other, one was for installation, and the other was for testing (unstable) branch. I figured if I needed to know, I was soon going to do that soon. But we never went back to it. It ended up that I only kept that page open so I could write this. 



Then in the top box with the installation command, "curl -L https://install.pivpn.io | bash", I simply copied and pasted it into the Terminal. My memory is a little hazy here. I think we started it, got most of the way through it, and then maybe accidentally cancelled it so I had to start it over again. It stopped part way through multiple times to ask me questions like did I want to turn this on, is this the right thing, do want this or that, whatever, I forgot all the things. If it helps, I answered yes to all of them except for the ones I answered no to or specified otherwise. Doug said you just hit yes every time and there was something I answered no to. Later I will wonder if this was the cause of a problem I'd have later.  


There was a point around the end of the installation while it was asking a bunch of questions and it asked what port number I wanted the VPN to use, which I would configure both Roberts and my routers firewalls to forward. I selected it and recorded it. 

After it was done installing, I had this fulfilling feeling like "Yeah, I have a VPN, I made it myself!" Then I realized I basically hadn't done anything yet. My instructor then came by and said that I needed to do this, this, and this. I wrote it down after the third time of him listing ten slightly complicated things that I actually didn't understand yet at all. Classmates were looked at me. Well, one of them. But you have to understand that you're an idiot about something until you're not. I finally had my epiphany that allowed me to wrap my head around what he was saying but for good measure, I wanted to write everything down anyway because I knew I'd get home, forget a mundane detail and the whole thing would suddenly make no sense.

But I got home and immediately got to work on it and asked my roommate, Robert, to locate my router on his routers network and set up port forwarding on the port number that I selected during the installation process by the installation process of the PiVPN. But I hit a brick wall. I just keep seeming to run into these things. It's like I am experiencing every conceivable problem that can be had in the making of a Pi Hole and a PiVPN. 

So when I got home I thought I understood Doug's instructions, but life happened. And only after life was out of the way again and I could try to work the problem again, everything he said slipped back into obscurity. I took notes on what he wanted to do and why, but I didn't foresee the confusion I would later have over my the notes regarding something that I didn't understand. I had new realizations that then confused me later. Things similar to, wait, if this, then why this? Did I misunderstand Doug?

At least the part I kind of understood that absolutely couldn't be done in class was sort of understandable. Since my roommate has a router and I now have a router, and mine is configured to work with my Pi Hole, now I needed to create this port forward on both my routers or else this VPN will pass through my router and get blocked by his because ingoing and outgoing transmissions that would go over the VPN would be blocked by his routers firewall. We created the port forward for my router in class, and ny router had to be connected to his routers network in order for my router to be found on his network so the port forward could be applied to the specific device that it was meant to accommodate. It would be pretty unsecure to create this port forward and allow any app inside or outside the network to access the network through it. To plug any potential hole, you create a port forward for a specific app so certain traffic goes through that port only and not only doesn't get blocked by either his or my routers firewalls, but also this port forward has to be specified to only be used with my router on his end so that nefarious outside devices don't take advantage of a gaping hole in the firewall. 

I went into my Asus router, and for my router, I could go to the WAN selection under Advanced Settings > select Virtual Server / Port Forwarding > Enable Port Forwarding under Basic Config > and then below that under Port Forwarding List, was an Add Profile button so I could select what device it was for, what the external port was, what the internal port was, what IP address this port was going to work with, and select the protocol it would be using, either TCP or UDP or both, in which case I want both. One is Transmission Control Protocol and the other is User Datagram Protocol. TDP ensures delivery of data and messages over a network and UDP allows data to transfer over the network without having to verify delivery. You want UDP to be able to skip verification because this takes time, which would otherwise cause more gaming and video streaming lag than there already is. 

These ports refer to the routers firewall. Only certain traffic is allowed through these ports, which can be used to block any number of different kinds of internet traffic from being able to block social media sites, video streaming sites, porn sites, and provides a measure of security from the internet to simply access the network--certain traffic has to be allowed aka whitelisted so that applications will function properly. But firewalls also block many types of attacks because they filter all data that moves through the network and certain ports in a firewall are used for certain things. Imagine a hole in a dam that's a certain size, debris, water and rocks of only such a size can flow through it, anything larger for example would be blocked from passing, although a network firewall doesn't get clogged when something too large tries to go through, and these firewalls don't filter according to file size but rather other attributes and many rules are applied to them to filter certain traffic or even keeping certain users from accessing certain parts of a network.

I had a bonehead problem and it took me all day to solve it. I lost a whole day off to it. I needed to get my roommate to create the Port Forward on his router, and so when he did this, he sent me a list of all the devices on his routers network, even listing the individual devices I had plugged into my personal router. I saw my phone, my old laptop, my new laptop, my desktop, but I didn't see my router anywhere. Nothing that said Asus on it. There were a whole bunch of random devices with names that clearly identified them on the network and they all had an address next to them. I assumed this was an IPv6 address because it had several segments of hexadecimal characters. 

So I spent the whole day trying to ascertain what my routers IPv6 address was. I asked google how to look it up, I asked google what it was if it knew, I asked ChatGPT how to find it, and when I did finally figure out how to get it, it didn't match anything on my roommates list. I thought maybe this was because I found the option under the IPv6 button on the side panel to activate it. I did so and this required a reboot from the router and a progress bar that it was completed, at which point the router froze and I had to risk resetting my router if when I unplugged it for a minute and rebooted that it didn't work properly, and I really didn't want to have to reset it because there has been so much configuration that's taken weeks to do and this would wipe it out instantly. 

Then my instructor texted saying that I didn't need to activate IPv6 to do this. He said something to the affect that it wasn't the best way to do it or something. But finally around midnight I was about to go to bed when I just happened to see on the main page of my router what its MAC address was, and realized it was quite similar to all the addresses next to the network names of all these devices on my roommates routers network. Everything hinged on being able to select my router as the device this port forward was going to be used for, we simply couldn't move on to the next step without doing so because this is the reason why firewalls work in the first place. 

So I matched my routers MAC address with one of the unknown devices addresses on my roommates routers network and I texted him asking if when he finished his nightshift, could he finish the port forward. He texted three minutes later saying it was all done and I was confused. He was at work, it was midnight. He wasn't here to do it. Then I realized he was using an app on his phone to remotely access the configuration settings of his router and he simply applied the changes and used the firewall port my instructor and I decided to use. 

When picking a firewall port to do a port forward with, you want to pick a number between 10,000 and 60,000 because any port lower than that is mostly spoken for. 

So it was done, I was so excited. I actually thought that I was done with the PiVPN for the second time in this project and that everything should work now. I thought this project was finally over and I could move on with other things I've always wanted to do and didn't have a clue where to start. A few days later I realized I was sorely mistaken. I still needed to add my phone and my laptop to the VPN and the WireGuard VPN app had to be installed on my phone and the program on my laptop, then they had to be linked through the terminal on the Pi hole VPN to make it work so the VPN would be working with the proper devices. Plus the tunnel just isn't complete without something to create a tunnel for. The VPN is ready to work but there's only a tunnel once a device is connected at the other end.

One of the things my instructor wanted me to do was to do the "pivpn -a" command in the terminal to add a device. I spent so much time trying to solve the previous problem that I completely forgot about it.  

And nearly a week later when my instructor said that I needed to bring my router into class again to get this port forward and everything finished. Then I had the idea to remote into my desktop at home from my laptop in class, and then SSH into the Pi from there. I thought for sure my instructor would go for it because it adds another layer of difficulty. I would have to use an application like Anydesk or something on both my laptop and desktop, and make sure my desktop stays running while I'm in class because I didn't want to change any sleep settings, I could run a video on loop in VLC player, which almost always overrides any sleep timers or screensavers, except in the case that its time to update VLC player. Then VLC intentionally allows the screensaver to interrupt the video. 

I told this to my instructor and he loved it, all configuration could then be done. Alternatively I could come to class for the first hour, go hone and call him on the phone and he could help me that way. He liked both options but I liked my remote desktop idea so I prepared to do this. 

I hadn't used any remote access applications such as Anydesk or Remote Desktop in a little bit so I forgot that you have to create connection request on one device and accept it on the other device you're trying to connect to. No one was home when I was in class and it was go time, so I asked my instructor if he would allow me to run home and accept the connection, which he did, I ran home and in about twenty five minutes I was back in class and had started the remote connection successfully. 

Doug, came by and helped for about five or ten minutes before being called away again, so meanwhile I was doing research on the next step and also on initiating remote connections without having to accept the connection on the other end. I did find a solution to that. But then the remote connection ended itself automatically due to inactivity. I was so bummed. Next week I guess.

I had a bunch of time to figure out what I was going to do about this auto disconnect due to inactivity and I had looked around on Anydesk to see if there was an option to start a connection from one end of the connection. There was. 


My research pointed me what is called unattended access. I found that under the equivalent to googles hamburger menu in Anydesk located in the upper right corner, there's a button in that menu that says "set password".  This brings up a window to set a password and confirm it, and after this, you get a window of a number of options including enabling two-factor authentication. I think I also had to check one of the boxes on this page to allow an unattended access remote connection, so when I select my desktop from my laptop remotely, and I enter the password, it will grant me access to my desktop without any further ado. I chose to activate two-factor authentication which also required me to download an app on my phone that offered up time-based one-time codes, which recycles about every ten seconds or something like that. So this would be extremely hard to brute force your way into. 

I should add a few things, when you get to this menu to set up the unattended access, don't forget to scroll down because there's quite a few check boxes in this window and you could miss them if you don't realize most of them have to be scrolled to. Plus, I think before my desktop allowed me to create this unattended access, I think I might have had to go into settings and turn on unattended access because I remember opening this window and the new password box was grayed out and couldn't be typed in. 

Now I was ready. I told my instructor about this new development with Anydesk and I was hoping we'd finally make some progress after two weeks of stagnation. It was also kind of discouraging me that on top of not finishing this project for over a month and a half now when I thought it'd only take one class or two at the most, I needed to take a test that I was unable to take, I was having problems with insurance, I was having trouble at work, I had so many projects I wanted to do and couldn't seem to ever get to them, but on top of all that, the Pi hole seemed to stop working, so I had a non-completed VPN with its own issues that needed troubleshooting and then the Pi hole itself had problems too. 

Apparently it was still blocking ads, I went to a Pi Hole tester website that threw a hundred and fifty ads at my Pi Hole, and six were blocked. My instructor said to update the gravity and that we needed to add to the ad list, so I tried updating gravity and this did nothing. I thought I needed to update my Pi Zero, so I did, I thought maybe nothing takes affect till you reboot so I did, and none of that did anything.

 I even tried to solve this on my own. I did research and found a terminal command, "pihole -g --status". I tried it and saw a number of upgrades that errored out. My research indicated that I needed to look into each one individually and I thought at this point, I already don't know what I'm doing and so to dive into a sublevel of things that I was already oblivious to at the surface level would only serve to confuse. I might go back to class to have Doug say I wasted my time. He'll of course say that it's all good stuff to learn but it was unnecessary for this project.  

Then Thursday, project day, finally came again. 

2023-09-22 - update 

I went to class yesterday. It was a bit different, many students didn't come to class. They did tell us at the beginning of the term that we will either love project day or hate it. So Doug was able to focus his attention on my problem way more than usual. I made sure before I left my apartment to open Anydesk and leave something playing on a loop in VLC player. Now in class, I simply opened Anydesk, accessed by device, entered the first password and then the second, and I was in my desktop. I was so thrilled! Compared to last week this was a cinch. I opened PowerShell on my desktop back home and SSH'd from there into my Pi Terminal to complete everything. 

Doug was surprised my Pi Hole wasn't working even though it initially worked great. We checked the amount of Domains for adlists. There was something like one hundred thousand and twenty three. I didn't know any better, that could be great for all I knew. He pointed at that and said, oh yeah well we need to still add those other adlists and update your gravity. I told him I already did that and even updated and upgraded the Pi and rebooted it. But that did nothing to fix the problem. I think at this point he probably said, "Well you only have a little over a hundred thousand adlists." 

Then we updated gravity again just for good measure. And this is where Doug showed me some other adlists for my Pi Hole to block ads with. We updated the ad list from GitHub using Blocklist Project. I went over this part already in part 1. Doug explained to me that I was only using something like 200,000 blocklists and that wasn't that many, and then after installed the Block List Project, Then I head nearly a million. And that is about how much I want to have. If I do any more than that, then there starts to be a sort of negative effect, not only do all ads get blocked with the more adlists you include on the Pi Hole, but they'll start to block regular internet traffic too. So it won't catch all of them but it will catch a great deal of them, maybe most of them. 

It will unfortunately not block any YouTube ads because a few years ago when this got popular enough, YouTube decided they were going to circumvent Pi Holes by internally linking ads to their videos so the ads load as if they're apart of the video and the Pi Hole cannot recognize that. The ads must come from an ad server online that is recognized by the adlists. This isn't quite the end of the story, I have been using AdBlock on my desktop and laptop for YouTube for many years now, and they catch everything on Youtube. I've also started using the SponsorBlock plugin as well, which blocks even those parts of videos you might commonly skip, as though they weren't even there. You can still watch them, the plugin highlights them on the timeline of the video. 

But while we were adding adlists from Blocklist Project, I learned you could download lists for not just more ad protection, but for smart tv bloat, porn, abuse, drugs, phishing, torrents, ransomware, malware, etcetera. I took some of this stuff too but Doug recommended against downloading very many because I am only running a Pi Zero, he can run much more because he's using a Pi 4. He showed me his pi hole interface from home. 

Now that we added more adlists from GitHub, there are now nearly a million domains on adlists.

When we had initially finished installing PiVPN and were finally ready to add users to the VPN, we typed "pivpn -a". This is where you name the user. I got confused about this and thought that I was actually connecting the device with this command, so I went to find the network name of the computer I wanted to add and learned that this did nothing. Now I just and a really stupid name for a VPN user. But this was a later occasion when I had to do this again and couldn't remember how we did it the first few times. And of course I forgot to wrote it down so this is actually future me beaming into the past to add to my past selves entry. 

Doug sort of took the reigns for a bit here and did a whole bunch of stuff really fast so that's probably why I didn't write any of it down because I didn't know what he did and then I was going to address it later but there were too many other things to take care of and it never got done till now. So, clipping my hands and rubbing them together, sighing, and here we go. 

Doug typed the pivpn -a command and named the user for my phone and had me install Wireguard on my phone. The command line asked if we wanted a QR code for a phone and he typed the requisite command to do so. We added my phone to the VPN and created the tunnel. And at first it seemed to work. He then created a full tunnel, which I explain later. It was the end of class and for some reason the I didn't get adblocking on my phone or any internet on my phone either. But there was no more class time so we had to leave it for next Thursday. 

We did get a lot of stuff done, everything was installed for Pi Hole, PiVPN, Wireguard on my phone and laptop. But for some reason there was an unidentified problem with my Pi Hole and another one with the Pi VPN. 

The next week, we logged back in through Anydesk on my laptop to my desktop with unattended access, and it was now a simple effort connecting without driving home to accept connections. Then we would SSH into the Pi through my desktop at home. 

Doug thought for sure my roommate didn't create the port forward correctly but I showed him that the port forward was created, on both routers, using the same port, for both the internal outfacing ports and the external inward facing ports, and were configured to use both UDP and TCP. I had proof, and showed him the screen shot my roommate sent of his router configuration.  

And then I showed him my routers configuration, which showed the same data but in a different way. 

It just didn't make sense that when we turned it on for my phone that my internet was nonexistent. We did research to see what else the problem could be other than just updating gravity and updating and upgrading the Pi Hole, and while I did that, Doug just quickly reinstalled the VPN again for the third time overall since the project began. 

So just to recap since this is important for how we connected my laptop to the VPN, it needs to be understood that we were connected from my laptop to my desktop back at home through Anydesk, and then used my desktop at home to SSH into my server there on the same network. This is crucial information for how we connected the laptop. We had reinstalled the VPN and probably Wireguard on my phone as well, and we updated gravity on the Pi Hole but also added the GitHub adlists and this brough the adlist count up to nine hundred thousand something I think. . After this, we did the pivpn -a command. It gave the option of using a configuration file with a key to connect any device, or just a QR code. We did the QR code for my phone again and this time it worked. There was just a simple error while installing and it was corrected y reinstalling it. Now when connected to my VPN, instead of just bytes of data transferring and no real internet connection and no adblocking, it was blocking ads, and it was transferring data in kilobytes and there was internet. 

This initial VPN tunnel was a full tunnel, and Doug created a second tunnel known as a half-tunnel. I explain this later. And then for my laptop, we were already connected to my desktop at home on my network through Anydesk, and my desktop was connected to my pi through SSH, so when the command line offered the configuration file with the key for the VPN connection to that new VPN user, we simply opened it and copied it to my laptops clipboard, and then pasted it into Wireguard on my laptop and created a half tunnel and full tunnel with this key. (I write another post on how to do this the more traditional way for any device that isn't a phone and can't use the QR code, but I didn't run into this until late December and had to figure it out on my own so I did and wrote another blogpost on that more traditional method where I had to install a utility to transfer that key from one computer to another using SSH on the same network, but it does the SSH and command line commands for you in the background and you simply use this utility on the GUI to do it the way many would consider the easier way if scared by the command line. I am a technical person so once I had done both, neither of them seemed any harder than the other. 

Now to alleviate confusion, we didn't need to create a VPN connection for my desktop at home because the whole reason for using a VPN is if the device you're using isn't in the physical network you want to use it in. So to securely do this, you use a VPN tunnel, a virtual connection to that physical network that is encrypted. This is unnecessary for my desktop which is already there inside the network so no VPN user and no tunnel necessary. 

To explain the full tunnel and half tunnel, the full tunnel causes that every last bit of your connection to the internet and your private network to pass over the VPN between the physical network and the devices accessing it. The down side of this is that the internet is slower and you don't need to make queries online from your phone to go through your VPN back home to then be made from your private physical network and then sent back to your phone over the VPN. This is overkill if all you want is adblocking and things like that, or you're using your phones data plan so it's using its own secure network to the internet through the phone company, so you can access your private network through this as well. For that reason, you create a half tunnel and that way only access to the private network and DNS requests and things like that will go over the VPN. This is done by simply changing what IP address ranges the tunnels use. _________________________________________. So if a half tunnel is so much more efficient why would you ever use a full tunnel? The full tunnel is far more secure than the half tunnel. But it's not always necessary. The kinds of times when you definitely want to use a full tunnel are times lie when you're at the airport or some other public network where your private physical network back home could be hacked into by the tunnel you're using on an unsecure network with other devices whos' users may have bad intentions.  

2023-10-24 - update

I started to notice today that ads were not being blocked on this one game I play on my phone. I thought, well its been around a month or so and I haven't updated gravity yet so I accessed my Pi Hole interface through my browser over the VPN. I navigated to Tools on the side panel > Update Gravity > I pressed the big, long, blue Update button at the top of the page. This took about a minute or so. 

And then at the bottom of the page, I discovered a button flashing white and red that said Pi Hole Update Available or something like that so I inquired about just pressing this button to update and learned that it just leads to the location of an update although I don't know this because I didn't press it. 

I decided to SSH into my Pi Hole. This took me some time though because I couldn't remember the IP address for my Pi Hole, which is also my DNS server, so I entered ipconfig into the Power Shell window before entering an SSH session and I didn't see an IP address I recognized. So I did an "ipconfig /all" command. I also didn't recognize it there. Then I checked my WireGuard app on my phone and didn't find it there and then checked the bookmark in my browser for the Pi Hole interface, and then I found it. 

So I typed the network name of the Pi Hole with an @ symbol followed by this IP address, all one word. Then I typed "pihole -up", which updates the Pi Hole software. This took several minutes. And there were many times that it seemed to sit for some time and so I thought it was done just to discover that all of the sudden it started updating another thing and added lines to the page and kept scrolling down and down with more processes. And so I started to wonder how I would know that it was actually done. I researched online and found something that when it was done, it would the command line would return to a normal prompt, which in my case suddenly happened. It displayed a green "Pihole1@pihole" which it always shows when ready to accept a new command. 

There were two other indicators too though, one would show in the Pi Hole log file, which you would access from the command line, would show a message indicating a successful update. The other was that the Pi Hole website would show the latest version of Pi Hole. 

Up next for large projects, a Fileserver running an old motherboard, with Linux Ubuntu Server, using RAID 5.

This has been Truncat3d 00000000111100010100110______________end of line

Wednesday, September 6, 2023

2023-09-06 - What's Client Side Scripting?

 I'm studying for the A+ Fundamentals exam and have decided that a great way to to remember it is to explain it here and have it as a post. So I had a question asking me about a scenario where I encounter a users computer where the browser won't open an application. 

There are multiple choices of what to pick for why this is the case, but in my particular case, I was curious why one of the options to pick from wasn't applicable as it seemed very applicable. So you have to pick three of the four available multiple choice questions: What should you check, is the browser approved for that browser and version, is the script blocker enabled, is the browser version the most recent, is client side scripting enabled?

Now I was using a website that does practice tests, and I had it set to let me know what the answer was and if I got it right. It told me no because I had picked Client Side Scripting. But I was having trouble understanding why Client Side Scripting wasn't an answer. 

My instructor explained at first that its not as common anymore, did some research and realized it used JavaScript and we still use JavaScript quite widely. JavaScript is a computer language that allows you to write applications--don't quote me because apparently there are scripting languages which JavaScript is, and they are not the same as languages you would write a program like Photoshop or iTunes with. But for all I know, Facebook probably uses it. However, to help me understand this, my instructor guided me to an old game website called Cookie Clicker, or https://orteil.dashnet.org/cookieclicker/. 

We right-clicked on the cookie and selected Inspect, which brought up a window in Google Chrome, showing all of the code that makes up the site. Something I didn't know was that not everything was in one language. A bunch of things were not scripts, but ever so often, there was a line that started with the word 'script', and there would be a dropdown arrow, that when you click it a bunch more code would appear and this was the script. And these were in the JavaScript language. 

Anyway, he used this to explain that this code was an example of Client Side Scripting, it wasn't encrypted and the scripting was accessible on my end. He explained that you don't want applications to be done in Client Side Scripting because then people could just click inspect and go through the code and manipulate the code and make it do all sorts of things, and it wouldn't be secure--because the scripting is being done on the clients end on the computer itself rather than on a server somewhere online before it's served to the user. 

It's also good to know that Java is an interpreted language but JavaScript, despite being tricky because it has script in the title, it's actually a Markup Language such as XML, HTML, etc. So JavaScript runs client side as a Markup language online, so it runs scripts through HTML. JavaScript runs through HTML, and there are script injections throughout, in JavaScript. And the purpose of these particular scripts is that they do calculations client side of the server vs the client when on the internet, and those calculations will produce more HTML for the web page. 

This has been Truncat3d 00000000111100010100110______________end of line

2026-05-10 - MWB constant reconnect fix and all other ultimate fixes until now for my whole home computer setup

     Now I was fixing a problem with a couple things that annoy me on my laptop. Every once in a while I manage to fix another little thing ...