I was filling out applications for an IT job and looking for criteria jobs wanted and trying to apply the ones which applied to me onto my resume so I might stand out better. One application said something about an applicant knowing how to check backups. It struck me when I read that and thought, how hard can that possibly be? I could add that right now assuming certain factors. So I looked up online what that entails and as I was looking it up the thought occurred to me that surely there must be applications that dp this. How could a person possibly verify an entire backup is legitimate when it could contain many thousands or millions of files? And so an application built to do this specifically would have to do this. I just couldn't imagine some dude in a back room using multiple laptops to open tens of pictures and documents and databases at a time per machine and just checking at a glance that yep they opened, and not even the possibility of combing through them to make sure the files are current, or checking timestamps or something. So I was right, there is software that does this and I thought, I may have to install one of these and try them out for myself.
At just that moment. I thought I would run a few checks which now almost two weeks later, I can't even remember what I did, but I was looking at the possibility of doing this for my file server and I think I ran "df -h" and saw something that alarmed me. My roughly 11 TB raid 5 array was only about 3/5ths full, and the Iron wolf 12 TB drive that has basically the same usable space because terabytes in binary units, the word is tebibytes (TiB) and so I guess the space is actually just slightly less than that, basically, both the array and my IronWolf are the same size in usable space, long story short. Anyway, the IronWolf back up the RAID 5. But the IronWolf is full and the RAID 5 array is just more than half full. How can this be?
After some research and help from understanding what Doug and I did when building my file server and how we set up the Cron job implementing RSYNC once a week, we were running a -a flag for archiving, but because I am prone to reorganizing my server occasionally to make things easier to find, and I rename folders containing a lot of files and only occasionally add files, many files were copied and not deleted from their original forms after the changes, and over the course of the last year, it appears none of my files added to the RAID array for the past year have been backed up to the IronWolf because of how many files I changed and even files I deleted from the RAID array and never got deleted. This cast into the light many uncomfortable truths I took for granted and false securities I clinged to. A lot of my stuff was not backed up. I needed some way to find out of RSYNC was even successful in its backup attempts each week, I needed to know whether the IronWolf was full, changes needed to be made to the RSYNC command in Cron so that files no longer on my RAID would get deleted.
Verifying and Troubleshooting Rsync Backups on Ubuntu
I wanted to be able to say I’ve verified my backups — not just that they run — but that they’re actually good. My setup:
-
RAID 5 array with four 4TB drives (live data).
-
12TB Seagate IronWolf drive used as rsync cron job backup target.
-
Rsync runs weekly at ~2 AM.
Previously, I’d restored files once after an accidental deletion (took all day, stressful). But I’d never formally checked the backup’s integrity.
How to Check an Rsync Backup
Finding Paths
To know what to compare, I listed disks and mount points:
This showed:
-
/mnt/raid → live RAID (11T total, 6.3T used).
-
/mnt/ironwolf → backup drive (11T total, 11T used, 100% full).
Problem Discovered
The IronWolf backup drive was completely full even though the RAID only used 6.3T.
Likely causes explored:
-
Rsync cron job missing --delete → old/deleted files accumulate forever.
-
Cron job creating dated snapshot folders weekly → duplicates pile up.
-
Hidden trash/recycle folders from Samba/NFS shares.
Investigating Why the Backup Drive Filled Up
After discovering the IronWolf was completely full while the RAID still had plenty of free space, I dug deeper.
Step 1: Find What’s Taking Space
To see the largest directories on the IronWolf:
This shows which folders are consuming the most storage.
Why the Drive Filled
The problem came down to rsync not being run with --delete:
-
When files are deleted from the RAID, rsync leaves them untouched on IronWolf.
-
Over time, IronWolf accumulates every file that’s ever existed on the RAID.
-
New or changed files keep being added, but deleted ones never get cleaned up.
-
Eventually, the IronWolf fills completely even though the RAID only uses 6.3T.
This is not “archive mode.” Rsync’s -a (archive) flag just preserves metadata (permissions, ownership, timestamps, symlinks, devices). It does not keep multiple versions or automatically reclaim space.
So the IronWolf wasn’t “virtually” full — it was literally full with stale files. Rsync won’t reuse that space unless the exact filenames still exist on the RAID and get updated.
Solutions Explored
This deletes any files on IronWolf that don’t exist on RAID, freeing up space and making it a true mirror.
Cron Job Fix
To prevent the problem from happening again, the cron job should include --delete:
Runs every Wednesday at 2 AM, keeping IronWolf in sync with RAID.
Safely Running Rsync to Clean the Backup
Once the problem was understood — IronWolf full due to old deleted files not being removed — the next step was to safely sync the RAID and IronWolf.
Step 1: Verify Mount Points
Check the live RAID and backup drive locations:
Confirmed:
Step 2: Dry-Run Rsync Inside Screen
To see what would be copied or deleted without affecting any files:
Start an interactive screen session:
Run the dry-run rsync with sudo:
-
-a preserves permissions, ownership, timestamps, symlinks, etc.
-
--delete flags files on IronWolf that don’t exist on RAID for removal.
-
--dry-run shows actions without changing files.
Detach from screen: Ctrl+A then D
Reattach later: screen -r rsyncdryrun
Step 3: Dry-Run Result
The dry-run returned nothing — meaning:
-
IronWolf already contained all current files from RAID.
-
No new files on RAID to copy.
-
No files on IronWolf matched for deletion (aside from previously deleted files).
The backup is functionally in sync, but IronWolf is still full due to stale files from earlier rsync runs that didn’t use --delete.
Step 4: Real Sync to Free Space
To clean IronWolf and mirror RAID exactly:
-
Removes all stale files no longer on RAID.
-
Ensures IronWolf matches RAID exactly.
-
Can be run inside the same screen session to survive SSH disconnects.
Notes on Screen vs Tmux
-
Screen: simpler, widely available, enough for a single long-running command.
-
Tmux: more modern, supports multiple panes, extra flexibility.
For this setup, screen was used to run rsync safely and maintain the session if the SSH connection drops.
Step 5: Cron Job Update
Ensure the weekly cron job uses --delete to prevent future overfills:
Runs every Wednesday at 2 AM, keeping IronWolf a true mirror.
Understanding Why IronWolf Was Full
After running the dry-run with:
-
No output appeared → IronWolf already contained all files currently on RAID.
-
The problem: IronWolf still showed 100% usage.
-
Explanation: rsync had never actually deleted old files because previous cron jobs ran without --delete. Stale files from RAID deletions had accumulated.
Solution: Run the real sync to clean up IronWolf:
This will:
Runtime Notes
-
Dry-run finishes almost instantly.
-
Real sync with only deletions should take minutes to a couple of hours depending on the number of files.
-
Copying full 12 TB would take much longer (tens of hours to days depending on interface speed).
Checking What’s Using Space
Use sudo to see everything, including root-owned files:
Top-level folder sizes:
20 largest items anywhere:
Example output from your system:
-
The main space hog is /mnt/ironwolf/raid
-
This folder contains both current RAID files and old deleted files because rsync wasn’t using --delete.
-
Running the real sync with --delete cleans it up.
How Rsync Archive Mode Works
-
-a preserves permissions, timestamps, ownership, symlinks, etc.
-
--delete controls whether old files on the destination get removed.
-
Current cron job (rsync -a) preserves deleted files as a stop-gap for accidental deletions, but leads to the drive filling up over time.
IronWolf File Access via SMB
You can create an SMB share for IronWolf just like RAID:
Samba configuration snippet:
Reload Samba:
Map it on Windows (use a different drive letter than RAID, e.g., T:):
-
Drops directly into /mnt/ironwolf/raid, where all backups reside.
-
Can stay connected to RAID share (S:) simultaneously.
This setup lets you inspect IronWolf files directly and verify space usage before running the actual cleanup sync.
Accessing IronWolf Backups from Windows via Samba: Notes for Blogpost
Goal: Access IronWolf backups from Windows, opening directly in the folder containing backups.
Backups location on Ubuntu:
/mnt/ironwolf/raid
Checked with:
Output showed the raid folder where all backups live.
Samba Share Concept:
A Samba share exposes a Linux folder over the network so Windows can access it. Existing RAID share already exists; a new share will point to IronWolf backups. Multiple shares from the same server can coexist.
Creating the Share:
No new .conf file is needed. Use the existing Samba config:
/etc/samba/smb.conf
Open for editing:
Structure of Samba Config:
Existing Config Check:
Adding IronWolf Share:
-
Add block at bottom of smb.conf:
-
Save and exit nano (Ctrl+X, Y, Enter).
-
Reload Samba:
-
Map share on Windows (example with drive T:):
Optional Parameters:
-
create mask, directory mask → set default file/folder permissions.
-
force create mode, force directory mode → enforce permissions.
-
write list → explicit write access.
-
follow symlinks, wide links → allow symlink navigation, including outside the share.
Key Lessons Learned:
-
Absolute paths in Linux allow direct access to files for editing.
-
Samba can serve multiple shares from one config file.
-
Share blocks define independent access and permissions.
-
Minimal share block is enough for basic access; optional extras improve control over files and symlinks.
IronWolf Backup Samba Share — Continued Setup Notes
Objective: Provide read-only Windows access to /mnt/ironwolf/raid for checking backups and recovery.
Minimal safe Samba share configuration (read-only):
Optional parameters (not needed for read-only, can be added later if full access is required):
-
create mask = 0664
-
directory mask = 0775
-
write list = christian
-
follow symlinks = yes
-
wide links = yes
Apply changes:
Mapping on Windows:
-
The “Folder” box in Map Network Drive wizard → \\<server-ip>\<share-name>
-
Authentication: username = Samba user, password = set via smbpasswd.
Troubleshooting Mapping:
-
Sections: Server info (version + PID), Services (current connections), Locked files.
-
Seeing the version and no errors → Samba is live.
-
Check share visibility on Windows: Explorer → \\192.168.50.153\
-
Confirm share registered in Samba:
Key Lessons Learned:
-
Minimal read-only share is sufficient for backup access.
-
Absolute Linux paths required in smb.conf; Windows sees only the share name.
-
Samba defaults may hide explicit settings in testparm; absence doesn’t mean misconfiguration.
-
Restarting or reloading Samba ensures changes take effect.
-
Samba authentication requires the specific Samba user/password, not system passwords.
Interpreting Your Samba & Windows Drive Mapping Output
1. Samba config load output:
-
Confirms Samba read /etc/samba/smb.conf successfully.
-
Config syntax is valid (no errors).
-
Weak crypto allowed → older SMB protocols/ciphers permitted (fine for home setup).
-
ROLE_STANDALONE → Samba is just a standalone file server, not a domain controller or member server.
-
This does not confirm your new share exists — check further in testparm or smbclient output.
2. smbclient usage:
-
Replace <server-ip> with your Samba server IP (e.g., 192.168.50.153).
-
Replace <username> with the Samba user you added (sudo smbpasswd -a <username>).
-
Prompts for Samba password (not your Linux password unless you set it that way).
-
Output lists all shares available on the server — look for IronWolfColdStorageBackup.
3. Windows prompts:
-
Password for [WORKGROUP\<username>] → WORKGROUP is default Samba/Windows workgroup; <username> is the Samba user.
-
Enter the Samba password you set.
-
If you never set a Samba password, run:
4. Mapping in Windows (folder box vs path):
* → asks for password each time, no saved credentials.
5. net use visibility quirks:
-
Mapped drives are session-specific.
-
Drives mapped in standard user sessions won’t appear in elevated PowerShell or admin CMD.
-
To make them visible across sessions:
-
Open regedit
-
Navigate: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System
-
Add DWORD 32-bit → EnableLinkedConnections = 1
-
Restart PC
6. PowerShell scripting notes:
-
@echo off → works in cmd/batch files, not PowerShell.
-
PowerShell interprets @ as splatting operator → causes Unexpected token '@' error.
-
For PowerShell scripts, use Write-Host instead of echo and avoid @echo off.
Next Practical Steps:
-
Confirm share appears via:
-
On Windows, check Explorer → \\192.168.50.153\ for IronWolfColdStorageBackup.
-
Map network drive using share name, decide if you want persistence or always-prompt passwords.
-
Adjust for session visibility if mapping in admin/PowerShell vs standard CMD.
Samba side
-
sudo testparm passed → config likely OK.
-
Path exists and permissions correct → share should be live.
-
Shares can point to different drives—backup drive can coexist with RAID shares.
-
Windows mapping issues
-
Your net use command uses the share name, not the username, in \\<ip>\<share>; /USER:<username> supplies auth.
-
“Nothing happens” or multiple prompts = credential mismatch or Windows caching old creds.
-
Mapped drives are session-specific; use /persistent:yes for auto-reconnect.
-
Stepwise check
-
On Ubuntu: smbclient -L //<ip> -U <username> → confirm backup share appears.
-
On Windows: \\<ip> in Explorer → see the share listed; try connect with Samba username/password.
-
Clear all cached creds: net use * /delete /y and remove Windows Credential Manager entries for the server.
-
Retry mapping: net use S: "\\<ip>\<share>" /USER:<username> <password> /persistent:yes /verbose to catch errors.
-
Tips
-
If credentials fail, create a dedicated Samba user for the backup share.
-
Use Explorer mapping with “Connect using different credentials” if command-line fails.
-
Check logs on both sides for auth failures or network issues.
✅ This setup can work; your problem is almost certainly Windows credential handling or exact share name mismatch, not the Linux/Samba config.
TL;DR: You don’t need to include the full file path in net use, just \\<IP>\<share name>. You don’t have to remove other shares; Windows can map multiple shares from the same server if credentials align. Your “nothing happens” issue is likely Windows caching or a stale mapping for that drive letter. Clear the S: mapping, optionally clear credentials, then retry net use S: "\\<ip>\<share>" /USER:<username> <password> /persistent:yes /verbose or map via File Explorer. Multiple shares are fully supported.
This has been Truncat3d 00000000111100010100110______________end of line