In 2021 I shared my backup script that I was using to back up my web server. At the time I was using VestaCP, man a lot has changed since then. For one, I switched over to HestiaCP which was a great decision, but this post isn’t about that. The old script was a quick little script I punched out that had some flaws in the organization of the files.
The original idea was to back up everything daily, which is still the idea. You see, HestiaCP, like VestaCP, only keeps a limited amount of backups based on the package you assigned the account. Sure you COULD set this to 1000 backups, but that means its all stored online and takes up quite a bit of space. So I created the script to make a daily backup, so if anything went wrong after the 3 days of backups on the server had passed, I could still recover it from my offsite backups.
The original format was this, backup all files into a backup folder based on the Year, Month, and Date, this would prevent issues where you have millions of files in a single directory. This worked out great for the last several years, but this created 365 folders per year, and every folder had a backup of every account… so lots of locations to choose from. But I recently decided I wanted to keep better track of my accounts. so instead of doing “/backupDir/Year/Month/Day/” I decided to move this around a bit and make it “/backupDir/Username/Year/Month”, this would only create 12 folders per year, per account. This creates fewer folders as long as you have less than ~30 accounts.
Now this concept sounds great, but my original script just grabbed files based on the date the file was created, as all I was doing was transferring all the files to the same directory. Now, I wanted to grab only the *.tar files (some *.log files were now being created) and I wanted to separate them via Username and upload them to separate directories based on those usernames.
After about an hour of trying to figure out the best way to make it work, I was having a really hard time with it. At this point, I decided to try and give ChatGPT a try, and at first, I was trying to update my already created script, I mean I just needed to reformat how I grabbed the file list. After another 30 minutes of asking ChatGPT to help me fix my already-created script, I was starting to give up, and then it dawned on me. Why not just have ChatGPT create the script from scratch? I mean a lot has changed in the last 3 years since I first created the script, and none of my coding formats matched ChatGPT, so why not have ChatGPT just re-write the whole script?
You would think that even that would be simple, and if you are good at prompt generator, it might be, but for me… not so easy. I had to rephrase my prompts several times before ChatGPT would give me anything similar to what I was looking for.
I only had 3 criteria for this to work the way I wanted it.
- I needed the script to grab the files based on their user accounts, in the format username.Year-Month-Date_Hours-Minutes-Seconds.tar (example: mkoster.2024-01-13_04-33-01.tar)
- I needed the script to create a directory on my remote server based on the user name – “/backupDir/username/year/month” (example: /home/sshUser/backups/mkoster/2024/01)
- I needded the script to SCP the file from the /backup/ directory on the server, to the correct user folder on the remote server.
Now this gave me a good base, but it had to be tweaked. I was getting an error off the first run, which basically told me it was expecting 3 sections to the file names, but it only found 2. This was caused by the fact there were files called username.log in the directory, and I asked the script to get the username off the files that were username.date.tzr so when it came to the log files, it only had two sections instead of three. So again, I needed to refine my request:
When I tested the new output ChatGPT gave me, it grabbed the latest file which is what I asked, but the result was not quite what I was looking for. There was a file in there that was old, about 2 years old, I did not want this to be selected, so again, had to refine it a bit.
With all that minor tweaking, I finally had a great script, and it only took me about 20 minutes to have ChatGPT finalize the script, make a few custom modifications, test and deploy. Here is the final result:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
import os import subprocess from datetime import datetime, timedelta def select_latest_file_for_each_user(backup_folder): latest_files = {} now = datetime.now() for root, dirs, files in os.walk(backup_folder): for fname in files: if not fname.endswith('.tar'): continue username, date_str = fname.rsplit('.', 2)[:2] date = datetime.strptime(date_str, '%Y-%m-%d_%H-%M-%S') if now - date > timedelta(hours=48): continue if username not in latest_files or date > latest_files[username][1]: latest_files[username] = (os.path.join(root, fname), date) return latest_files def create_remote_folder(sshUser, sshHost, sshPort, username, year, month, sshDir): cmd = f"ssh {sshUser}@{sshHost} -p{sshPort} 'mkdir -pv /home/{sshUser}/{sshDir}/{username}/{year}/{month}'" subprocess.run(cmd, shell=True) def upload_files_to_remote(latest_files, sshUser, sshHost, sshPort, sshDir): for username, (filepath, date) in latest_files.items(): year = date.year month = str(date.month).zfill(2) create_remote_folder(sshUser, sshHost, sshPort, username, year, month, sshDir) dest_dir = f"/home/{sshUser}/{sshDir}/{username}/{year}/{month}" cmd = f"scp -P {sshPort} {filepath} {sshUser}@{sshHost}:{dest_dir}" subprocess.run(cmd, shell=True) if __name__ == "__main__": # Replace these with your actual values backup_folder = "/backup/" # server backup directory sshUser = "sshUser" # ssh user for remote server sshHost = "sshHost.com" # remote server URI or IP sshPort = "22" # ssh Port of remote server sshDir = "backups/websites" # no starting or trailing slash latest_files = select_latest_file_for_each_user(backup_folder) upload_files_to_remote(latest_files, sshUser, sshHost, sshPort) |
This script did not gro much, the old script was only 38 lines, this one was 50, and now I have backups based on Usernames instead of Date alone.