diff --git a/README.md b/README.md index ce943fe..f9e9c9a 100644 --- a/README.md +++ b/README.md @@ -24,30 +24,20 @@ PIP_REQUIRE_VIRTUALENV=false python3 -m pip install --break-system-packages requ 2. Copy the `jmap-backup.py` file to a directory in your `$PATH` (I suggest `/usr/local/bin` if you're unsure) and make sure it's executable (`chmod +x jmap-backup.py`) -3. Create a configuration file (JSON) to store your API key, destination directory where the backup will be kept, and other settings. You can create multiple config files to back up different accounts or to keep copies on different storage (local, SMB/NFS etc). +3. Export your environment variables. At minimum: -A bare minimum config file must contain at least the `dest_dir` and `token` keys, for example: - -```js -{ - "dest_dir": "/Volumes/storage/backups/Fastmail", - "token": "{your_api_key_here e.g. fmu1-xxxxxx...}" -} +```shell +export JMAP_TOKEN='fmu1-xxxxxx...' +export JMAP_DEST_DIR='/Volumes/storage/backups/Fastmail' ``` -> _The configuration is now in JSON format (prior to v1.1 it was stored as YAML). This change was made because Python can read _and_ write it without requiring the PyYAML module. If you're not comfortable converting your legacy config file to JSON by hand, I suggest using [`yq`][5]:_ -> -> ```sh -> yq -p yaml -o json fastmail.yml >fastmail.json -> ``` - 4. Finally, start the backup by running ```shell -jmap-backup.py -c ~/.jmapbackup/fastmail.json +jmap-backup.py ``` -> If you don't specify a config file with the `-c` option, the program will assume a default path of `~/.jmapbackup/fastmail.json` +> Backup progress is stored in `~/.jmapbackup/state.json` by default. Override with `JMAP_STATE_FILE` or `--state-file`. Progress messages will be printed to the console. When the job is finished, you should see your messages in the destination directory, organized in folders in `YYYY-MM` format. The individual messages are saved as standard `.eml` format files with the filename made up of a datestamp, messageid and subject. @@ -63,24 +53,13 @@ Some have requested a Docker configuration to make it easier to set up and run, git clone https://github.com/luckman212/jmap-backup && cd jmap-backup ``` -2. Create the directories to persistently store your configuration and backups +2. Create the directory to persistently store your backups ```shell -mkdir -p cfg backups/Fastmail +mkdir -p backups/Fastmail ``` -3. Set up your config file. It will be slightly different for Docker since the `dest_dir` can either be a local Docker mount/volume or a network share if one is available to your container. Sample config file below: - -```shell -cat <cfg/fm-docker.json -{ - "delay_hours": 24, - "dest_dir": "/backups/Fastmail", - "not_before": "2020-01-01", - "token": "fmu1-xxxx..." -} -EOF -``` +3. Set your container environment variables (`JMAP_DEST_DIR` should point to your mounted backup path, for example `/backups/Fastmail`). 4. Build the Docker image: @@ -93,40 +72,38 @@ docker build -t jmap-backup . ```shell docker run --rm \ --name jmap-backup-1 \ --v /root/jmap-backup/cfg:/cfg \ -v /root/jmap-backup/backups:/backups \ +-v /root/jmap-backup/state:/state \ +-e JMAP_TOKEN='fmu1-xxxx...' \ +-e JMAP_DEST_DIR='/backups/Fastmail' \ +-e JMAP_STATE_FILE='/state/jmap-state.json' \ -e JMAP_DEBUG=true \ -jmap-backup \ --c /cfg/fm-docker.json -``` - -## Additional (optional) parameters for the config file - -| Key | Description | Example value | -|:------------- |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:------------- | -| `delay_hours` | Back up only messages at least this many hours old | `24` | -| `not_before` | Cut off date before which messages will not be backed up | `2018-06-01` | -| `pre_cmd` | Command (and args) to run prior to execution, most often used to mount some remote storage location such as an SMB or NFS share. It is formatted as an array so you can provide additional args as needed. | (see below) | -| `post_cmd` | Command to run post-execution (e.g. unmount the share) | (see below) | - -Example of pre/post commands in config file (`~` chars will be expanded by Python): - -```js -{ - "pre_cmd": [ - "/sbin/mount", "-t", "smbfs", - "//luckman212:hunter2@nas/backups", "/mnt/jmap" - ], - "post_cmd": [ - "/sbin/umount", "-t", "smbfs", "/mnt/jmap" - ] -} +jmap-backup ``` ## Environment Variables -- Export `JMAP_DEBUG` to `True` to see additional debugging info printed to the console. -- You can export `NOT_BEFORE` to override the default of `2000-01-01` or whatever date is specified in the config file +| Variable | Required | Description | Example value | +|:------------------ |:-------- |:---------------------------------------------------------------------------------------------------------------------- |:---------------------------------------- | +| `JMAP_TOKEN` | yes | Fastmail API token | `fmu1-xxxx...` | +| `JMAP_DEST_DIR` | yes | Destination directory for backups | `/backups/Fastmail` | +| `JMAP_DELAY_HOURS` | no | Back up only messages at least this many hours old (default: `24`) | `24` | +| `JMAP_NOT_BEFORE` | no | Cutoff date (`YYYY-MM-DD`) before which messages are skipped (default: `2000-01-01`) | `2018-06-01` | +| `JMAP_PRE_CMD` | no | Command run before backup starts. Parsed like a shell command. | `/sbin/mount -t smbfs //user:pw@nas/x /mnt/jmap` | +| `JMAP_POST_CMD` | no | Command run after backup finishes. Parsed like a shell command. | `/sbin/umount -t smbfs /mnt/jmap` | +| `JMAP_STATE_FILE` | no | Path to state file used for incremental runs (default: `~/.jmapbackup/state.json`) | `~/.jmapbackup/state.json` | +| `JMAP_DEBUG` | no | Set to `true`/`1`/`yes`/`on` for debug output | `true` | + +Example: + +```shell +export JMAP_TOKEN='fmu1-xxxx...' +export JMAP_DEST_DIR='/mnt/jmap/Fastmail' +export JMAP_NOT_BEFORE='2020-01-01' +export JMAP_PRE_CMD='/sbin/mount -t smbfs //luckman212:hunter2@nas/backups /mnt/jmap' +export JMAP_POST_CMD='/sbin/umount -t smbfs /mnt/jmap' +jmap-backup.py +``` ## Verification @@ -141,4 +118,3 @@ I've been using this script for a few months with good success, but it has been [2]: https://github.com/luckman212/jmap-backup/issues [3]: https://www.soma-zone.com/LaunchControl/ [4]: https://github.com/luckman212/jmap-backup/releases/latest -[5]: https://github.com/mikefarah/yq diff --git a/jmap-backup.py b/jmap-backup.py index df0b509..790249b 100755 --- a/jmap-backup.py +++ b/jmap-backup.py @@ -13,14 +13,15 @@ import argparse import collections import datetime as dt +import importlib +import json import os +import shlex import string -import sys import subprocess -import importlib -import json +import sys -ADDITIONAL_MODULES = [ 'requests' ] +ADDITIONAL_MODULES = ["requests"] # prereqs for module in ADDITIONAL_MODULES: @@ -28,69 +29,80 @@ m = importlib.import_module(module) globals()[module] = m except ImportError: - sys.exit(f"{module} module could not be loaded, check README for installation requirements") + sys.exit( + f"{module} module could not be loaded, check README for installation requirements" + ) + def str_to_bool(s): - return s and s.lower() in [ 'true', '1', 'yes', 'on' ] + return s and s.lower() in ["true", "1", "yes", "on"] -Session = collections.namedtuple('Session', 'headers account_id api_url download_template') -Email = collections.namedtuple('Email', 'id blob_id date subject') -DEBUG = str_to_bool(os.getenv('JMAP_DEBUG')) -NOT_BEFORE = os.getenv('NOT_BEFORE', '2000-01-01') -DEFAULT_CONFIG = '~/.jmapbackup/fastmail.json' + +Session = collections.namedtuple( + "Session", "headers account_id api_url download_template" +) +Email = collections.namedtuple("Email", "id blob_id date subject") +DEBUG = str_to_bool(os.getenv("JMAP_DEBUG")) +DEFAULT_NOT_BEFORE = os.getenv("JMAP_NOT_BEFORE", os.getenv("NOT_BEFORE", "2000-01-01")) +DEFAULT_STATE_FILE = "~/.jmapbackup/state.json" CONNECT_TIMEOUT = 3 -READ_TIMEOUT = 20 +READ_TIMEOUT = 20 + def dbg(*args, newline=True): if not DEBUG: return - s = ' '.join(map(str, args)) + s = " ".join(map(str, args)) if newline: print(s, file=sys.stderr) else: - print(s, file=sys.stderr, end='') + print(s, file=sys.stderr, end="") + def get_session(token): - headers = {'Authorization': 'Bearer ' + token} - r = requests.get('https://api.fastmail.com/.well-known/jmap', + headers = {"Authorization": "Bearer " + token} + r = requests.get( + "https://api.fastmail.com/.well-known/jmap", headers=headers, - timeout=(CONNECT_TIMEOUT, READ_TIMEOUT)) + timeout=(CONNECT_TIMEOUT, READ_TIMEOUT), + ) dbg("Status code (get_session):", r.status_code) dbg("Response text (get_session):", r.text) - [account_id] = list(r.json()['accounts']) - api_url = r.json()['apiUrl'] - download_template = r.json()['downloadUrl'] + [account_id] = list(r.json()["accounts"]) + api_url = r.json()["apiUrl"] + download_template = r.json()["downloadUrl"] return Session(headers, account_id, api_url, download_template) + def query(session, start, end): json_request = { - 'using': ['urn:ietf:params:jmap:core', 'urn:ietf:params:jmap:mail'], - 'methodCalls': [ + "using": ["urn:ietf:params:jmap:core", "urn:ietf:params:jmap:mail"], + "methodCalls": [ [ - 'Email/query', + "Email/query", { - 'accountId': session.account_id, - 'sort': [{'property': 'receivedAt', 'isAscending': False}], - 'filter': { - 'after': start.strftime('%Y-%m-%dT%H:%M:%SZ'), - 'before': end.strftime('%Y-%m-%dT%H:%M:%SZ'), + "accountId": session.account_id, + "sort": [{"property": "receivedAt", "isAscending": False}], + "filter": { + "after": start.strftime("%Y-%m-%dT%H:%M:%SZ"), + "before": end.strftime("%Y-%m-%dT%H:%M:%SZ"), }, - 'limit': 50, + "limit": 50, }, - '0', + "0", ], [ - 'Email/get', + "Email/get", { - 'accountId': session.account_id, - '#ids': { - 'name': 'Email/query', - 'path': '/ids/*', - 'resultOf': '0', + "accountId": session.account_id, + "#ids": { + "name": "Email/query", + "path": "/ids/*", + "resultOf": "0", }, - 'properties': ['blobId', 'receivedAt', 'subject'], + "properties": ["blobId", "receivedAt", "subject"], }, - '1', + "1", ], ], } @@ -104,48 +116,96 @@ def query(session, start, end): dbg("Status code (query):", response.status_code) dbg("Response text (query):", response.text) if response.status_code == 403: - sys.exit("Permission denied: Disallowed capabilities: urn:ietf:params:jmap:mail") + sys.exit( + "Permission denied: Disallowed capabilities: urn:ietf:params:jmap:mail" + ) full_response = response.json() - if any(x[0].lower() == 'error' for x in full_response['methodResponses']): - sys.exit(f'Error received from server: {full_response!r}') + if any(x[0].lower() == "error" for x in full_response["methodResponses"]): + sys.exit(f"Error received from server: {full_response!r}") - response = [x[1] for x in full_response['methodResponses']] + response = [x[1] for x in full_response["methodResponses"]] - if not response[0]['ids']: + if not response[0]["ids"]: return - for item in response[1]['list']: - date = dt.datetime.fromisoformat(item['receivedAt'].rstrip('Z')) - yield Email(item['id'], item['blobId'], date, item['subject']) + for item in response[1]["list"]: + date = dt.datetime.fromisoformat(item["receivedAt"].rstrip("Z")) + yield Email(item["id"], item["blobId"], date, item["subject"]) + + query_request = json_request["methodCalls"][0][1] + query_request["anchor"] = response[0]["ids"][-1] + query_request["anchorOffset"] = 1 - query_request = json_request['methodCalls'][0][1] - query_request['anchor'] = response[0]['ids'][-1] - query_request['anchorOffset'] = 1 def email_filename(email): subject = ( - email.subject.translate(str.maketrans('', '', string.punctuation))[:50] - if email.subject else '') - date = email.date.strftime('%Y%m%d_%H%M%S') - directory = email.date.strftime('%Y-%m') - filename = f'{date}_{email.id}_{subject.strip()}.eml' + email.subject.translate(str.maketrans("", "", string.punctuation))[:50] + if email.subject + else "" + ) + date = email.date.strftime("%Y%m%d_%H%M%S") + directory = email.date.strftime("%Y-%m") + filename = f"{date}_{email.id}_{subject.strip()}.eml" return directory, filename + def run_if(cmd): if cmd: if os.path.exists(cmd[0]): - dbg(f'executing: `{" ".join(cmd)}`') + dbg(f"executing: `{' '.join(cmd)}`") subprocess.run(cmd) else: - print(f'invalid command: {cmd}', file=sys.stderr) + print(f"invalid command: {cmd}", file=sys.stderr) + + +def env_or_exit(name): + value = os.getenv(name) + if not value: + sys.exit(f"Error: environment variable '{name}' is required") + return value + + +def cmd_from_env(name): + raw = os.getenv(name, "").strip() + if not raw: + return [] + return [os.path.expanduser(part) for part in shlex.split(raw)] + + +def load_state(state_path): + if not os.path.exists(state_path): + return {} + try: + with open(state_path, "r") as fh: + state = json.load(fh) + if not isinstance(state, dict): + raise ValueError("state file must contain a JSON object") + return state + except Exception as e: + sys.exit(f"error reading state file '{state_path}': {e}") + + +def save_state(state_path, state): + try: + state_dir = os.path.dirname(state_path) + if state_dir: + os.makedirs(state_dir, exist_ok=True) + with open(state_path, "w") as fh: + json.dump(state, fh, indent=4) + except Exception as e: + sys.exit(f"error writing state file '{state_path}': {e}") + def check_dest_dir(dest_dir, retry=True): dir_exists = os.path.exists(dest_dir) if retry or dir_exists: return dir_exists else: - sys.exit(f"Error: destination path '{dest_dir}' does not exist (you may need to mount it?)") + sys.exit( + f"Error: destination path '{dest_dir}' does not exist (you may need to mount it?)" + ) + def download_email(session, email, base_dir): try: @@ -159,77 +219,98 @@ def download_email(session, email, base_dir): session.download_template.format( accountId=session.account_id, blobId=email.blob_id, - name='email', - type='application/octet-stream', + name="email", + type="application/octet-stream", ), headers=session.headers, - timeout=(CONNECT_TIMEOUT, READ_TIMEOUT) + timeout=(CONNECT_TIMEOUT, READ_TIMEOUT), ) r.raise_for_status() - with open(full_path, 'wb') as fh: + with open(full_path, "wb") as fh: fh.write(r.content) - dbg(f'Downloaded {email.id} {email.date.strftime("%Y-%m-%d %H:%M:%S")}') + dbg(f"Downloaded {email.id} {email.date.strftime('%Y-%m-%d %H:%M:%S')}") except requests.RequestException as e: dbg(f"Failed to download {email.id}: {e}") return False return True -if __name__ == '__main__': - parser = argparse.ArgumentParser(description='Back up a Fastmail JMAP mailbox in .eml format', add_help=False) - parser.add_argument('-h','--help', action='store_true', help=argparse.SUPPRESS) - parser.add_argument('-v','--verify', action='store_true', help='Fully verify backed up emails and redownload if missing') - parser.add_argument('-o','--open', action='store_true', help='Open the configured dest_dir in Finder') - parser.add_argument('-c','--config', help=f'Path to config file (default: {DEFAULT_CONFIG})', nargs=1) + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Back up a Fastmail JMAP mailbox in .eml format", add_help=False + ) + parser.add_argument("-h", "--help", action="store_true", help=argparse.SUPPRESS) + parser.add_argument( + "-v", + "--verify", + action="store_true", + help="Fully verify backed up emails and redownload if missing", + ) + parser.add_argument( + "-o", + "--open", + action="store_true", + help="Open the configured dest_dir in Finder", + ) + parser.add_argument( + "-s", + "--state-file", + help=f"Path to state file (default: {DEFAULT_STATE_FILE} or JMAP_STATE_FILE)", + nargs=1, + ) args = parser.parse_args() if args.help: parser.print_help() sys.exit(0) - if args.config: - cfg_file = os.path.expanduser(args.config[0]) + if args.state_file: + state_file = os.path.expanduser(args.state_file[0]) else: - cfg_file = os.path.expanduser(DEFAULT_CONFIG) - if not os.path.exists(cfg_file): - sys.exit(f"Error: configuration file '{cfg_file}' does not exist") + state_file = os.path.expanduser( + os.getenv("JMAP_STATE_FILE", DEFAULT_STATE_FILE) + ) + state = load_state(state_file) + # load configuration from environment variables + token = env_or_exit("JMAP_TOKEN") + dest_dir = os.path.expanduser(env_or_exit("JMAP_DEST_DIR")) try: - with open(cfg_file, 'r') as fh: - config = json.load(fh) - except Exception as e: - sys.exit(f'error: {e}') + delay_hours = int(os.getenv("JMAP_DELAY_HOURS", "24")) + except ValueError: + sys.exit("Error: JMAP_DELAY_HOURS must be an integer") + if delay_hours < 0: + sys.exit("Error: JMAP_DELAY_HOURS must be >= 0") + PRE_COMMAND = cmd_from_env("JMAP_PRE_CMD") + POST_COMMAND = cmd_from_env("JMAP_POST_CMD") - # parse pre- and post-commands - PRE_COMMAND = [os.path.expanduser(c) for c in config.get('pre_cmd', [])] - POST_COMMAND = [os.path.expanduser(c) for c in config.get('post_cmd', [])] run_if(PRE_COMMAND) - dest_dir = config['dest_dir'] check_dest_dir(dest_dir, False) if args.open: - subprocess.run(['open', dest_dir]) - #subprocess.run(POST_COMMAND) + subprocess.run(["open", dest_dir]) + # subprocess.run(POST_COMMAND) sys.exit(0) - #calculate date window - session = get_session(config['token']) - delay_hours = config.get('delay_hours', 24) + # calculate date window + session = get_session(token) end_window = dt.datetime.now(dt.timezone.utc).replace(microsecond=0) - dt.timedelta( hours=delay_hours ) - # On first run, use 'not_before' if set in config (YYYY-MM-DD); otherwise use NOT_BEFORE var - not_before_str = str(config.get('not_before', NOT_BEFORE)) - dbg(f'Will not archive email prior to {not_before_str}') - not_before = dt.datetime.strptime(not_before_str, '%Y-%m-%d').replace(tzinfo=dt.timezone.utc) - + not_before_str = os.getenv("JMAP_NOT_BEFORE", DEFAULT_NOT_BEFORE) + dbg(f"Will not archive email prior to {not_before_str}") + not_before = dt.datetime.strptime(not_before_str, "%Y-%m-%d").replace( + tzinfo=dt.timezone.utc + ) + if args.verify: - dbg('Verification enabled (this will take longer)') + dbg("Verification enabled (this will take longer)") start_window = not_before - last_verify_count = config.get('last_verify_count', None) + last_verify_count = state.get("last_verify_count", None) else: - start_window = config.get('last_end_time') + start_window = state.get("last_end_time") if start_window and isinstance(start_window, str): start_window = dt.datetime.fromisoformat(start_window) else: @@ -245,7 +326,7 @@ def download_email(session, email, base_dir): full_path = os.path.join(full_directory, filename) if os.path.exists(full_path): - dbg(f'{full_path} ok') + dbg(f"{full_path} ok") else: if download_email(session, email, dest_dir): num_results += 1 @@ -257,34 +338,36 @@ def download_email(session, email, base_dir): num_verified += 1 if num_verified % 100 == 0: if last_verify_count and last_verify_count > 0: - pct = '{:.1f}'.format((num_verified / last_verify_count) * 100) - dbg(f'\rVerified {pct}% ({num_verified} of {last_verify_count})', newline=False) + pct = "{:.1f}".format((num_verified / last_verify_count) * 100) + dbg( + f"\rVerified {pct}% ({num_verified} of {last_verify_count})", + newline=False, + ) else: - dbg(f'\rVerified {num_verified}', newline=False) - dbg('\n') + dbg(f"\rVerified {num_verified}", newline=False) + dbg("\n") - dbg('Done!') + dbg("Done!") # retry failed downloads if failed_downloads: - dbg(f'Retrying {len(failed_downloads)} failed downloads') + dbg(f"Retrying {len(failed_downloads)} failed downloads") for email in failed_downloads: if download_email(session, email, dest_dir): num_results += 1 if args.verify: num_verified += 1 else: - dbg(f'Failed to download {email.id} after retry') + dbg(f"Failed to download {email.id} after retry") if args.verify: - print(f'Verified: {num_verified}') - print(f'Archived: {num_results}') + print(f"Verified: {num_verified}") + print(f"Archived: {num_results}") if isinstance(end_window, dt.datetime): - end_window = end_window.strftime('%Y-%m-%dT%H:%M:%SZ') - config['last_end_time'] = end_window + end_window = end_window.strftime("%Y-%m-%dT%H:%M:%SZ") + state["last_end_time"] = end_window if num_verified > 0: - config['last_verify_count'] = num_verified - with open(cfg_file, 'w') as fh: - json.dump(config, fh, indent=4) + state["last_verify_count"] = num_verified + save_state(state_file, state) run_if(POST_COMMAND)