gogofilemon

gogofilemon monitors one or more directories on a remote SSH/SFTP server, waits for new files to stop changing, and then downloads them to local directories.

Features

• SSH key authentication with known_hosts host verification
• Interactive first-run setup when the config file is missing
• Interactive terminal config editor powered by Bubble Tea
• Configurable reconnect attempts with backoff
• SSH keepalive traffic for long idle periods
• Polling-based remote directory monitoring over SFTP
• Optional per-watch agent mode for near-instant file detection via fsnotify on the remote host
• Optional recursive subdirectory monitoring with local path mirroring
• File stability detection before download (size and mtime must remain unchanged)
• Atomic local downloads using a temporary file and rename
• Persistent download state (bbolt) to avoid re-downloading files after restart
• Automatic state hydration from existing local files on first run
• JSONL application logs with daily rotation and configurable retention
• Concurrent downloads with configurable parallelism
• Windows service hosting support

Requirements

• Go 1.26+
• Network access to the SSH/SFTP server
• A private key the target server accepts
• A valid known_hosts entry for the remote host

Build

make build

The binary is written to ./bin/ggfm.

Build a Windows executable from any platform:

make build-windows

Quick Start

# Build
make build

# First run: interactive setup creates config.json and starts monitoring
./bin/ggfm

# Or copy the example config and edit manually
cp config.json.example config.json
./bin/ggfm --config ./config.json

If no config file exists at the target path, the app launches an interactive first-run setup in the terminal, saves the new config, and starts monitoring immediately.

Configuration

By default the app loads ./config.json. Override with --config.

Example Config

{
  "connection": {
    "host": "sftp.example.com",
    "port": 22,
    "user": "syncuser",
    "private_key_path": "~/.ssh/id_ed25519",
    "key_passphrase_env": "GGFM_SSH_KEY_PASSPHRASE",
    "password_env": "GGFM_SSH_PASSWORD",
    "known_hosts_path": "~/.ssh/known_hosts",
    "reconnect_attempts": 5,
    "reconnect_delay": "10s",
    "keepalive_interval": "20s",
    "agent_path": "/usr/local/bin/ggfm"
  },
  "polling": {
    "poll_interval": "10s",
    "stability_window": "30s",
    "download_temp_suffix": ".partial",
    "max_concurrent_downloads": 4,
    "agent_reconcile_interval": "5m"
  },
  "watches": [
    {
      "remote_dir": "/upload/incoming",
      "local_dir": "./downloads/incoming",
      "file_pattern": "*.csv"
    },
    {
      "remote_dir": "/upload/archive",
      "local_dir": "./downloads/archive",
      "max_depth": 3
    },
    {
      "remote_dir": "/upload/realtime",
      "local_dir": "./downloads/realtime",
      "mode": "agent"
    },
    {
      "remote_dir": "/upload/recent-only",
      "local_dir": "./downloads/recent-only",
      "max_file_age": "7d",
      "max_folder_age": "30d"
    }
  ],
  "paths": {
    "log_dir": "./logs",
    "state_file": "./logs/state.db",
    "log_retention_days": 7,
    "log_level": "info"
  }
}

Config Fields

Connection

Field	Default	Description
host	(required)	SSH hostname or IP
port	22	SSH port
user	(required)	SSH username
private_key_path	(one auth method required)	Path to the SSH private key file. Encrypted keys are supported — supply the passphrase via key_passphrase/key_passphrase_env
key_passphrase	(none)	Literal passphrase for an encrypted private key. Prefer the env form below. See docs/ssh-credentials.md
key_passphrase_env	(none)	Name of the environment variable holding the key passphrase; overrides the default $GGFM_SSH_KEY_PASSPHRASE
password	(none)	Literal SSH password. Prefer the env form below
password_env	(none)	Name of the environment variable holding the SSH password; overrides the default $GGFM_SSH_PASSWORD
known_hosts_path	~/.ssh/known_hosts	Path to the known_hosts file
reconnect_attempts	3	Number of reconnect attempts after a disconnect (0 disables reconnect)
reconnect_delay	30s	Delay between reconnect attempts
keepalive_interval	30s	SSH keepalive interval while idle
agent_path	(none)	Path to the ggfm binary on the remote server; required when any watch uses mode: "agent"

At least one auth method is required: a private key or a password. For each secret, ggfm resolves it environment-first — it checks the env var named by *_env (if set), then the default env var ($GGFM_SSH_PASSWORD / $GGFM_SSH_KEY_PASSPHRASE), then the literal config value. This keeps secrets out of config.json by default. A passphrase-protected key with the passphrase in the environment is recommended over a stored password. See docs/ssh-credentials.md for setting persistent env vars on Windows/Linux/macOS service accounts.

Polling

Field	Default	Description
poll_interval	10s	How often the remote directory is scanned via SFTP
stability_window	30s	How long file size and mtime must remain unchanged before download
download_temp_suffix	.partial	Suffix for temporary local files during download
max_concurrent_downloads	4	Maximum parallel file downloads per polling cycle
agent_reconcile_interval	5m	How often the remote agent performs a full directory scan as a safety net for missed fsnotify events

Watches

Each entry in the watches array defines a remote-to-local sync mapping:

Field	Default	Description
remote_dir	(required)	Remote directory to monitor
local_dir	(required)	Local target directory for downloads
file_pattern	(none)	Optional filename glob (e.g. *.csv); when omitted all files are synced
max_depth	0	Subdirectory recursion depth; 0 = flat (directory itself only); -1 = unlimited; 1+ = that many levels deep
mode	"poll"	"poll" uses SFTP directory listing; "agent" launches the ggfm binary on the remote host for near-instant fsnotify-based detection
max_file_age	(none)	Skip files whose mtime is older than this duration (e.g. "7d", "24h"). Does not stop scanning folders that contain old files. Omit or set to "0" for no limit
max_folder_age	(none)	Skip directories whose own mtime is older than this duration (e.g. "30d"). Excluded folders are not descended into — their direct files and any subfolders are entirely ignored. Applies to both poll and agent modes. Omit for no limit

Duration fields accept the standard h/m/s units plus Nd (days) and Nw (weeks) — so 7d, 4w, 24h, and 1h30m are all valid. The d/w forms do not combine with finer units; use 36h instead of 1d12h.

Paths

Field	Default	Description
log_dir	./logs (relative to config file)	Directory for JSONL log files
state_file	<log_dir>/state.db	Path to the persistent download state database (bbolt)
log_retention_days	7	Number of days to keep rotated log archives
log_level	"info"	Minimum log level: "debug", "info", "warn", "error". The agent inherits this setting from the client

CLI Flags

All flags are optional. Flags override values from the config file.

Mode Flags

Flag	Description
--config <path>	Path to the config file (default: ./config.json)
--version	Print the version and exit
--edit-config	Open the interactive config editor and exit
--hydrate	Force rebuild of state.db from existing local files, then continue syncing
--agent	Run in agent mode (internal use; reads config from stdin, writes events to stdout)
--windows-service	Run using the Windows service host
--install	Install as a Windows service and exit
--uninstall	Uninstall the Windows service and exit
--service-account <name>	(Windows --install only) Service logon account, e.g. DOMAIN\user or a gMSA DOMAIN\account$. Default: LocalSystem
--service-password <password>	(Windows --install only) Password for --service-account. Omit for gMSA accounts. See note below about password handling

Override Flags

Flag	Description
--host <hostname>	SSH host
--port <number>	SSH port
--user <username>	SSH username
--key-path <path>	SSH private key path
--known-hosts <path>	known_hosts file path
--agent-path <path>	Path to the ggfm binary on the remote server
--poll-interval <duration>	Polling interval (e.g. 10s, 1m)
--stability-window <duration>	Required stable time before download
--log-dir <path>	Directory for JSONL logs
--state-file <path>	Path for persistent sync state
--log-retention-days <number>	Days to keep log archives
--log-level <level>	Minimum log level: debug, info (default), warn, error
--watch /remote/dir=/local/dir	Add a watch mapping (can be repeated)

Examples

# Run with defaults
./bin/ggfm

# Use a specific config
./bin/ggfm --config /etc/ggfm/config.json

# Override connection settings
./bin/ggfm --host sftp.example.com --user syncuser --key-path ~/.ssh/id_ed25519

# Add watches from the command line
./bin/ggfm --watch /upload/incoming=./downloads/incoming --watch /upload/archive=./downloads/archive

# Force state rebuild from existing local files
./bin/ggfm --hydrate

# Open the interactive config editor
./bin/ggfm --edit-config

State Hydration

When state.db is empty (first run, deleted, or new deployment), the app automatically compares local files against the remote SFTP listing before starting the sync loop. For each remote file that already exists locally with the same size, it is marked as downloaded in state.db. This prevents re-downloading files that are already present.

Automatic hydration happens when:

• state.db does not exist (first run)
• state.db exists but is empty (no entries)

Forced hydration with --hydrate:

• Runs hydration even if state.db has entries
• Useful for recovery after state corruption or migration from another sync system
• After hydration completes, normal sync continues (no restart needed)

Hydration works the same whether running interactively or as a Windows service.

Agent Mode

Watches can use "mode": "agent" for near-instant file detection instead of periodic SFTP polling. In agent mode, the client launches the ggfm binary on the remote server via SSH exec, sends the watch configuration over stdin, and receives file-ready events over stdout as JSONL.

The remote agent uses fsnotify to watch for filesystem changes, applies the same stability window as poll mode, and emits a ready event once a file has stopped changing. A configurable reconciliation scan (agent_reconcile_interval) provides a safety net for any events that fsnotify might miss.

On each session start, the client sends its state.db records to the agent so already-downloaded files are not re-emitted. After each successful download, the client sends an acknowledgement so the agent suppresses re-emission on subsequent reconciliation ticks.

Agent Mode Requirements

• The ggfm binary must be present on the remote server at the path specified by connection.agent_path
• The SSH user must have execute permission on the binary and read access to the watched directories
• No configuration file is needed on the remote server -- the client sends all settings at session start

The agent is designed for low memory usage: fsnotify buffers are minimized (4KB per watched directory instead of the default 64KB), deferred stability checks use batched timers instead of per-file goroutines, and the Go heap is capped at 64MB via GOMEMLIMIT. Typical memory usage is 15-25MB even with tens of thousands of files.

If the agent fails to start or crashes mid-session, the affected watch automatically falls back to SFTP polling with a warning logged. Downloads always use SFTP regardless of mode. The agent writes its own dated log file next to its binary on the remote host with the same daily rotation and retention as the client.

Logging and State

• Logs are written as JSONL to application.YYYY-MM-DD.jsonl (UTC date) in the configured log directory at the configured log_level (default info)
• Use --log-level debug to see per-file events, reconciliation scans, and other operational details
• A new dated file is opened at UTC midnight; files are pruned after log_retention_days
• Download state is stored in a bbolt database (.db extension) to avoid re-downloading after restart
• Files are identified by (host, path, size, modtime) -- if a remote file changes, it is re-downloaded and overwrites the local copy

Running

# Build and run
make run

# Run directly with go run
make go

# Run tests
make test

Local Test Environment

# Create test workspace with config and both binaries
make testenv

# Run the service against the test workspace
make testenv-run

# Browse the test workspace's state.db (read-only)
make testenv-browse

make testenv builds both ggfm and ggfm-browse into ./testenv/ alongside a copy of config.json.example. The browse binary opens the state.db read-only, so you can inspect it while make testenv-run is actively populating it in another terminal.

Windows Service

The same binary runs interactively or under the Windows Service Control Manager.

Important Deployment Notes

• Use an absolute --config path for service installs
• Use absolute paths inside the config for private_key_path, known_hosts_path, log_dir, state_file, and local watch targets
• The service account must be able to read the SSH key and write to the log, state, and download directories

Build a Windows Binary

From a non-Windows machine:

GOOS=windows GOARCH=amd64 go build -o bin/ggfm.exe ./cmd/ggfm

On Windows:

go build -o .\bin\ggfm.exe .\cmd\ggfm

Install with ggfm --install

The built-in installer registers the service with SCM (display name gogofilemon, service identifier ggfm, auto-start). Run from an elevated PowerShell session:

# Default: runs as LocalSystem
.\ggfm.exe --install --config C:\path\to\config.json
Start-Service ggfm

To run under a specific account, pass --service-account (and --service-password unless it's a gMSA):

# gMSA account (recommended — Active Directory manages the password)
.\ggfm.exe --install --config C:\path\to\config.json `
  --service-account "GLACIERMEDIA\gmsa-ggfm$"

# Regular domain or local user
.\ggfm.exe --install --config C:\path\to\config.json `
  --service-account "DOMAIN\svc-ggfm" `
  --service-password "hunter2"

The account must already exist and have the "Log on as a service" right on the target host (usually granted via Group Policy for gMSA accounts). Whichever account is used, it must be able to read the SSH key and write to the log, state, and download directories.

Password security. --service-password appears in shell history and in process lists during the brief --install run. On shared hosts, strongly prefer gMSA accounts, or set the account via sc.exe config ggfm obj= ... password= ... after installing with a placeholder.

Uninstall with .\ggfm.exe --uninstall (also elevated).

Install with sc.exe

Run an elevated Command Prompt:

sc.exe create ggfm binPath= "\"C:\path\to\ggfm.exe\" --windows-service --config C:\path\to\config.json" start= auto
sc.exe start ggfm

Install with PowerShell

Run an elevated PowerShell session:

New-Service `
  -Name "ggfm" `
  -BinaryPathName '"C:\path\to\ggfm.exe" --windows-service --config C:\path\to\config.json' `
  -DisplayName "ggfm" `
  -StartupType Automatic

Start-Service ggfm

Manage the Service

sc.exe stop ggfm
sc.exe delete ggfm

Browsing state.db

ggfm-browse is a read-only TUI for inspecting the persistent download state. It opens the bbolt file without taking the exclusive write lock, so it's safe to run against a live service's state.db.

make build-browse           # build ./bin/ggfm-browse
./bin/ggfm-browse            # resolves state-file via ./config.json
./bin/ggfm-browse --state-file /var/lib/ggfm/state.db

Keys:

• j / k (or arrow keys) — move down / up by one row
• Ctrl-D / Ctrl-U — half-page down / up
• Ctrl-F / Ctrl-B (or PgDn / PgUp) — full-page down / up
• g / G — jump to top / bottom
• tab — cycle host filter (includes an "all hosts" sentinel)
• / — substring filter on remote path (esc clears, enter applies)
• q or Ctrl-C — quit

The footer shows the filtered count, total entry count, distinct host count, and total byte size. The binary is not built by make build — use make build-all or make build-browse.

Notes

• Encrypted private keys are not currently supported
• Symlinks are never followed during directory monitoring
• Downloaded files mirror the remote directory structure relative to remote_dir
• The app does not delete local files when remote files are removed
• Remote paths are normalized to forward slashes internally; both Windows and POSIX remote servers are supported