Data Refresh

Loads the latest production data into a development FileMaker file using the FMDataMigration tool, run via AWX. The development file’s schema is preserved — only its data records are replaced with production data.

AWX template: Generic - FMS Data Transfer - Refresh
Primary use: Refresh dev with prod data.

A data refresh never targets a production server. The portal enforces this — production servers are excluded from the refresh host list, and the AWX role asserts it at runtime.

All pages in this flow show an Experimental warning banner.

The wizard has four steps:

Select a client
Select the production data file
Select the file to refresh
Review and confirm

The two servers involved are named for what they contribute:

The data host (production) provides the data file whose records are copied.
The refresh host (development) holds the file to refresh — its schema is kept, its data is replaced.

Unlike a data migration, the data host here really does provide the data file, but the same file-first names are used in both wizards so the terms mean the same thing everywhere.

1. Select a client

Choose the client group this refresh belongs to. Only groups with refresh-eligible servers (those with a configured deploy chain) are shown.

2. Select the production data file

The data host provides the production data that will replace the records in the file being refreshed.

This is typically your production server. A warning banner on this step reminds you of the data flow.

Select the data server, then click Load databases to list its hosted .fmp12 files via the FileMaker Admin API. All files known to FMS are shown regardless of open/closed status. Select the file whose data you want to copy.

Choosing the backup to copy from

The refresh never reads the live production file — the job copies a backup of the selected file from the data host’s FMS backup directory. The wizard says so in three places: the step banner, a note at the top of the file list, and the Backup to copy from card below it.

That card offers two choices:

Latest full backup (default) — whatever the newest full backup of the selected file is when you launch the refresh. As soon as you select a data file, this option looks up the data host and shows which backup is currently the newest (path, size, and date), so the default is never a mystery. The backup chooser screen shows the same line. At launch the wizard re-resolves the newest backup and pins its exact path into the AWX job, so the job never repeats the search — and always copies precisely the file the portal resolved.
A specific backup — clicking Next then shows the backup chooser: the wizard searches the data host’s backup directory over SSH — with the same search rules the AWX role uses, skipping any directory whose path contains InProgress, ansible, migration, recovery, or staging — and lists every backup of the selected file with its size and timestamp, newest first. Pick one, or keep Latest full backup (default).

Clicking Next with the default selected resolves the latest backup server-side before the next step loads, so the following pages always show the actual backup file that will be copied. If the data file has no backups at all — or the backup directory can’t be reached — the wizard stays on this step with an error instead of continuing toward a refresh that would fail.

A chosen backup is passed to the job as the file entry’s backup_path, and is re-validated at launch: the path must be one the backup search still finds, so a backup that has rotated out (or a hand-crafted path outside the backup directory) blocks the launch with an error.

3. Select the file to refresh

The refresh host is the development server whose file will be refreshed. Production servers are never permitted here.

Production servers are not permitted as refresh hosts. They are excluded from the list automatically. A banner on this step reinforces that prod is never refreshed.

Select the refresh server (only non-production servers in the data server’s deploy chain are shown), then click Load databases. Select the development file you want to refresh.

The summary card at the top shows a Backup to copy from entry alongside the data file: the pinned backup if you chose one, otherwise the backup that is currently the latest (resolved on the previous step) — with its size and timestamp — so the data file’s hosted path is never mistaken for the file being copied.

4. Review and confirm

A summary card shows:

Data host — production server name, data file, and the backup to copy from: the pinned backup if one was chosen, otherwise the currently-latest backup’s path — each with the backup’s size and timestamp (and, for the default, a note that launch re-checks and pins whatever is newest at that moment)
→ — direction of the transfer
Refresh host — dev server name and file to be refreshed

Disk space

A stacked bar shows the refresh host’s databases volume: what is already used, the backup being copied over (the refresh copies a backup, not the live file, so the backup’s size is what is charged), and — when Backup before refreshing is enabled — the extra copy of the file being replaced. The disk-capacity policy applies: above 90% projected usage the Run Data Refresh button stays disabled until you tick the acknowledgement, and above 95% the refresh is refused. The disk is read at launch, so the decision reflects the volume as it is then, not as the review page found it.

Options

Backup before refreshing (`fms_transfer_backup`)

When checked, the target file is moved out of its database folder under a timestamped name before the refreshed file takes its place. The backup path is shown in the UI, on the refresh host:

<FMS backup dir>/auto-fms-data-transfer-backups/filename_pre-fms-refresh-YYYYMMDDTHHMMSS.fmp12

filename is the target file’s own name without its extension, and the timestamp is the job’s start time (20260804T115910). Note the pre-fms-refresh marker — the role names it after the mode it ran in, so a refresh backup and a migration backup sit side by side in the same folder under different names.

<FMS backup dir> is the refresh host’s FileMaker Server backup directory, read from the Admin API — the same root the Host from Backup — On-Server browser starts from, so you can host the backup back over the refreshed file with the steps in Hosting the pre-migration backup.

These backups are pruned after 14 days. At the end of every refresh or migration run, the role deletes .fmp12 files older than fms_transfer_backup_retention_days (14) from that folder, so a backup that has aged out disappears the next time either job runs against the host. Copy the file elsewhere if you need to keep it longer.

Data migration flags (`fms_transfer_flags`)

Optional flags passed through to the FileMaker FMDataMigration tool. Check any that apply; the selected flags are space-joined and submitted as fms_transfer_flags. None are selected by default.

Flag	Label	Effect
`-ignore_valuelists`	Ignore value lists	Keep value lists from the clone instead of migrating them from the source data file.
`-ignore_accounts`	Ignore accounts	Keep accounts and privileges from the clone instead of migrating them from the source data file.
`-ignore_fonts`	Ignore fonts	Skip migrating font information from the source data file.
`-reevaluate`	Re-evaluate auto-enter calcs	Re-evaluate auto-enter and unstored calculations during migration.
`-rebuildindexes`	Rebuild indexes	Rebuild field indexes after migration instead of carrying them over.

1Password overrides (`op_dmt_creds` / `op_data_ear_key` / `op_schema_ear_key`)

By default the AWX job reads its secrets from the standard entries in the automation’s own vault — fms-dmt-creds_<host> for the DMT account and fms-ear-key_<host> for EAR keys. Creating an entry with one of those names in another vault does not get picked up. Three optional fields point any of those lookups at a specific entry of your choosing for this refresh:

Field	Redirects
DMT credentials entry	The FileMaker account the migration tool signs in with on the refresh host. The entry must provide `username` and `password` fields.
Data file EAR key entry	The EAR key for the production data file on the data host.
File-to-refresh EAR key entry	The EAR key for the file being refreshed on the refresh host.

Each field requires a full op://<vault>/<item>[/<field>] secret reference — the vault must always be named, it is never assumed. DMT credential references must be entry-level (op://<vault>/<item>); EAR key references may name the field holding the key (defaults to password). Leave blank to use the standard entry. Secrets are always pulled from 1Password at run time — the portal never stores them, only which entry to look up.

To use an entry outside the central automation vault, you must grant the 1Password automation user access to the vault holding it — the wizard’s help text names the user (systems-shared by default; deployments configure it via the OP_AUTOMATION_USER env var). The automation cannot read entries it has not been given access to — without the grant, the job fails at run time when it tries to resolve the reference.

A Verify access button under the fields resolves each entered reference and reports per field whether the automation can read it — an entry that is missing, empty, or in an unshared vault shows as not accessible. Use it before launching. The same check runs automatically when you save a template with overrides (an inaccessible entry blocks the save) and on the template’s Verify page.

Save as template

The Save as template card saves everything on this step — servers, files, the chosen source backup (when one was pinned), backup option, flags, and 1Password overrides — as a reusable template under a name of your choice. Saved templates live in the Templates section (linked from the wizard’s start page), are shared with everyone who has access to both servers, and can be verified and run from there without re-walking the wizard. Saving again with a name you already used updates your existing template.

Admin debug panel

Staff users see a collapsible AWX extra vars panel showing the exact variables that will be submitted to AWX. Useful for verifying the configuration before launch.

Job output

After launch, the portal redirects to the job output page, which polls AWX every 4 seconds and displays:

Status — Pending, Running, Succeeded, Failed, Canceled
Duration — elapsed time
Output — stdout from the Wait for FMDataMigration to complete for AWX task. Output longer than 100 lines scrolls within a fixed-height container. No output is shown for canceled jobs.

A link to the job in AWX is shown in the card header. A Cancel button is available while the job is running.

Below the job card, an Options submitted to AWX summary lists the run’s parameters — hosts, files, the exact backup pinned for the copy, backup-before-refresh, flags, and any 1Password overrides. It is read back from the AWX job’s own extra_vars, so it always reflects what was actually submitted (and reappears if you revisit the page later).

AWX variables submitted

Variable	Value
`fms_transfer_mode`	`refresh`
`fms_transfer_source_host`	Data host hostname (production)
`fms_transfer_target_host`	Refresh host hostname (dev)
`fms_transfer_files`	`[{"schema_file": "<target basename>", "data_file": "<source basename>", ...}]` — plus `backup_path` when a specific backup was chosen, and `op_dmt_creds` / `op_data_ear_key` / `op_schema_ear_key` when 1Password overrides are set
`fms_transfer_backup`	`true` / `false`
`fms_transfer_flags`	Space-joined selected `FMDataMigration` flags (omitted if none selected)

Note the schema_file/data_file assignment: in refresh mode the schema comes from the refresh host (dev, the AWX target) and the data comes from the data host (production, the AWX source) — the inverse of migration mode.

History

Completed refreshes appear under the Refreshes tab of the History page. Each record shows the data and refresh hosts, files involved, the user, and the result (OK / Failed / Canceled). Running jobs are pinned at the top of the page with a View output button linking to the live job output. Finished jobs show a Result badge and a View output button. Failed jobs show the error in a tooltip on the badge.