Skip to content

Online Deployment Tutorial

Jump to bottom
Will Benoit edited this page Oct 19, 2026 · 18 revisions

Aframe Account

Logging In

First, ensure that your local ssh key has been added to your ssh agent:

ssh-add <path/to/sshkey>

Next, login to a node on CIT with ssh agent forwarding enabled:

ssh -A marie.curie@ldas-grid.ligo.caltech.edu

Once logged in, ssh to the aframe node with as the aframe user:

ssh aframe.online@aframe.ldas.cit

Directory Structure

Once logged in, you can ls to see the directory structure:

(base) [aframe.online@aframe ~]$ pwd
/home/aframe.online
(base) [aframe.online@aframe ~]$ ls
archive  dev  images  miniconda3  miniforge3  production  public_html  robot  runs  software

Here's a high-level summary of the directory contents:

  • ~/robot: Aframe's kerberos keytab, used by the search to refresh its scitoken without a password
  • ~/images/aframe: Apptainer images for the online search and the other Aframe projects
  • ~/dev: Development analyses (i.e. not uploading triggers to gracedb.ligo.org)
  • ~/production: Production analyses
  • ~/archive: Finished runs; see Archiving an Online Run
  • ~/software: GitHub repositories for aframe and amplfi
  • ~/public_html: Summary pages for each run, served at https://ldas-jobs.ligo.caltech.edu/~aframe.online/

The scitoken used for authentication during the search is stored at $XDG_RUNTIME_DIR/bt_u$(id -u) (currently /run/user/100114/bt_u100114). The search refreshes it automatically, but its current status can be examined with:

httokendecode -H $XDG_RUNTIME_DIR/bt_u$(id -u)

Initializing an Online Run

The easiest way to initialize a new online run is with the aframe-init command line tool that is installed in the base aframe environment:

(base) [aframe.online@aframe ~]$ pwd
/home/aframe.online
(base) [aframe.online@aframe ~]$ cd software/aframe/
(base) [aframe.online@aframe aframe]$ uv run aframe-init online --help
usage: aframe-init [options] online [-h] [-d DIRECTORY] [--weights-dir WEIGHTS_DIR]

optional arguments:
  -h, --help            Show this help message and exit.
  -d DIRECTORY, --directory DIRECTORY
                        (required, type: <class 'Path'>)
  --weights-dir WEIGHTS_DIR
                        (type: <class 'Path'>, default: null)
(base) [aframe.online@aframe aframe]$ uv run aframe-init online -d /home/aframe.online/dev/online-tutorial
(base) [aframe.online@aframe aframe]$ cd /home/aframe.online/dev/online-tutorial
(base) [aframe.online@aframe online-tutorial]]$ ls
config.yaml  crontab  prior.yaml  run.sh

This will create a new directory for you run at the specified path. You should notice 4 files in the directory:

  • config.yaml: Main configuration file for the run
  • prior.yaml: Prior for filtering parameters produced by AMPLFI (this is referenced within config.yaml)
  • run.sh: Bash script for launching the run
  • crontab: Template cron file. The line that matters is the @reboot line, which relaunches the search when the node reboots; see Starting, stopping and restarting the search

Setting Up the Run Directory

Running aframe-init creates the run directory and some of the necessary files, but other necessary components are not (yet) automatically created.

Model weights

The paths to the model weights need to be specified in the run.sh file. Traditionally, we've placed the weights in a directory named models and named them aframe.pt, amplfi-hl.ckpt, and amplfi-hlv.ckpt. These correspond to the AFRAME_WEIGHTS, AMPLFI_HL_WEIGHTS, and AMPLFI_HLV_WEIGHTS environment variables.

Background and foreground data

The files containing the background distribution, foreground distribution, and rejected testing waveform parameters must be specified in the run.sh file. These are usually placed in a data directory and named background.hdf5, foreground.hdf5, and rejected-parameters.hdf5. These correspond to the ONLINE_BACKGROUND_FILE, ONLINE_FOREGROUND_FILE, and ONLINE_REJECTED_FILE environment variables. It's crucial that these files were generated by the aframe.pt model mentioned above.

Completing the run.sh file

aframe-init fills in most of run.sh: the scitoken location, the model and data paths above, and AFRAME_ONLINE_OUTDIR, which is always <run_dir>/output. The following still need checking or filling in:

  • AFRAME_CONTAINER: The online search container. Defaults to $HOME/images/aframe/online.sif; point it at a specific version, e.g. $HOME/images/aframe/online:v1.3.2.sif, so that the version a run used is recorded.
  • ARRAKIS_SERVER: Arrakis server to stream data from. Defaults to grpc://arrakis:31206, which should not normally need changing.
  • CUDA_VISIBLE_DEVICES: Left empty by aframe-init and must be set. There are two GPUs available on the aframe node. GPU 1 is reserved for Aframe, but GPU 0 is part of the condor pool and may be used by other jobs. Any LLPIC runs and productions should have this variable set to 1. Other search instances should set this to 0.
  • MONITOR_OUTDIR: See Viewing the run summary page. The monitoring block is commented out by default.

ONLINE_DATADIR is only used when reading from frame files, which is deprecated (see below), and can be left as it is.

Updating the config.yaml file

The config.yaml file contains the model configuration parameters for the online search and should be updated to reflect how the models were trained/tested. In particular, loading the AMPLFI models requires specifying the exact architecture, and this may change from model to model. Most other parameters can remain unchanged, but here are some important ones to verify before running:

  • data_source: Where the strain data is read from. This should be "arrakis". The "frames" option still exists in the code, but frame files are no longer delivered to the cluster, so it no longer works here.
  • replay_id: The Arrakis replay to stream from, e.g. "O4LLPIC" for the LLPIC replay. Some channels are only available within a replay. Set to null for live data.
  • channels: The Hanford, Livingston, and Virgo strain channels to analyze. These change depending on whether we're analyzing an MDC, LLPIC, or live data; e.g. the LLPIC replay uses H1:GDS-CALIB_STRAIN_CLEAN_INJ1_O4Replay and its L1 and V1 equivalents.
  • far_threshold: The false alarm rate, in units of inverse days, below which we'll upload an event. This controls how frequently we upload events to GraceDB on average, and should typically be set to 12, but for testing purposes it can be useful to set a higher rate.
  • emails: A list of emails to send notifications to. Recipients will be alerted when the process starts, when an event is detected below email_far_threshold, and when an error occurs and the pipeline crashes.
  • server: The GraceDB server to which to upload events: local, playground, test, or production, with playground being reserved for LLPIC uploads. Events are submitted through the Kafka broker that goes with the chosen server. Setting to local will just write out data products locally and is useful for testing.

Verifying Pipeline Activity

a. Viewing the run summary page

Summary pages for individual events and the search as a whole are created automatically during the run by a monitoring process, which writes them to the directory given by MONITOR_OUTDIR in run.sh. The monitoring block in run.sh is commented out by default. Set MONITOR_OUTDIR and uncomment both lines, e.g.:

export MONITOR_OUTDIR=$HOME/public_html/o4_llpic_playground

Visiting https://ldas-jobs.ligo.caltech.edu/~aframe.online/ and navigating to the directory given by MONITOR_OUTDIR allows for viewing these pages in a browser. At the top level of this directory is a summary.html page that indicates whether the search is active and whether the data is in analysis-ready mode. Also note that the UTC time of the last update is listed at the top of the page. If the page hasn't been refreshed in a while, the status information may be out of date.

image

Summary pages for individual events can be found in the events directory and will be discussed below.

b. Check that the search is running

If the summary page hasn't been updated, the search may have crashed, but it's possible that only the monitoring process failed. Check for the search's processes:

(base) [aframe.online@aframe ~]$ pgrep -af bin/online | head -n3
12434 /opt/env/bin/python /opt/env/bin/online --config /home/aframe.online/dev/ir1_llpic_playground/config.yaml
12769 /opt/env/bin/python /opt/env/bin/online --config /home/aframe.online/dev/ir1_llpic_playground/config.yaml
12775 /opt/env/bin/python /opt/env/bin/online --config /home/aframe.online/dev/ir1_llpic_playground/config.yaml

The --config path shows which run each process belongs to. The search starts many worker processes, so the count isn't a useful check on its own. If there are none for your run, the search isn't running; see the section below on restarting it.

To see whether the search is still processing data, look at the end of its current log file (see Log Files), which should be updated every second:

(base) [aframe.online@aframe ~]$ tail -f <run_dir>/output/logs/<latest>/online.log
2025-07-09 17:57:07,221 (17:57:07,221) - root - DEBUG - Read successful
2025-07-09 17:57:07,227 (17:57:07,228) - root - DEBUG - Updating input buffer
2025-07-09 17:57:07,228 (17:57:07,228) - root - DEBUG - Updating snapshotter
2025-07-09 17:57:07,228 (17:57:07,228) - root - DEBUG - Whitening data
2025-07-09 17:57:07,230 (17:57:07,230) - root - DEBUG - Performing inference
2025-07-09 17:57:07,234 (17:57:07,234) - root - DEBUG - Updating output buffer
2025-07-09 17:57:07,310 (17:57:07,310) - root - DEBUG - Searching for event...

Starting and stopping the search

Starting

Start the search from the run directory, inside a tmux session so that it keeps running after you log out:

(base) [aframe.online@aframe ~]$ tmux new -s <run_name>
(base) [aframe.online@aframe ~]$ cd <run_dir>
(base) [aframe.online@aframe <run_name>]$ ./run.sh

Detach with Ctrl-b d, and reattach later with tmux attach -t <run_name>. run.sh restarts the search automatically whenever it crashes, and writes each crash to monitoring.log.

Only ever run one search per run directory.

Relaunching after a reboot

So that the search restarts after the node reboots, add an @reboot line for the run to the account's crontab with crontab -e:

@reboot sleep 120 && cd /home/aframe.online/dev/<run_name> && /bin/bash run.sh

The sleep 120 gives the node time to finish starting up before the search launches.

A search relaunched this way isn't in a tmux session.

Stopping

If the search is running in tmux, attach to the session and press Ctrl-c. run.sh then stops both the search and the monitoring process.

If it isn't in tmux (e.g. it was relaunched after a reboot), find the process group of its run.sh and stop the whole group:

(base) [aframe.online@aframe ~]$ ps -o pid,pgid,args -u aframe.online | grep run.sh
   6382    6382 /bin/bash run.sh
  12245    6382 /bin/bash run.sh
(base) [aframe.online@aframe ~]$ kill -- -6382

The - in front of the group ID sends the signal to every process in the group. If more than one run is active, check which group is which with pgrep -af bin/online and ps -o pid,pgid,args.

If the search shouldn't come back after the next reboot, also remove or comment out its @reboot line with crontab -e.

Log Files

There are multiple log files that capture the output of different parts of the search. In the top level of the run directory, there are monitoring.log and summary_pages.log files. Additionally, in output/logs, there are directories that correspond to each instance of the search being started. These directories contain daily log files, with the active file always being online.log.

(base) [aframe.online@aframe logs]$ pwd
/home/aframe.online/dev/ir1_llpic_playground/output/logs
(base) [aframe.online@aframe logs]$ ls
2026-10-07T14:21:13  2026-10-07T21:14:04  2026-10-13T15:43:25
(base) [aframe.online@aframe logs]$ cd 2026-10-13T15:43:25
(base) [aframe.online@aframe 2026-10-13T15:43:25]$ ls
online.log  online.log.2026-10-13  online.log.2026-10-14  online.log.2026-10-15

Each directory is named for the time (UTC) the search started.

Log file descriptions

Log file name Description When to examine
monitoring.log Tracks pipeline restarts and start-up errors Search failing to start
summary_pages.log Logging from the summary html pages process Summary pages failing
online.log Main search logging statements Search crash, investigation of search behavior for an event
online.log.<date> Historical contents from online.log Investigation of search behavior for an event

Summary pages

The output directory of the monitoring process is structured as follows:

outdir/
├── event_data.hdf5
├── summary.html
├── monitor.log
├── plots/
│   ├── aframe_latency.png
│   └── ...
└── events/
    ├── event_1436032137/
    │   ├── event.html
    │   └── plots/
    │       ├── H1_qtransform.png
    │       └── ...
    ├── event_1436033156/
    └── ...

At the top level, there is a summary.html page that contains information about the status of the run, as discussed above. Additionally, there is a histogram of event reporting latency. Typical latencies for MDCs should be less than 5 seconds, and less than 15 seconds for production runs. Events with latencies greater than this are worth investigating. The event_data.hdf5 file information on a per-event basis and can be a good place to look to get started. Also on the summary page are event rate plots. When running on non-injected data streams, we want to see that the event rate roughly stays at the expected level of 1 per 2 hours. All of the individual plots can be found in the plots directory at this level.

The summary page also tracks the search's duty cycle from the record of segments the search writes to <run_dir>/output/segments. The duty cycle is the time Aframe analyzed divided by the time it could have analyzed. Time when the detectors weren't analysis ready (Detectors not ready) or when no data arrived from Arrakis (Missing data) is left out. Uptime is the fraction of time the search was running at all.

A "Where the search lost time" table breaks down the time that counts against us:

Cause Meaning
Starting up Loading models and filling buffers after the search starts
Filter warm-up Data was available, but the whitening filter didn't yet have enough data to be ready
Search down Time with no record from the search at all, i.e. it wasn't running.

Short Search down entries of about a second after a Missing data gap are expected, as the search needs about a second of fresh data before it can analyze again. Longer ones mean the search crashed or was stopped. If the segment record ever overlaps itself, a red warning appears above the duty cycle table.

Inside the events directory is a directory for each detected event. Each event has an event.html page which shows Q-plots for each active detector, Aframe's detection statistic timeseries, AMPLFI's skymaps and corner plot, and the ASD from each active detector. At the top of each event page is a link to the event in GraceDB. All of the individual plots are again located in the plots directory for the event, and these can be a good place to start to get an overview of how the algorithms performed.

image

Re-examining events offline

If Aframe is offline when an event occurs, or if there's a reason to re-analyze an event, buoy can run the online models over it. ~/buoy holds a set of config files that match the models of the current online run:

  • buoy.yaml: the events to analyze, the output directory, and the paths to the model weights and the configs below
  • aframe_config.yaml, amplfi-hl-config.yaml, amplfi-hlv-config.yaml: the models' preprocessing and AMPLFI architectures
  • prior.yaml: the online run's prior.yaml, with luminosity_distance renamed to distance, the name buoy expects

To analyze an event, list it in events in buoy.yaml, as an S-event, G-event or GPS time; multiple events can be given as strings in a list. Then run:

(base) [aframe.online@aframe ~]$ cd ~/buoy
(base) [aframe.online@aframe buoy]$ export TOKEN=$XDG_RUNTIME_DIR/bt_u$(id -u)
(base) [aframe.online@aframe buoy]$ apptainer exec --nv --bind /ceph,$XDG_RUNTIME_DIR \
    --env CUDA_VISIBLE_DEVICES=0,GWDATAFIND_SERVER=datafind.ldas.cit:80,BEARER_TOKEN_FILE=$TOKEN,SCITOKEN_FILE=$TOKEN \
    ~/images/buoy/buoy.sif /opt/env/bin/buoy --config buoy.yaml
  • --bind /ceph and GWDATAFIND_SERVER let buoy find and read the archived strain data. Without them it falls back to NDS, which fails.
  • The token lets buoy look up S-events and G-events in GraceDB.
  • Set CUDA_VISIBLE_DEVICES to a GPU the live search isn't using (check with nvidia-smi). An event takes about a minute.

Results are written to ~/public_html/buoy_results/<event>, and the summary page for an event is at https://ldas-jobs.ligo.caltech.edu/~aframe.online/buoy_results//summary.html. buoy skips events it has already analyzed; add --force true to analyze them again.

When the online models change, update the weight paths in buoy.yaml and copy the new architectures, preprocessing and prior into the other configs.

Archiving a finished run

When a run is finished, move it to ~/archive following Archiving an Online Run.

Clone this wiki locally