Skip to content

markuxt/sync-publications

Use this GitHub action with your project
Add this Action to an existing workflow or create a new one
View on Marketplace
BranchesTags

Repository files navigation

Sync Publications from OpenAlex

A GitHub Action that fetches publications from OpenAlex based on your members' ORCIDs and writes them to your publications directory as markdown. Each generated publication file contains the title, authors, ORCIDs, year, DOI, venue, keywords, a reconstructed abstract, the PDF link, and — when an open-access PDF is available — a screenshot of the page containing the abstract.

Features

  • ORCID-driven: scans existing member markdown for ORCIDs and pulls works per author from OpenAlex.
  • Institution filter: only fetches publications affiliated with your institution via its ROR ID.
  • Rich output: frontmatter includes title / authors / ORCIDs / year / DOI / venue / keywords / abstract_screenshot; the body is the reconstructed abstract.
  • Abstract-page screenshot: when an open-access PDF is available, it is downloaded, the page containing the abstract is located, and a high-resolution PNG is rendered (shortest side ≥ 1000 px).
  • Smart deduplication: OpenAlex ID + normalized DOI + Jaccard title similarity + author overlap; a CJK-safe tokenizer.
  • Backfill: existing publication files missing openalex_id and/or authors_orcid are looked up on OpenAlex and completed in place, with the body preserved verbatim.
  • Local-first: the same code runs in GitHub Actions and locally via .env.development.

Quick start

As a GitHub Action

- uses: markuxt/sync-publications@v1
  with:
    ror_id: 'https://ror.org/03y4dt428'
    contact_email: 'contact@example.com'
    members_dir: 'src/members'
    publications_dir: 'src/publications'

See the full workflow example below.

Run locally

pnpm install
cp .env.example .env.development
# Edit .env.development — fill in ROR_ID and CONTACT_EMAIL
pnpm dev          # run via tsx (no build needed)
# or:
./scripts/test-local.sh           # recommended — equivalent to pnpm dev
./scripts/test-local.sh --build   # run the compiled dist/index.js

You can also override any environment variable on the command line:

ROR_ID=https://ror.org/other pnpm dev

Inputs

ror_id (required)

The institution's ROR ID. Look yours up at https://ror.org.

contact_email (required)

The contact email required by OpenAlex's polite-pool policy.

openalex_api_key (optional)

An OpenAlex API key for the premium pool (higher rate limits). The action runs fine without it on the free polite pool (contact_email alone). When provided, it is appended as api_key= to every OpenAlex request. Pass it via a repository secret — see the full workflow example:

openalex_api_key: ${{ secrets.OPENALEX_API_KEY }}

Locally, set OPENALEX_API_KEY in .env.development. An empty/undefined key is treated as "no key" (polite pool).

members_dir (optional, default src/members)

The directory scanned for member markdown files (relative to the repo root or absolute). Set this if your members live somewhere other than src/members (e.g. src/people).

publications_dir (optional, default src/publications)

The directory where generated publication markdown and screenshots are written (relative to the repo root or absolute). Defaults to src/publications and is independent of members_dir, so members and publications can live under different roots.

Outputs

new_publications_count

The number of publication files written by this run.

new_publications_files

A newline-separated list of new file paths (emitted via a multi-line heredoc format so it is never truncated).

backfilled_publications_count

The number of existing publication files that had missing openalex_id and/or authors_orcid filled in during this run.

backfilled_publications_files

A newline-separated list of backfilled file paths.

Member file format

Member markdown files live at <members_dir>/**/*.md (default src/members). They must contain an orcid field:

---
name: John Doe
orcid: 0000-0001-2345-6789
---

Members with _hidden: true are skipped. ORCIDs are validated via the ISO 7064 11-2 check digit; an invalid ORCID is skipped with a warning, so a typo in one file won't poison the whole sync.

Publication file format

Generated publication files are written to a flat layout (one .md per publication, no subdirectory):

<publications_dir>/<year>/<title-slug>.md
<publications_dir>/<year>/<title-slug>.png   (only when an OA PDF was processed)

The filename is a slugified form of the title (lowercase / whitespace and punctuation → - / CJK preserved / capped at 80 chars), falling back to the OpenAlex ID when the title is empty, and appending a -<shortId> suffix on collisions within the same year directory. See src/utils/slugify.ts.

Each <title-slug>.md looks like:

---
_hidden: false
title: Publication Title
authors:
  - Doe, John
authors_orcid:
  - 0000-0001-2345-6789
  - null
year: 2024
doi: https://doi.org/10.1000/example
openalex_id: W123456789
venue: Conference Name 2024
abstract_screenshot: publication-title.png   # bare filename — lives next to the .md; markuxt resolves it like a member photo
keywords:
  - control systems
  - robotics
---

The reconstructed abstract text, rebuilt from OpenAlex's inverted index…

Abstract-page screenshot notes

  • Rendered via pdftoppm (poppler) at 200 DPI. On A4 paper the shortest side is roughly 1654 px — well above the 1000 px minimum.
  • GitHub Actions runners ship with pdftoppm.
  • macOS locally requires a one-time install: brew install poppler.
  • If pdftoppm is unavailable, the run is not interrupted — the markdown is still written, with abstract_screenshot left empty (the OA PDF URL and abstract page are resolved internally and not recorded either way).

Deduplication strategy

Three layered checks (a hit on any one ⇒ skipped):

  1. OpenAlex ID (with or without a leading W).
  2. Normalized DOI (https://doi.org/ / https://dx.doi.org/ / doi: are all folded into a lowercase bare DOI).
  3. Similarity heuristic: year difference ≤ 1, title Jaccard ≥ 0.85, author overlap ≥ 0.5.

CJK / Hangul / Kana titles are tokenized per character, so non-Latin publications aren't tokenized into the empty set (a previous implementation falsely flagged these as duplicates).

Author deduplication: OpenAlex's authorships are expanded by (author × institution), so the same person can appear multiple times. The parser deduplicates on two keys at parse time — ORCID (preferred) and name (fallback) — keeping the authors and authors_orcid arrays parallel and preserving the first-seen spelling.

Within the same pending batch, older versions of the same publication are marked _hidden: true; only the newest version stays visible.

Backfilling missing fields

When an existing publication file is missing its openalex_id and/or authors_orcid frontmatter (common for files written by hand or by an older version of the action), the run looks the work up on OpenAlex and completes the fields in place — every other frontmatter field and the body are left untouched.

Lookup priority: an existing openalex_id, then the doi, then a title + year search. The search fallback is guarded by title Jaccard similarity (≥ 0.85) and author overlap (≥ 0.5), so an uncertain match never writes a wrong ID. authors_orcid is built parallel to the existing authors list, matching each author by name (unknown authors get null).

This runs on every sync and is idempotent — complete files are skipped, so there is no ongoing cost after the first run. The count and list of backfilled files are exposed via backfilled_publications_count / backfilled_publications_files.

Backfilling missing screenshots

A second pass (screenshots_backfilled_count / screenshots_backfilled_files) renders an abstract-page screenshot for any existing publication that has an OpenAlex ID but no abstract_screenshot yet — it re-queries OpenAlex for the work's OA PDF URL, downloads it, locates the abstract page, and writes the PNG next to the .md plus the abstract_screenshot (bare filename) field. Idempotent, and it uses a browser-like User-Agent and tries every OA location, but publisher anti-bot (Cloudflare/Akamai — e.g. Wiley, MDPI, Elsevier) still blocks most closed-access hosts; only hosts that actually serve the PDF (IEEE, arXiv, many repositories) will succeed.

Project structure

sync-publications/
├── src/
│   ├── index.ts              # main entry — reads env vars, orchestrates the sync
│   ├── types.ts              # shared TypeScript types
│   ├── utils/
│   │   ├── abstract.ts       # rebuilds the abstract from the inverted index
│   │   ├── deduplication.ts  # tokenization, Jaccard, author overlap
│   │   ├── doi.ts            # DOI normalization
│   │   ├── env.ts            # loads .env / .env.<NODE_ENV>
│   │   ├── github.ts         # writes GITHUB_OUTPUT (heredoc-aware)
│   │   ├── glob.ts           # markdown file discovery
│   │   ├── http.ts           # fetch + timeout + backoff retry
│   │   ├── openalex.ts       # OpenAlex API client
│   │   ├── pdf.ts            # PDF download / text extraction / screenshot
│   │   ├── formatters.ts     # author-name / ORCID formatting
│   │   └── yaml.ts           # YAML frontmatter parse + serialize
│   ├── scanners/
│   │   ├── members.ts        # scans members for ORCIDs
│   │   └── publications.ts   # scans existing publications for dedup
│   └── workers/
│       ├── parser.ts         # OpenAlex work → PendingPublication
│       └── deduplicator.ts   # filters + dedupes the pending list
├── tests/                    # vitest suite (140 tests)
├── action.yml                # GitHub Action metadata
├── dist/                     # compiled artifact loaded by the node24 runtime
└── package.json

Development

Prerequisites

  • Node.js ≥ 24.0.0
  • pnpm (recommended) or npm
  • (optional, for local screenshots) poppler: brew install poppler

Scripts

pnpm install             # install dependencies
pnpm dev                 # run via tsx (no build needed)
pnpm build               # bundle to a single self-contained dist/index.js (tsup, all deps inlined)
pnpm start               # run the compiled dist/index.js
pnpm test                # run the vitest suite
pnpm test:watch          # interactive watch mode
pnpm test:coverage       # vitest + v8 coverage
./scripts/test-local.sh          # equivalent to pnpm dev, auto-loads .env.development
./scripts/test-local.sh --build  # same, but runs dist/index.js

Environment variable files

  • .env.example — committed template listing all variables.
  • .env.development — gitignored; auto-loaded by pnpm dev and ./scripts/test-local.sh.
  • .env — also supported (lower priority than .env.development).

Existing process.env values always win, so ROR_ID=... pnpm dev on the command line overrides the value in .env.development.

Build & release

dist/ is intentionally committed. pnpm build uses tsup to bundle src/index.ts together with all dependencies (dotenv / glob / unpdf / yaml) into a single self-contained dist/index.js. GitHub Actions' node24 runtime loads it directly (see action.yml) and does not need node_modules at runtime. Because of this, any PR that changes src/ must rebuild dist/ before merge.

pnpm build
git add dist/
git commit -m 'build: rebuild dist'

Full workflow example

name: Sync Publications

on:
  workflow_dispatch:
  schedule:
    - cron: '0 0 * * 0'  # every Sunday at midnight UTC

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Sync publications from OpenAlex
        id: sync
        uses: markuxt/sync-publications@v1
        with:
          ror_id: 'https://ror.org/03y4dt428'
          contact_email: 'research-lab@example.com'
          members_dir: 'src/members'
          publications_dir: 'src/publications'

      - name: Commit new publications
        env:
          COUNT: ${{ steps.sync.outputs.new_publications_count }}
          FILES: ${{ steps.sync.outputs.new_publications_files }}
        run: |
          if [ -n "$FILES" ]; then
            git config --local user.email "action@github.com"
            git config --local user.name "GitHub Action"
            git add src/publications/
            git commit -m "chore: sync $COUNT publication(s) from OpenAlex"
            git push
          fi

License

Apache-2.0

Support

For issues, please report them at GitHub Issues.

About

Fetches publications from OpenAlex based on member ORCIDs and syncs to content directory

Resources

Readme

License

Apache-2.0 license
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases 4

v0.0.4 Latest
Jun 27, 2026
+ 3 releases

Packages

 
 
 

Contributors

Languages