proposal: reconsider the policy_ naming convention for model server config/env variables #1658

Summary

Consider renaming the policy_-prefixed convention used for model server configuration — the policy_model server ID and the ${policy_base_url} / ${policy_api_key} / ${policy_model_name} env.yaml interpolation variables — to something role-neutral that reads well for evaluation use cases, not just RL training.

Motivation

policy is RL/training jargon ("the policy model" = the model being trained). Two problems:

1. It's opaque to non-training users. Only people with an RL background recognize the term. NeMo Gym is heavily used for plain evaluation, where this vocabulary is unfamiliar.
2. It's semantically wrong for eval. When you're using a model purely for evaluation (e.g. the new inference_provider server, which is explicitly eval-oriented), calling it the "policy model" doesn't make sense — there's no policy being optimized.

This surfaced from review feedback on #1286 / the inference providers docs: the env.yaml keys carry a policy_ prefix (policy_api_key, policy_model_name, policy_base_url) while the server's own config fields (api_key, model, base_url) do not, which is easy to misread. The root cause is that policy_* is a training-flavored name appearing on eval-focused surfaces.

Where it's used

This is a repo-wide convention, not specific to any one server. It appears in 250+ files, including:

• Every model server config: responses_api_models/{inference_provider,vllm_model,openai_model,azure_openai_model,...}/configs/*.yaml use ${policy_base_url} / ${policy_api_key} / ${policy_model_name} and the top-level policy_model server ID.
• Hundreds of resources-server and agent configs that reference the policy_model server ID and/or the policy_* env vars.
• Core library: nemo_gym/cli.py, nemo_gym/global_config.py.
• Agent + resources-server code and tests that hardcode the policy_model server name.
• Docs across all versions (fern/versions/{latest,v0.3.0,v0.2.1}/...) — quickstart, configuration reference, troubleshooting, model-server pages, many benchmark READMEs.

Considerations / open questions

• Breaking change. Existing users' env.yaml files and any external configs use policy_*. A rename breaks them unless we provide aliases.
• Agent configs hardcode the policy_model server ID by default (agents reference this name) — renaming the server ID is the riskiest part.
• Possible approaches:
  1. Pick a new role-neutral name (e.g. model, target_model, eval_model, inference_model, main_model, ...) and migrate everything.
  2. Introduce the new names as the documented convention while keeping policy_* as backward-compatible aliases for a deprecation period.
  3. Leave the internal convention as-is but improve docs so the policy_ term is explained once (centrally) and not surprising on eval-focused pages.
• Needs a single naming decision + a mostly-mechanical migration (grep/replace) with careful handling of the policy_model server ID and the policy_model_name vs policy_model distinction (the _name suffix exists because policy_model is already the instance ID).

Scope

Out of scope for #1286 (which only adds the inference_provider server). Filing as a standalone proposal so the naming decision can get team buy-in before any large mechanical change.

Ref #1286, #1289