first experiments

[ArneBinder]

general

all the experiments should be executed with the multi-model variant of the individual task models

• target-model only
• frozen pre-trained target-model + bert-base-cased
• frozen pre-trained target-model + frozen other-task-model
• pre-trained target-model + frozen other-task-model

Hyperparameters:

• learning rate
• training time: start with a lot of epochs (50?); early stopping (see patience parameter)?
• warming: for later (seems interesting, but requires effort)

Co-ref

@tanikina @ArneBinder

• target-model only:
  • train target: config, W&B run, results
  • frozen target: config, W&B run, results
  • train target, but with aggregate=mean(!): W&B run, results
• frozen pre-trained target-model + bert-base-cased: config, W&B run, results
• frozen pre-trained target-model + frozen other-task-model:
  • other=NER: W&B run, results
  • other=RE: config, W&B run, results
  • other=EQA: W&B run, results
• pre-trained target-model + frozen other-task-model:
  • other=NER: W&B run, results
  • other=RE: config, W&B run, results
  • other=EQA: W&B run, results

EDIT:

• PR that adds respective configs and updates the log.md: Coreference experiments #64
• experimental results: log.md#2023-09-28
• wandb report with the val/f1 and val/loss graphs (experiments from 2023-09-28 and 2023-09-29): https://wandb.ai/tanikina/conll2012-multi_model_coref_hoi-training/reports/Coreference-Experiments--Vmlldzo1NjAwNTMy

NER

@harbecke

• target-model only
• frozen pre-trained target-model + bert-base-cased
• frozen pre-trained target-model + frozen other-task-model
• pre-trained target-model + frozen other-task-model

PR that adds the NER configs and update the experiment log: #71

RE

@leonhardhennig

• target-model only: config, W&B run, results(TODO @ArneBinder)
• frozen pre-trained target-model + bert-base-cased: config, W&B run, results(TODO @ArneBinder)
• frozen pre-trained target-model + frozen other-task-model: config W&B run results
• pre-trained target-model + frozen other-task-model: config W&B run results

Sanity Checks

• frozen pre-trained target-model with mean "aggregation" config + "model.aggregate=mean" W&B run results
• frozen pre-trained target-model with attention-based "aggregation" config W&B run results

extractive QA

@StalVars

• target-model only
• frozen pre-trained target-model + bert-base-cased
• frozen pre-trained target-model + frozen other-task-model
• pre-trained target-model + frozen other-task-model

Weights & Biases project for EQA: https://wandb.ai/stalvars/dataset+squadv2-task+extractive_question_answering-training

[leonhardhennig]

RE results here: https://wandb.ai/leonhardhennig/tacred-multi_model_re_text_classification-training

[ArneBinder]

Tatiana and me just tried to compile some configs for the individual checklist items and I asked myself what you used @leonhardhennig. I checked the W&B runs, but couldn't find the experiment configs from the commands. Could you add the configs that you used via a new PR? i.e.

• tacred_multimodel_bert_frozen_re.yaml
• tacred_multimodel_frozen_re.yaml
• tacred_multimodel_frozen_re_ner.yaml
• tacred_multimodel_re_frozen_ner.yaml

Otherwise it is a bit hard to follow what exactly was executed and to check if this is aligned with the experiments for the other tasks.

EDIT: Maybe also link the individual W&B runs in the RE section above (or mention their W&B run names, then I can do that) so that we know which run belongs to which checked item (and also can infer what exact command was used).

[ArneBinder]

Results

GENERAL NOTES:

• reported metric is val/f1 (at best epoch!)
• link each value to the respective section in the log.md to easily verify the metric value and to jump to further resources (W&B run, command, best epoch, ...)
• all experiments use model.aggregate=attention.

	COREF	RE	NER	EQA
1. tuned bert	0.736	0.764	0.914
2. tuned pre-trained target-model + frozen other-task-model	0.743 (with RE)	0.762 (with NER)	0.911 (with coref)
3. frozen pre-trained target-model + tuned bert	0.736	0.733	0.911
4. frozen pre-trained target-model	0.735	0.727	0.897
5. frozen pre-trained target-model + frozen other-task-model	0.738 (with RE)	0.730 (with NER)	0.902 (with coref)

Notes regarding the choice of setups:

• 1. to 3. (tuning of any base model) were primarily meant as sanity check if the aggregation mechanism over models does not introduce any issues at its own
• 4. and 5. (all base models are frozen) are meant to support the actual hypothesis: Information encoded in a base model trained on one task can be helpful for another (related) task (TODO: does anyone have a better version of this?)