Last updated: May 2022
HaRT model's release information. Inspired by Model Cards for Model Reporting (Mitchell et al.).
This model was developed as a first solution to the task of Human Language Modeling (HuLM) which further trains large language models to account for statistical dependence between documents (i.e. multiple documents written by the same human). This version of HaRT builds upon Open AI GPT-2 (Radford et al., 2019) small and is developed with a base code from HuggingFace.
May 2022, trained on social media (Twitter) dataset spanning tweets posted from 2009 through 2020.
Human language model (can use anywhere language models are used).
hart-gpt2sml-twt-v1 (153M parameters)
The primary intended use of this model is to make AI more human-centered. We encourage using HaRT wherever you would use a traditional large language model. We also envision HaRT's user states to present oppotunites in models designed to enable bias correction and fairness techniques.
The primary intended users of the model are AI and Psychology researchers and practioners
At this point, our model is not intended to be used in practice for mental health care nor labeling of individuals publicly with personality or age scores. Additionally, the current publicly available version of the model is not intended to be used for generation yet.
PEW statistics on Twitter users (skews young; everything else is fairly representative):
Gender: 50% female, 50% male
Age groups & percentage: 18-29 (29%), 30-49 (44%), 50-64 (19%), 65+ (8%)
Race/Ethnicity: 11% black, 60% white, 17% hispanic
Income range & perecntage: <$30,000 (23%), $30-$74,999 (36%), $75,000+ (41%)
Education groups & percentage: Less than high school (4%), High school graduate (54%), College graduate+ (42%)
Reference: Pew Research Center
The released model version trains only on the paper's Twitter dataset, which is a subset of the County Tweet Lexical Bank Giorgi et al., 2018 appended with newer 2019 and 2020 tweets, in total spanning 2009 through 2020.
Note: The model in our paper also trained on Facebook data and thus achieved slightly better evalution results. We are not able to release the Facebook data but we will soon be releasing a model that includes other public social media data sources for training.
We filter the datasets to only include tweets marked as English from users who have at least 50 total posts and at least 1000 words in total, ensuring moderate language history for each user.
We use:
- Perplexity for language modeling evaluation.
- Weighted F1 for document-level classification tasks (fine-tuning).
- Pearson r and Disattenuated pearson-r for user-level regression tasks (fine-tuning).
We evaluate the utility of fine-tuning our pre-trained HaRT for document- and user-level tasks. Document-level tasks include stance detection, sentiment classification. User-level tasks include age estimation, and personality (openness) assessment tasks.
Language modeling evaluation: The pre-training dataset is split into a test set of new users (not seen during training), as described in the paper.
Stance Detection: SemEval2016 dataset ([Mohammad et al., 2016] (https://aclanthology.org/S16-1003/) with the historical context as a subset of the extended dataset from [Lynn et al. (2019)] (https://aclanthology.org/W19-2103/)
Sentiment Analysis: SemEval-2013 dataset ([Nakov et al., 2013] (https://aclanthology.org/S13-2052/) with the historical context as a subset of the extended dataset from [Lynn et al. (2019)] (https://aclanthology.org/W19-2103/)
Twitter, Stance and Sentiments datasets are released and can be found on our website and on our github
We can not release the Facebook user-level task datasets for privacy considerations: participants did not consent to have their data shared openly.
The motivation behind using Twitter is that it is a public dataset of human language (documents connected to users) that is prevalently used for language reserach.
We filter the extended context datasets from Lynn et al. (2019) to include the only the historically created posts with respect to the labeled dataset from SemEval tasks.
Unlike other human-centered approaches, HaRT is not directly fed user attributes as part of the pre-training thus the model parameters do not directly encode user attributes. While the multi-level human-document-word structure within HuLM can enable bias correcting and fairness techniques, the ability to better model language in its human context also presents opportunities for unintended harms or nefarious exploitation. For example, models that improve psychological assessment are not only useful for research and clinical applications, but could be used to target content for individuals without their awareness or consent. In the context of use for psychological research, such models may risk release of private research participant information if trained on private data without checks for exposure of private information. To negate this potential, at this time we are only releasing a version of HaRT that is not trained on the consented-use private Facebook data. We will consider release of a model trained on the private posts as well if we can verify it meets differential privacy standards.
While modeling the human state presents opportunities for reducing AI bias, prior to clinical or applied use, such models should be evaluated for failure modes such as error across target populations for error or outcome disparities (Shah et al., 2020).