Skip to content

Update methodshub.qmd #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chainsawriot merged 1 commit into from
Aug 19, 2025
Merged

Update methodshub.qmd #104

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 98 additions & 51 deletions methodshub.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,36 @@ format:
gfm: default
---

## Description
# oolong - Create Validation Tests for Automated Content Analysis
<!--
General specifications:
- This specification of the Methods Hub friendly README often uses the word 'should' to indicate the usual case. If you feel you need to do it differently, add a comment to argue for your case when you submit your method.
- A Methods Hub friendly README should contain all sections below that are not marked as optional, and can contain more sections.
- A Methods Hub friendly README should contain as few technical terms as possible and explain (or link to an explanation of) all used technical terms.
- A Methods Hub friendly README should link to all code files that it mentions using the [text](URL relative to this file) format. The relative URL (i.e., no "https://github.com") is neccessary for proper versioning in Methods Hub.
- A Methods Hub friendly README should contain an explanation (in the text) and an alternative for each image it contains (e.g., data models, pipeline, schema structure). Format: ![alternative text that describes what is visible in the image](URL relative to this file).
- A Methods Hub friendly README should link to authoritative sources rather than containing a copy of the information (e.g., documentation).
- A Methods Hub friendly README should use a uniform citation style for all references, for example APA7 https://apastyle.apa.org/style-grammar-guidelines/references/examples

Title:
1. The title must be the README's only first-level heading (line starting with a single '#').
2. The title should make the method's purpose clear.
3. The title (line 1 of this file) must be changed by you, but all other headings should be kept as they are.
4. The title must be appropriate (not harmful, derogatory, etc.).

Section templates:
The README template comes with text templates for each section (after each comment) that can be used, customized or removed as desired.
-->

<!-- - Provide a brief and clear description of the method, its purpose, and what it aims to achieve. Add a link to a related paper from social science domain and show how your method can be applied to solve that research question. -->
## Description
<!--
1. Provide a brief and exact description of the method clearly mentioning its purpose i.e., what the method does or aims to achieve in abstract terms (avoiding technical details).
2. The focus should be on explaining the method in a way that helps users with different levels of expertise understand what it does, without going into technical details. It should clearly describe what inputs are needed and what outputs can be expected.
3. Briefly explain the input and output of the method and its note worthy features.
4. Provide link(s) to related papers from the social science domain using the method or similar methods for solving social science research questions.
5. In a separate paragraph, highlight the reproducibility aspect of the method providing details or references to the resources used by the method, the data used in building the pre-trained modules etc.
6. It should also discuss the decisions and parameters controlling the behavior of the method.
-->

Intended to create standard human-in-the-loop validity tests for typical automated content analysis such as topic modeling and dictionary-based methods. This package offers a standard workflow with functions to prepare, administer and evaluate a human-in-the-loop validity test. This package provides functions for validating topic models using word intrusion, topic intrusion (Chang et al. 2009, <https://papers.nips.cc/paper/3700-reading-tea-leaves-how-humans-interpret-topic-models>) and word set intrusion (Ying et al. 2021) [doi:10.1017/pan.2021.33](https://doi.org/10.1017/pan.2021.33) tests. This package also provides functions for generating gold-standard data which are useful for validating dictionary-based methods. The default settings of all generated tests match those suggested in Chang et al. (2009) and Song et al. (2020) [doi:10.1080/10584609.2020.1723752](https://doi.org/10.1080/10584609.2020.1723752).

Expand All @@ -18,53 +45,31 @@ Intended to create standard human-in-the-loop validity tests for typical automat
* Text Analysis
* Topic Model

## Science Usecase(s)

<!-- - Include usecases from social sciences that would make this method applicable in a certain scenario. -->
<!-- The use cases or research questions mentioned should arise from the latest social science literature cited in the description. -->
## Use Cases
<!--
1. The use cases section should contain a list of use cases relevant to the social sciences.
2. Each use case should start with a description of a task and then detail how one can use the method to assist in the task.
3. Each use case may list publications in which the use case occurs (e.g., in APA7 style, https://apastyle.apa.org/style-grammar-guidelines/references/examples).
-->

This package was used in the literature to valid topic models and prediction models trained on text data, e.g. [Rauchfleisch et al. (2023)](https://doi.org/10.1080/17512786.2022.2110928), [Rothut, et al. (2023)](https://doi.org/10.1177/14614448231164409), [Eisele, et al. (2023)](https://doi.org/10.1080/19312458.2023.2230560).

## Repository structure

This repository follows [the standard structure of an R package](https://cran.r-project.org/doc/FAQ/R-exts.html#Package-structure).

## Environment Setup

With R installed:

```r
install.packages("oolong")
```

<!-- ## Hardware Requirements (Optional) -->
<!-- - The hardware requirements may be needed in specific cases when a method is known to require more memory/compute power. -->
<!-- - The method need to be executed on a specific architecture (GPUs, Hadoop cluster etc.) -->

## Input Data
<!--
1. The input data section should illustrate the input data format by showing a (possibly abbreviated) example item and explaining (or linking to an explanation of) the data fields.
2. The input data section should specify which parts of the input data are optional and what effect it has to not provide these.
3. The input data section should link to a small example input file in the same repository that can be used to test the method (this test should be described in the section "How to Use").
-->

## Input Data

<!-- - The input data has to be a Digital Behavioral Data (DBD) Dataset -->
<!-- - You can provide link to a public DBD dataset. GESIS DBD datasets (https://www.gesis.org/en/institute/digital-behavioral-data) -->

The input data has to be a topic model or prediction model trained on text data. For example, one can train a topic model from the text data (tweets from Donald trump) included in the package by:
A sample input is a model trained on text data, e.g.

```r
```{r}
#| message: false
library(oolong)
library(seededlda)
library(quanteda)
trump_corpus <- corpus(trump2k)
tokens(trump_corpus, remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE,
split_hyphens = TRUE, remove_url = TRUE) %>% tokens_tolower() %>%
tokens_remove(stopwords("en")) %>% tokens_remove("@*") -> trump_toks

model <- textmodel_lda(x = dfm(trump_toks), k = 8, verbose = TRUE)
abstracts_seededlda
```

## Sample Input and Output Data

<!-- - Show how the input data looks like through few sample instances -->
<!-- - Providing a sample output on the sample input to help cross check -->

A sample input is a model trained on text data, e.g.

```{r}
Expand All @@ -74,9 +79,39 @@ library(seededlda)
abstracts_seededlda
```

## Output Data
<!--
1. The output data section should illustrate the output data format by showing a (possibly abbreviated) example item and explaining (or linking to an explanation of) the data fields.
2. The output data section should link to a small example output file in the same repository that can be re-created (as far as the method is non-random) from the input data (as described in the section "How to Use").
-->

The sample output is an oolong [R6 object](https://r6.r-lib.org/).

## Hardware Requirements
<!--
1. The hardware requirements section should list all requirements (storage, memory, compute, GPUs, cluster software, ...) that exceed the capabilities of a cheap virtual machine provided by cloud computing company (2 x86 CPU core, 4 GB RAM, 40GB HDD).
2. If the method requires a GPU, the hardware requirements section must list the minimal GPU requirements (especially VRAM).
-->

This package runs on any hardware that can run R.

## Environment Setup
<!--
1. The environment setup section should list all requirements and provide all further steps to prepare an environment for running the method (installing requirements, downloading files, creating directoriees, etc.).
2. The environment setup section should recommend to use a virtual environment or similar if the programming language supports one.
-->

With R installed:

```r
install.packages("oolong")
```

## How to Use
<!--
1. The how to use section should provide the list of steps that are necessary to produce the example output file (see section Output Data) after having set up the environment (see section Environment Setup).
2. The how to use section should explain how to customize the steps to one's own needs, usually through configuration files or command line parameters, or refer to the appropriate open documentation.
-->

Please refer to the [overview of this package](https://gesistsa.github.io/oolong/articles/overview.html) for a comprehensive introduction of all test types.

Expand Down Expand Up @@ -123,18 +158,30 @@ And then obtain the result of the test. For example:
oolong_test
```

## Contact Details

Maintainer: Chung-hong Chan <chainsawtiney@gmail.com>
## Technical Details
<!--
1. The technical details section should proview a process overview, linking to key source code files at every step of the process.
2. In case a publication provides the details mentioned below, the technical details section should link to this publication using a sentence like "See the [publication](url-of-publication-best-using-doi) for ...". In this case, the mentioned technical details can be omitted from the section.
3. The technical details section should list all information needed to reproduce the method, including employed other methods and selected parameters.
4. The input data section should link to external data it uses, preferably using a DOI to a dataset page or to API documentation.
5. The technical details section should mention how other methods and their parameters were selected and which alternatives were tried.
6. The technical details section should for employed machine learning models mention on what kind of data they were trained.
-->

Issue Tracker: [https://github.com/gesistsa/oolong/issues](https://github.com/gesistsa/oolong/issues)
See the official [CRAN repository](https://cran.r-project.org/web/packages/oolong/) for further information about technical details.

## Publication
## References
<!--
1. The references section is optional, especially if they are cited in a publication that explains the technical details (see section Technical Details).
2. The references section should provide references of publications related to this method (e.g., in APA7 style, https://apastyle.apa.org/style-grammar-guidelines/references/examples).
-->

1. Chan, C. H., & Sältzer, M. (2020). oolong: An R package for validating automated content analysis tools. The Journal of Open Source Software: JOSS, 5(55), 2461. <https://doi.org/10.21105/joss.02461>
Chan, C. H., & Sältzer, M. (2020). oolong: An R package for validating automated content analysis tools. The Journal of Open Source Software: JOSS, 5(55), 2461. <https://doi.org/10.21105/joss.02461>

<!-- ## Acknowledgements -->
<!-- - Acknowledgements if any -->
## Contact Details
<!--
1. The contact details section should specify whom to contact for questions or contributions and how (can be separate entitites; for example email addresses or links to the GitHub issue board).
-->

<!-- ## Disclaimer -->
<!-- - Add any disclaimers, legal notices, or usage restrictions for the method, if necessary. -->
Maintainer: Chung-hong Chan <chainsawtiney@gmail.com>
Issue Tracker: [https://github.com/gesistsa/oolong/issues](https://github.com/gesistsa/oolong/issues)
Loading