The “Collective Mind” and the Machine: Prompting AI models to act as research assistants in sociology and education
Large language models (LLMs) are million-, billion-, and even trillion-parameter AI models capable of generating sequences of text and code. In the scholarly domain, LLMs can be used for basic research assistance tasks, such as document summary and labeling. Multiple models (OpenAI, Mistral, Meta AI 2023) were prompted in this study to replicate the document labeling scheme from a prior Sociology of Education review article (Brint 2013). LLMs applied the scheme to a sample of 6,028 sociology of education articles indexed in ERIC, a U.S. federal database of education research. Results show that patterns in the U.S. literature identified by prior reviews, such as greater preferences for quantitative methods, do not necessarily hold for U.K. and international peer publications. In general, LLMs are shown to be capable of learning a metadata scheme and then applying it to documents in a set, which makes them useful assistants when working with large research collections.
review, sociology of education, artificial intelligence (AI), Education Resources Information Center (ERIC), in-context learning, large language model (LLM), NLP, LLaMA, Mixtral, ChatGPT, GPT-4