how to formulate a research topic pdf

The Sheridan Libraries

• Writing Resources
• Sheridan Libraries
• Develop a Research Topic or Question
• How to Access Full Text
• Google Scholar and Google Books
• Evaluate Your Sources This link opens in a new window
• Citing Sources This link opens in a new window
• Avoiding Plagiarism
• Books to Help You Write
• Copyright This link opens in a new window
• How To Read an Article
• Literature Reviews
• RefWorks Guide and Help This link opens in a new window
• Other Guides to Help You

Before You Start

• What do you already know about your subject? Keep a list of key words, names, and events.
• How long has your subject existed? Is it a relatively new concept with a lot published about it, or new and undiscovered?
• What discipline does your topic fall into? A discipline is an area of study or branch of learning (e.g., History, Biology). Each has its own best starting points.
• How are you viewing the topic? Think about what you are planning to emphasize: politics, history, or another aspect?
• What's the Timing? How long do you have to do this project? How long does it need to be?

Three Approaches for Developing a Topic

Approach #1: List Key Words of Interest Make lists of concepts and topics you find interesting, as well as lists of related words and synonyms. These can serve as your key search terms.

Approach #2: Draw It Out Sketch out the relationships between ideas.

Approach #3: Define it in Sentences Write an explanation of your topic, justifying it on multiple levels:

I am studying... conformity in Woolf’s Orlando in order to find out... how Orlando’s efforts to conform and fit in change over time in order to help my reader understand... the role maturity and self-awareness play in the character’s efforts to conform to societal norms.

Adapted from The Craft of Research (2003) by Wayne C. Booth, Gregory G. Colomb, and Joseph M. Williams. (We also own the latest edition, 8th edition, 2016 , in print.)

• << Previous: HOME - The Research Process
• Next: How to Access Full Text >>
• Last Updated: Jun 10, 2024 10:33 AM
• URL: https://guides.library.jhu.edu/writing

Writing Studio

Formulating your research question (rq).

In an effort to make our handouts more accessible, we have begun converting our PDF handouts to web pages. Download this page as a PDF: Formulating Your Research Question Return to Writing Studio Handouts

In a research paper, the emphasis is on generating a unique question and then synthesizing diverse sources into a coherent essay that supports your argument about the topic. In other words, you integrate information from publications with your own thoughts in order to formulate an argument. Your topic is your starting place: from here, you will develop an engaging research question. Merely presenting a topic in the form of a question does not transform it into a good research question.

Research Topic Versus Research Question Examples

1. broad topic versus narrow question, 1a. broad topic.

“What forces affect race relations in America?”

1b. NARROWER QUESTION

“How do corporate hiring practices affect race relations in Nashville?”

The question “What is the percentage of racial minorities holding management positions in corporate offices in Nashville?” is much too specific and would yield, at best, a statistic that could become part of a larger argument.

2. Neutral Topic Versus Argumentative Question

2a. neutral topic.

“How does KFC market its low-fat food offerings?”

2b. Argumentative question

“Does KFC put more money into marketing its high-fat food offerings than its lower-fat ones?”

The latter question is somewhat better, since it may lead you to take a stance or formulate an argument about consumer awareness or benefit.

3. Objective Topic Versus Subjective Question

Objective subjects are factual and do not have sides to be argued. Subjective subjects are those about which you can take a side.

3a. Objective topic

“How much time do youth between the ages of 10 and 15 spend playing video games?”

3b. Subjective Question

“What are the effects of video-gaming on the attention spans of youth between the ages of 10 and 15?”

The first question is likely to lead to some data, though not necessarily to an argument or issue. The second question is somewhat better, since it might lead you to formulate an argument for or against time spent playing video games.

4. Open-Ended Topic Versus Direct Question

4a. open-ended topic.

“Does the author of this text use allusion?”

4b. Direct question (gives direction to research)

“Does the ironic use of allusion in this text reveal anything about the author’s unwillingness to divulge his political commitments?”

The second question gives focus by putting the use of allusion into the specific context of a question about the author’s political commitments and perhaps also about the circumstances under which the text was produced.

Research Question (RQ) Checklist

• Is my RQ something that I am curious about and that others might care about? Does it present an issue on which I can take a stand?
• Does my RQ put a new spin on an old issue, or does it try to solve a problem?
• Is my RQ too broad, too narrow, or OK?
• within the time frame of the assignment?
• given the resources available at my location?
• Is my RQ measurable? What type of information do I need? Can I find actual data to support or contradict a position?
• What sources will have the type of information that I need to answer my RQ (journals, books, internet resources, government documents, interviews with people)?

Final Thoughts

The answer to a good research question will often be the THESIS of your research paper! And the results of your research may not always be what you expected them to be. Not only is this ok, it can be an indication that you are doing careful work!

Adapted from an online tutorial at Empire State College: http://www.esc.edu/htmlpages/writerold/menus.htm#develop (broken link)

Last revised: November 2022 | Adapted for web delivery: November 2022

In order to access certain content on this page, you may need to download Adobe Acrobat Reader or an equivalent PDF viewer software.

• Franklin University |
• Help & Support |
• Locations & Maps |

• | Research Guides

To access Safari eBooks,

• Select not listed in the Select Your Institution drop down menu.
• Enter your Franklin email address and click Go
• click "Already a user? Click here" link
• Enter your Franklin email and the password you used to create your Safari account.

Continue Close

Academic Writing Guide

How to develop a topic, background research, developing a research question or thesis statement, video: mapping your ideas.

• Find Information
• Understand / Evaluate Information

What is APA Style?

APA Style is a standardized writing format, established by the American Psychological Association, which you may need to follow when submitting projects or papers. If you have questions about APA formatting, look at our APA Style Guide .

• Tips on Developing a Topic
• Video: How to Develop a Good Research Topic
• C hoose a topic that interests you and will hold your attention. If you do, the research will be more enjoyable! 
• Choose a topic which has ample research available . When considering a topic, do a little preliminary searching to see if enough research exists for your to complete your assignment.
• Choose a topic that is not too broad.  You want to be able to cover your topic within the assignment parameters. If you are writing a 5 page paper, you will want to make sure your topic is narrow enough to cover in those 5 pages. 

Purpose of Background Research:

Provides a good overview of the topic if you are unfamiliar with it.

• Helps identify important facts -- terminology, dates, events, history, organizations, etc.
• Can help refine your topic
• Leads to bibliographies which provide additional sources of information

Recommended Databases for Background Research:

• Academic Search Complete This link opens in a new window Index, abstracts, and full text for many scholarly publications covering all academic areas of study.
• Newsbank This link opens in a new window Newsbank provides 12,000+ full text sources from over 200 countries and 250+ full color PDF versions of newspapers with same day delivery.
• Opposing Viewpoints in Context This link opens in a new window Covers today's hottest social issues, from capital punishment to immigration to marijuana. Presents each side of an issue with viewpoint reports, and adds journal articles, statistics, video/audio, news, and primary sources.
• Research Questions and Thesis Statements
• Other Sources of Information

After you have selected your topic and completed your background research, you must formulate a research question and/or thesis statement.

A research question is the topic you are trying to answer; a thesis statement provides the answer to that question.

• Research Question and Thesis Statement An overview of the research question and thesis statement from the Spartanburg Community College Library
• Starting Your Research: Research Question vs. Thesis Statement The difference between a research question and a thesis statement from the Harold Washington College Library
• Tips and Examples for Writing the Thesis Statement Tips for writing a thesis statement, with examples, from the Purdue OWL
• Where to Begin How to begin the writing process, from Purdue OWL. Includes a section on "writing your research question"

Video: Developing a Research Question

Video: Thesis Statements

• << Previous: Overview
• Next: Find Information >>
• Last Updated: May 29, 2024 3:15 PM
• URL: https://guides.franklin.edu/awg

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

HOW TO SELECT A RESEARCH TOPIC Selecting a Topic

Related Papers

Brian Mennecke , AM Townsend

Abstract This article provides an explanation of the process for selecting a research topic. The article uses Kuhn's classic work on scientific revolutions to delineate the steps in developing theoretical research within an area. The paper provides methods for preparing to develop a research topic, steps for approaching a research problem, as well as methods for problem theoretical development.

Furkan H . Yolcu

Selecting a research topic for dissertations can really trouble students sometimes here a quick and practicable quide

Jaqueline Kidd

After reading this chapter, you should be able to answer the following questions: • What are the initial steps for developing an action research project? • How do you generate a topic for action research? • How do you develop a question once you have chosen a topic? • Once you have developed a question, how do you proceed with your action research project? Chapter Aims and Goals The intent of this chapter is to initiate the strategic plan of your action research by identifying a topic of significance and to begin the process of formulating a research question to guide your study. As you proceed through this chapter, you will develop an understanding of • how to begin the action research process, • what makes for a meaningful and productive action research topic, • how to narrow the focus of potential topics, • how to clarify your topic by writing a statement of the problem, • how action research questions are formulated, and • how to evaluate your topic and potential research questions. The challenge of identifying a research topic for your action research project is that there are a multitude of possibilities for you to explore. Most teachers have many questions

Ricielle Precia Viluan

Your research paper, and the resulting thesis statement, must be an ARGUABLE issue. Be prepared to present the actual findings of your research convincingly even if you discover that your findings differ from your personal opinions. Remember, research is objective and not a " soap box " for personal views. The following topics have been divided by subject:

Lisa Hochtritt

Danilo Alain González

sylvia chard

International Journal of Innovative Research in Education

Miloud Bekkar

It is argued that the most critical moment in performing research work is the topic choice. This challenging step comes before undertaking any research work. Almost, the most common researcher’s anxiety that comes to his/her mind is about the success or on the contrary, the failure of the topic choice. This research aims to tackle the common challenges and difficulties while choosing one’s research problem. The study targets a group of postgraduate students of the English language Department at the University of Mascara. Around 25 subjects representing Master II 2021–2022 promotion participated in this study. The research tools include a questionnaire with students and a teacher’s experience in teaching Research Methodology Module at Mascara University. Most participants prefer the topics proposed by their future supervisors. Also, the study tempts to give recommendations for developing students’ research topics and titles. Keywords: Methodology, research work, students, topi...

Alan Dennis

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED PAPERS

Pragyaratna

Balram Adhikari

Elder Clair Rudison

College & Research Libraries

Melissa Bowles-Terry

Asian Journal of Education and Social Studies

jigme dorji

freddy leen

Jeff Immanuel

jpma.pakcyber.biz

Junaid RAza

JEMS : a journal of emergency medical services

Joseph Ornato

Sunil Prasad

Conor Horan

Lisa Adams , Gregory McAboy MEd

Anita Knight

Bakhtawer Zain

Dr. Awais H. Gillani

Dr Sunarsih

hakimi baharudin

Tutors India

Debbie Epstein

Science Park Research Organization & Counselling

S M Mukarram Jahan

Respiratory care

Charles Durbin

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

Build a Corporate Culture That Works

There’s a widespread understanding that managing corporate culture is key to business success. Yet few companies articulate their culture in such a way that the words become an organizational reality that molds employee behavior as intended.

All too often a culture is described as a set of anodyne norms, principles, or values, which do not offer decision-makers guidance on how to make difficult choices when faced with conflicting but equally defensible courses of action.

The trick to making a desired culture come alive is to debate and articulate it using dilemmas. If you identify the tough dilemmas your employees routinely face and clearly state how they should be resolved—“In this company, when we come across this dilemma, we turn left”—then your desired culture will take root and influence the behavior of the team.

To develop a culture that works, follow six rules: Ground your culture in the dilemmas you are likely to confront, dilemma-test your values, communicate your values in colorful terms, hire people who fit, let culture drive strategy, and know when to pull back from a value statement.

Start by thinking about the dilemmas your people will face.

Idea in Brief

The problem.

There’s a widespread understanding that managing corporate culture is key to business success. Yet few companies articulate their corporate culture in such a way that the words become an organizational reality that molds employee behavior as intended.

What Usually Happens

How to fix it.

Follow six rules: Ground your culture in the dilemmas you are likely to confront, dilemma-test your values, communicate your values in colorful terms, hire people who fit, let culture drive strategy, and know when to pull back from a value.

At the beginning of my career, I worked for the health-care-software specialist HBOC. One day, a woman from human resources came into the cafeteria with a roll of tape and began sticking posters on the walls. They proclaimed in royal blue the company’s values: “Transparency, Respect, Integrity, Honesty.” The next day we received wallet-sized plastic cards with the same words and were asked to memorize them so that we could incorporate them into our actions. The following year, when management was indicted on 17 counts of conspiracy and fraud, we learned what the company’s values really were.

• EM Erin Meyer is a professor at INSEAD, where she directs the executive education program Leading Across Borders and Cultures. She is the author of The Culture Map: Breaking Through the Invisible Boundaries of Global Business (PublicAffairs, 2014) and coauthor (with Reed Hastings) of No Rules Rules: Netflix and the Culture of Reinvention (Penguin, 2020). ErinMeyerINSEAD

Partner Center

How IBM helps Wimbledon use generative AI to drive personalised fan engagement

This collaboration with Wimbledon teams extends beyond the fan-facing digital platform, into enterprise-wide transformation.

Top 7 risks to your identity security posture

5 min read - Identity misconfigurations and blind spots stand out as critical concerns that undermine an organization’s identity security posture.

Intesa Sanpaolo and IBM secure digital transactions with fully homomorphic encryption

6 min read - Explore how European bank Intesa Sanpaolo and IBM partnered to deliver secure digital transactions using fully homomorphic encryption.

What is AI risk management?

8 min read - AI risk management is the process of identifying, mitigating and addressing the potential risks associated with AI technologies.

How IBM and AWS are partnering to deliver the promise of responsible AI

4 min read - This partnership between IBM and Amazon SageMaker is poised to play a pivotal role in shaping responsible AI practices across industries

June 27, 2024

IBM announces new AI assistant and feature innovations at Think 2024

June 26, 2024

A major upgrade to Db2® Warehouse on IBM Cloud®

June 25, 2024

Increase efficiency in asset lifecycle management with Maximo Application Suite’s new AI-power...

Achieving operational efficiency through Instana’s Intelligent Remediation

June 24, 2024

Manage the routing of your observability log and event data 

Best practices for augmenting human intelligence with AI

2 min read - Enabling participation in the AI-driven economy to be underpinned by fairness, transparency, explainability, robustness and privacy. 

Microcontrollers vs. microprocessors: What’s the difference?

6 min read - Microcontroller units (MCUs) and microprocessor units (MPUs) are two kinds of integrated circuits that, while similar in certain ways, are very different in many others.

Mastering budget control in the age of AI: Leveraging on-premises and cloud XaaS for success 

2 min read - As organizations harness the power of AI while controlling costs, leveraging anything as a service (XaaS) models emerges as strategic.

Highlights by topic

Use IBM Watsonx’s AI or build your own machine learning models

Automate IT infrastructure management

Cloud-native software to secure resources and simplify compliance

Run code on real quantum systems using a full-stack SDK

Aggregate and analyze large datasets

Store, query and analyze structured data

Manage infrastructure, environments and deployments

Run workloads on hybrid cloud infrastructure

Responsible AI can revolutionize tax agencies to improve citizen services

Generative AI can revolutionize tax administration and drive toward a more personalized and ethical future.

Speed, scale and trustworthy AI on IBM Z with Machine Learning for IBM z/OS v3.2 

4 min read - Machine Learning for IBM® z/OS® is an AI platform made for IBM z/OS environments, combining data and transaction gravity with AI infusion.

The recipe for RAG: How cloud services enable generative AI outcomes across industries

4 min read - While the AI is the key component of the RAG framework, other “ingredients” such as PaaS solutions are integral to the mix

Rethink IT spend in the age of generative AI

3 min read - It's critical for organizations to consider frameworks like FinOps and TBM for visibility and accountability of all tech expenditure.

6 hard truths CEOs must confront in the generative AI era

5 min read - A call to action for CEOs to confront the realities of generative AI and to seize its potential for your organization.

Immutable backup strategies with cloud storage

4 min read - IBM Cloud Object Storage is a versatile and scalable solution that is crucial for storing and protecting data backups.

T-Mobile unlocks marketing efficiency with Adobe Workfront

4 min read - Discover how the T-Mobile team creates a more seamless work management system for its content supply chain with Adobe and IBM Consulting.

IBM Newsletters

• Comment Comments
• Save Article Read Later Read Later

The Unpredictable Abilities Emerging From Large AI Models

March 16, 2023

Paul Chaikin/Quanta Magazine

Introduction

What movie do these emojis describe?

That prompt was one of 204 tasks chosen last year to test the ability of various large language models (LLMs) — the computational engines behind AI chatbots such as ChatGPT. The simplest LLMs produced surreal responses. “The movie is a movie about a man who is a man who is a man,” one began. Medium-complexity models came closer, guessing The Emoji Movie. But the most complex model nailed it in one guess: Finding Nemo .

“Despite trying to expect surprises, I’m surprised at the things these models can do,” said Ethan Dyer , a computer scientist at Google Research who helped organize the test. It’s surprising because these models supposedly have one directive: to accept a string of text as input and predict what comes next, over and over, based purely on statistics. Computer scientists anticipated that scaling up would boost performance on known tasks, but they didn’t expect the models to suddenly handle so many new, unpredictable ones.

Recent investigations like the one Dyer worked on have revealed that LLMs can produce hundreds of “emergent” abilities — tasks that big models can complete that smaller models can’t, many of which seem to have little to do with analyzing text. They range from multiplication to generating executable computer code to, apparently, decoding movies based on emojis. New analyses suggest that for some tasks and some models, there’s a threshold of complexity beyond which the functionality of the model skyrockets. (They also suggest a dark flip side: As they increase in complexity, some models reveal new biases and inaccuracies in their responses.)

“That language models can do these sort of things was never discussed in any literature that I’m aware of,” said Rishi Bommasani , a computer scientist at Stanford University. Last year, he helped compile a list of dozens of emergent behaviors, including several identified in Dyer’s project. That list continues to grow .

Now, researchers are racing not only to identify additional emergent abilities but also to figure out why and how they occur at all — in essence, to try to predict unpredictability. Understanding emergence could reveal answers to deep questions around AI and machine learning in general, like whether complex models are truly doing something new or just getting really good at statistics. It could also help researchers harness potential benefits and curtail emergent risks.

“We don’t know how to tell in which sort of application is the capability of harm going to arise, either smoothly or unpredictably,” said Deep Ganguli , a computer scientist at the AI startup Anthropic.

The Emergence of Emergence

Biologists, physicists, ecologists and other scientists use the term “emergent” to describe self-organizing, collective behaviors that appear when a large collection of things acts as one. Combinations of lifeless atoms give rise to living cells; water molecules create waves; murmurations of starlings swoop through the sky in changing but identifiable patterns; cells make muscles move and hearts beat. Critically, emergent abilities show up in systems that involve lots of individual parts. But researchers have only recently been able to document these abilities in LLMs as those models have grown to enormous sizes.

Language models have been around for decades. Until about five years ago, the most powerful were based on what’s called a recurrent neural network. These essentially take a string of text and predict what the next word will be. What makes a model “recurrent” is that it learns from its own output: Its predictions feed back into the network to improve future performance.

In 2017, researchers at Google Brain introduced a new kind of architecture called a transformer . While a recurrent network analyzes a sentence word by word, the transformer processes all the words at the same time. This means transformers can process big bodies of text in parallel.

Transformers enabled a rapid scaling up of the complexity of language models by increasing the number of parameters in the model, as well as other factors. The parameters can be thought of as connections between words, and models improve by adjusting these connections as they churn through text during training. The more parameters in a model, the more accurately it can make connections, and the closer it comes to passably mimicking human language. As expected, a 2020 analysis by OpenAI researchers found that models improve in accuracy and ability as they scale up.

But the debut of LLMs also brought something truly unexpected. Lots of somethings. With the advent of models like GPT-3, which has 175 billion parameters — or Google’s PaLM, which can be scaled up to 540 billion — users began describing more and more emergent behaviors. One DeepMind engineer even reported being able to convince ChatGPT that it was a Linux terminal and getting it to run some simple mathematical code to compute the first 10 prime numbers. Remarkably, it could finish the task faster than the same code running on a real Linux machine.

As with the movie emoji task, researchers had no reason to think that a language model built to predict text would convincingly imitate a computer terminal. Many of these emergent behaviors illustrate “zero-shot” or “few-shot” learning, which describes an LLM’s ability to solve problems it has never — or rarely — seen before. This has been a long-time goal in artificial intelligence research, Ganguli said. Showing that GPT-3 could solve problems without any explicit training data in a zero-shot setting, he said, “led me to drop what I was doing and get more involved.”

He wasn’t alone. A raft of researchers, detecting the first hints that LLMs could reach beyond the constraints of their training data, are striving for a better grasp of what emergence looks like and how it happens. The first step was to thoroughly document it.

Ethan Dyer helped probe what kinds of unexpected abilities large language models were capable of, and what could bring them about.

Gabrielle Lurie

Beyond Imitation

In 2020, Dyer and others at Google Research predicted that LLMs would have transformative effects — but what those effects would be remained an open question. So they asked the research community to provide examples of difficult and diverse tasks to chart the outer limits of what an LLM could do. This effort was called the Beyond the Imitation Game Benchmark (BIG-bench) project, riffing on the name of Alan Turing’s “imitation game,” a test for whether a computer could respond to questions in a convincingly human way. (This would later become known as the Turing test.) The group was especially interested in examples where LLMs suddenly attained new abilities that had been completely absent before.

“How we understand these sharp transitions is a great research question,” Dyer said.

As one would expect, on some tasks a model’s performance improved smoothly and predictably as complexity increased. And on other tasks, scaling up the number of parameters did not yield any improvement. But for about 5% of the tasks, the researchers found what they called “breakthroughs” — rapid, dramatic jumps in performance at some threshold scale. That threshold varied based on the task and model.

For example, models with relatively few parameters — only a few million — could not successfully complete three-digit addition or two-digit multiplication problems, but for tens of billions of parameters, accuracy spiked in some models. Similar jumps occurred for other tasks including decoding the International Phonetic Alphabet, unscrambling a word’s letters, identifying offensive content in paragraphs of Hinglish (a combination of Hindi and English), and generating a similar English equivalent of Kiswahili proverbs.

But the researchers quickly realized that a model’s complexity wasn’t the only driving factor. Some unexpected abilities could be coaxed out of smaller models with fewer parameters — or trained on smaller data sets — if the data was of sufficiently high quality. In addition, how a query was worded influenced the accuracy of the model’s response. When Dyer and his colleagues posed the movie emoji task using a multiple-choice format, for example, the accuracy improvement was less of a sudden jump and more of a gradual increase with more complexity. And last year, in a paper presented at NeurIPS , the field’s flagship meeting, researchers at Google Brain showed how a model prompted to explain itself (a capacity called chain-of-thought reasoning) could correctly solve a math word problem, while the same model without that prompt could not.

Yi Tay , a scientist at Google Brain who worked on the systematic investigation of breakthroughs, points to recent work suggesting that chain-of-thought prompting changes the scaling curves and therefore the point where emergence occurs. In their NeurIPS paper, the Google researchers showed that using chain-of-thought prompts could elicit emergent behaviors not identified in the BIG-bench study. Such prompts, which ask the model to explain its reasoning, may help researchers begin to investigate why emergence occurs at all.

Recent findings like these suggest at least two possibilities for why emergence occurs, said Ellie Pavlick , a computer scientist at Brown University who studies computational models of language. One is that, as suggested by comparisons to biological systems, larger models truly do gain new abilities spontaneously. “It may very well be that the model has learned something fundamentally new and different that it didn’t have at a smaller size,” she said. “That’s what we’re all hoping is the case, that there’s some fundamental shift that happens when models are scaled up.”

The other, less sensational possibility, she said, is that what appears to be emergent may instead be the culmination of an internal, statistics-driven process that works through chain-of-thought-type reasoning. Large LLMs may simply be learning heuristics that are out of reach for those with fewer parameters or lower-quality data.

But, she said, finding out which of those explanations is more likely hinges on a better understanding of how LLMs work at all. “Since we don’t know how they work under the hood, we can’t say which of those things is happening.”

Unpredictable Powers and Pitfalls

There is an obvious problem with asking these models to explain themselves: They are notorious liars. “We’re increasingly relying on these models to do basic work,” Ganguli said, “but I do not just trust these. I check their work.” As one of many amusing examples, in February Google introduced its AI chatbot, Bard. The blog post announcing the new tool shows Bard making a factual error .

Emergence leads to unpredictability, and unpredictability — which seems to increase with scaling — makes it difficult for researchers to anticipate the consequences of widespread use.

“It’s hard to know in advance how these models will be used or deployed,” Ganguli said. “And to study emergent phenomena, you have to have a case in mind, and you won’t know until you study the influence of scale what capabilities or limitations might arise.”

In an analysis of LLMs released last June, researchers at Anthropic looked at whether the models would show certain types of racial or social biases, not unlike those previously reported in non-LLM-based algorithms used to predict which former criminals are likely to commit another crime. That study was inspired by an apparent paradox tied directly to emergence: As models improve their performance when scaling up, they may also increase the likelihood of unpredictable phenomena, including those that could potentially lead to bias or harm.

“Certain harmful behaviors kind of come up abruptly in some models,” Ganguli said. He points to a recent analysis of LLMs, known as the BBQ benchmark, which showed that social bias emerges with enormous numbers of parameters. “Larger models abruptly become more biased.” Failure to address that risk, he said, could jeopardize the subjects of these models.

But he offers a counterpoint: When the researchers simply told the model not to rely on stereotypes or social biases — literally by typing in those instructions — the model was less biased in its predictions and responses. This suggests that some emergent properties might also be used to reduce bias. In a paper released in February, the Anthropic team reported on a new “moral self-correction” mode, in which the user prompts the program to be helpful, honest and harmless.

Emergence, Ganguli said, reveals both surprising potential and unpredictable risk. Applications of these large LLMs are already proliferating, so a better understanding of that interplay will help harness the diversity of abilities of language models.

“We’re studying how people are actually using these systems,” Ganguli said. But those users are also tinkering, constantly. “We spend a lot of time just chatting with our models,” he said, “and that is actually where you start to get a good intuition about trust — or the lack thereof.”

Get highlights of the most important news delivered to your email inbox

Comment on this article

Quanta Magazine moderates comments to facilitate an informed, substantive, civil conversation. Abusive, profane, self-promotional, misleading, incoherent or off-topic comments will be rejected. Moderators are staffed during regular business hours (New York time) and can only accept comments written in English. 

Next article