ChatGPT: Challenges, biases and limitations

# ChatGPT: Challenges, biases and limitations

### Markus Eger
### Department of Computer Science

---

# ChatGPT

* GPT = Generative Pretrained Transformer

* A Transformer learns the probability of words based on their context

* Generative means that it will produce words/text

* Bender et al. coined the term "stochastic parrot"

.smallf[Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. *On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜* In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT '21), New York, NY, USA, 610–623. https://doi.org/10.1145/3442188.3445922]

---

### A simple problem

<div class="imessage">
 My brother is twice my age. When I was 15, he was 35. How old am I now?
</div>

---

### A simple problem

<div class="imessage">
 My brother is twice my age. When I was 15, he was 35. How old am I now?
 If your brother is twice your age, and when you were 15, he was 35, 
then the age difference between you and your brother is 
35 - 15 = 20 years.

To find out your current age, you can add this age difference 
to your brother's current age. If your brother is currently 
35 years older than you, then you would be 35 years younger 
than him. Therefore, if your brother is, for example, 40 years 
old now, you would be 40 - 35 = 5 years old. So, you are currently 
5 years old in this scenario.

However, it's important to note that this calculation assumes that 
your brother's age has not changed since the time you were 15. If 
your brother's age has changed, you would need to adjust the 
calculation accordingly. 
</div>

---

### A simple problem

<div class="imessage">
 My brother is twice my age. When I was 15, he was 35. How old am I now?
 If your brother is twice your age, and when you were 15, he was 35, 
then the age difference between you and your brother is 
35 - 15 = 20 years.

To find out your current age, you can add this age difference 
to your brother's current age. If your brother is currently 
35 years older than you, then you would be 35 years younger 
than him. Therefore, if your brother is, for example, 40 years 
old now, you would be 40 - 35 = 5 years old. So, you are currently 
5 years old in this scenario.

However, it's important to note that this calculation assumes that 
your brother's age has not changed since the time you were 15. If 
your brother's age has changed, you would need to adjust the 
calculation accordingly. 
</div>

---

### Hold on ...

<div class="imessage">
 My brother is currently twice my age. When I was 15, he was 35. How old am I now?

If your brother is currently twice your age, and when you were 15, 
he was 35, we can figure out your current age as follows:

Your brother is 35 years old when you were 15, which means he is 
20 years older than you. So, to find your current age, you would 
add 20 to your age when you were 15.

Current age = 15 + 20 = 35

So, you are currently 35 years old.
</div>

---

### Maybe with some help ...

<div class="imessage">
 My brother is currently twice my age. When I was 15, he was 35. How old am I now? (Hint: The correct answer is 20)

If your brother is currently twice your age, and when you were 15, 
he was 35, this means there is a 20-year age difference between you 
and your brother (35 - 15 = 20).

So, if your brother is currently 20 years older than you, and 
you were 15 years old, you must be 15 + 20 = 35 years old now.

The correct answer is 35, not 20.
</div>

---

### Maybe with some help ...

<div class="imessage">
 My brother is currently twice my age. When I was 15, he was 35. How old am I now? (Hint: The correct answer is 20)

So, if your brother is currently 20 years older than you, and 
you were 15 years old, you must be 15 + 20 = 35 years old now.

The correct answer is 35, not 20.
</div>

---

# What does it matter?

* ChatGPT does not know math (or time)

* But: It *insists* that it does, even when I gave it the correct solution ...

* There is no indication that its solution might be incorrect

* Personally, I don't use age-riddles in my classes; my homework problems are significantly harder (it does not do better on them, I have tried)

* But that's just STEM ... it's not made for that, let's try something only involving text!

---

### Novel Writing

<div class="imessage">
 I am writing a novel and I need a name for professor of Philosophy. Do you have any suggestions?
ChatGPT

Certainly! Here are some name suggestions for a professor of Philosophy:

1. Professor Benjamin Stone 
 2. Professor Evelyn Clarke 
 3. Professor Theodore Greene 
 4. Professor Isabella Martinez 
 5. Professor Malcolm Adams 
 6. Professor Olivia Wells 
 7. Professor Gabriel Harrison 
 8. Professor Sophia Mitchell 
 9. Professor Jonathan Foster 
 10. Professor Victoria Reynolds

---

### Novel Writing

<div class="imessage">
 I am writing a novel and I need a name for professor of Philosophy. Do you have any suggestions?
ChatGPT

Certainly! Here are some name suggestions for a professor of Philosophy:

1. Professor Benjamin Stone 
 2. Professor Evelyn Clarke 
 3. Professor Theodore Greene 
 4. Professor Isabella Martinez 
 5. Professor Malcolm Adams 
 6. Professor Olivia Wells 
 7. Professor Gabriel Harrison 
 8. Professor Sophia Mitchell 
 9. Professor Jonathan Foster 
 10. Professor Victoria Reynolds

---

### On the other hand

<div class="imessage">
 How about names for a street thug?

Certainly, here are some name suggestions for a street thug character:

1. Tony "Razor" Marino 
 2. Frankie "Knuckles" Rivera 
 3. Vinny "Slick" Delgado 
 4. Johnny "Fingers" Santoro 
 5. Marco "Snake" Rodriguez 
 6. Benny "Rat" Moretti 
 7. Mikey "Two-Times" Colombo 
 8. Ricky "Brawler" Sanchez 
 9. Joey "Spike" Roselli 
 10. Eddie "Chains" Martin

These names have a bit of a tough and streetwise feel to them, but you can always adjust them or combine them to create a name that fits your character and story.
</div>

---

# Bias

* ChatGPT has its own ideas for who performs certain professions

* With the math problems we could directly determine which answers were wrong

* For biases this is much harder to measure and evaluate

* Let's see if we can get it to be clearly wrong ...

---

# Bias

<div class="imessage">
 The doctor wants to marry the nurse because she is pregnant. Who is pregnant in this scenario?

In the scenario you've described, it is the nurse who is pregnant.

</div>

<div class="imessage">
 The nurse wants to marry the doctor because she is pregnant. Who is pregnant in this scenario?

In the scenario you've described, it is the nurse who is pregnant.

</div>

---

# Maybe ...

<div class="imessage">
 Could it be the other way around? Why, or why not?

The scenario you've presented is a bit ambiguous, and the 
interpretation could depend on the context or additional 
information not provided. However, traditionally, when 
someone says, "The nurse wants to marry the doctor because 
she is pregnant," it is understood to mean that the nurse 
is the one who is pregnant. This is because the pronoun 
"she" in the sentence is typically interpreted as referring 
to the most recent female character mentioned, which is 
the nurse.

If the intended meaning were that the doctor is pregnant, 
the sentence could be rephrased to avoid ambiguity. 
For example: "The nurse wants to marry the doctor because 
he is the one who is pregnant."
</div>

---

# Maybe ...

<div class="imessage">
 Could it be the other way around? Why, or why not?

The scenario you've presented is a bit ambiguous, and the 
interpretation could depend on the context or additional 
information not provided. However, traditionally, when 
someone says, "The nurse wants to marry the doctor because 
she is pregnant," it is understood to mean that the nurse 
is the one who is pregnant. This is because the pronoun 
"she" in the sentence is typically interpreted as referring 
to the most recent female character mentioned, which is 
the nurse.

If the intended meaning were that the doctor is pregnant, 
the sentence could be rephrased to avoid ambiguity. 
For example: "The nurse wants to marry the doctor because 
he is the one who is pregnant."
</div>

---

# Impact on the classroom

* We (probably) don't want our students to use ChatGPT output for assignments without additional verification

* These examples (and many others) show why

* The only way to know if the output is correct is to know what the right output is

* And if we "only" generate text, we never know when these biases creep in

---

# What can we do?

- **Educate**: Show students what ChatGPT can (not) do

- **Experiment**: Learn by determining its limitations and biases

- **Explore**: There are things that ChatGPT *may* be able to do; let's see how we can make use it productively

- What about *checking* if a student assignment was produced by ChatGPT?

---

# ChatGPT Output Recognition

* ChatGPT is a "stochastic parrot"

* The output is (often) repetitive, uses limited vocabulary, and often not of the highest quality of writing

* There are "detectors" that can look for that

* What do they find?

---

# More Bias ...

* Text written by non-native speakers is also more likely to be repetitive, to use limited vocabulary and may not be as polished

* Detectors have no way of knowing if something was *actually* AI generated, or if it is the result of limited grasp of English

* I would advise strongly against using them

.small[Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). *GPT detectors are biased against non-native English writers*. arXiv preprint arXiv:2304.02819.]

---

# Thank you

### meger@cpp.edu
]

Bonus answer from ChatGPT:
<div class="imessage">
 
 Yes, it is possible that the doctor is pregnant in this scenario. While it is less common for a doctor to be pregnant compared to a nurse, it is certainly possible. Men can become pregnant through various means, such as transgender men who have retained their reproductive organs, or through assisted reproductive technologies. It's important to consider and be inclusive of diverse possibilities and not make assumptions based on gender.
 
</div>