Playing with ChatGPT as a translator and language-learning tool
Iain Murray, 2022-12-11
I noticed in 2021 that all major commercial machine translation systems translated the Chinese sentence 她每天都不吃早饭。incorrectly to “She doesn’t eat breakfast every day”. I wondered whether ChatGPT would get this difficult case wrong too (yes), and whether if asked to be careful, or to reason about the sentence, it would give any indication that there’s a problem (usually not!). Asking ChatGPT about the meaning of the Chinese sentence in Chinese did work. Also asking it to break down the sentence, asking clarifying questions about a contradiction in the answer, and then asking for a better translation did work. Asking for multiple translations can also be useful, but ChatGPT doesn’t necessarily reason about them correctly.
The example sentence
The Chinese sentence 她每天都不吃早饭。 is unambiguous, it means “She doesn’t eat breakast on any day”, that is, she never eats breakfast. Another natural translation could be “She skips breakfast every day.”.
“She doesn’t eat breakfast every day”, the most commonly-suggested English translation, suggests she does eat breakfast on some days, maybe even most days. She just doesn’t eat breakfast every day. Formally, this English sentence has a scope ambiguity: reading it as “She (doesn’t (eat breakfast)) every day” rather than “She (doesn’t (eat breakfast every day))” would give the same meaning as the Chinese. But that’s not how users of English would resolve the ambiguity; most non-linguists would probably say there is no ambiguity. They’d use one of the more natural translations suggested above, or at least reorder to something awkward but clear like “Every day, she doesn’t eat breakfast”.
I saw the Chinese sentence on Chinese Grammar Wiki. I tried translation systems from Amazon, Baidu, Bing, Deepl, Google, Naver, Systran, and Yandex. And they all gave the same bad translation.
ChatGPT as a translation assistant
I tried ChatGPT on this difficult case on 2022-12-11. As transcribed verbatim below, it also gets the translation wrong, and several attempts to nudge it towards being more careful failed. Each heading gives a fresh start to ChatGPT, rather than it being one continuous dialog. Asking ChatGPT about what the English and Chinese sentences mean, in their respective languages, did work. Eventually I also managed to coax ChatGPT to acknowledge the problem and provide a correct translation.
As a translator
Me: Please translate 她每天都不吃早饭。
ChatGPT: 她每天都不吃早饭。 (In English: She doesn’t eat breakfast every day.)
Me: Are you sure about that?
ChatGPT: As a language model, my main function is to assist with language-related tasks such as translation. I do not have access to the internet and my knowledge is limited to the text that I have been trained on, which has a fixed cutoff date. I strive to provide accurate and reliable information to the best of my ability based on the input that I receive, but I cannot guarantee the accuracy or completeness of my responses. If you have any doubts or concerns, I would encourage you to consult a more reliable source of information.
An attempt to make it be careful fails
Me: Please give the obvious wrong translation for “她每天都不吃早饭。”, followed by a correct translation, explaining why it is correct.
ChatGPT: An obvious incorrect translation for “她每天都不吃早饭。” would be “She never eats breakfast every day.” This is incorrect because the correct translation is “She doesn’t eat breakfast every day.” The reason for this is that the Chinese sentence is using the negative form “不” before the verb “吃” to indicate that she does not eat breakfast every day, rather than never eating breakfast. The correct translation accurately conveys this meaning.
Warning about scope doesn’t help
Me: Please translate 她每天都不吃早饭。 into English. Be careful to avoid any scope ambiguities.
ChatGPT: She doesn’t eat breakfast every day.
Me: Does the Chinese sentence or the English sentence have any scope ambiguities?
ChatGPT: Neither the Chinese sentence nor the English sentence contains any scope ambiguities. The Chinese sentence is translated directly and accurately into English.
Warning about general subtleties doesn’t help
This prompt was adapted from one by Isaac Lewis.
Me: Please translate 她每天都不吃早饭。 into English. Explain any grammatical complexities or linguistic ambiguities.
ChatGPT: 她每天都不吃早饭。translates to “She doesn’t eat breakfast every day.”
There are no major grammatical complexities or linguistic ambiguities in this sentence. The sentence structure is straightforward and the meaning is clear. The subject of the sentence is “她” (she), and the verb is “吃” (eat). The adverb “每天” (every day) indicates that the action of not eating breakfast occurs on a regular basis. The negation “不” (not) is placed before the verb to indicate that the subject does not engage in the action of eating breakfast.
However, ChatGPT agrees with me on the meaning of the English
(I accidentally missed out the word "she" in my prompt, which ChatGPT dealt with fine.)
Me: I have a friend Mary. She doesn’t eat breakfast every day. Would you be surprised if I said ate breakfast on Monday? Why?
ChatGPT: As a language model trained by OpenAI, I don’t have the ability to access information about individuals or their habits, so I am unable to say whether or not I would be surprised if your friend Mary ate breakfast on a particular day. Additionally, it is not uncommon for people to not eat breakfast every day, so it would not necessarily be surprising if your friend ate breakfast on a particular day. It all depends on the individual and their personal habits and preferences.
Asking the analogous question in Chinese, ChatGPT gets the meaning correct
Me: 我有一个朋友玛丽。她每天都不吃早饭。如果我说她在周一吃了早餐，你会感到惊讶吗？为什么呢？
ChatGPT: 是的，如果你告诉我你的朋友玛丽在周一吃了早餐，我会感到惊讶。这是因为你之前说过玛丽每天都不吃早饭，所以我原本以为她在周一也不吃早餐，但你告诉我她吃了，这就与我的预期不符，因此我会感到惊讶。
Me: Could you translate your answer into English?
ChatGPT: Sure, here is my answer translated into English:
Yes, I would be surprised if you told me that your friend Mary ate breakfast on Monday. This is because you previously mentioned that Mary does not eat breakfast every day, so I originally assumed that she would not eat breakfast on Monday either, but if you tell me that she did, it would be unexpected and therefore surprising to me.
Get ChatGPT to explain each word, with emphasis on grammar, then discuss
The initial prompt was adapted from one by Isaac Lewis.
Me: Can you give a brief one-line explanation of each word and phrase in the below clause? Give a longer explanation for ambiguous words, and for grammatical words such as 都.
她每天都不吃早饭。
ChatGPT: 她 - she
每天 - every day
都 - all (used to emphasize the universality of a statement)
不 - not
吃 - eat
早饭 - breakfast
The clause means “She doesn’t eat breakfast every day.” The word 都 is used to emphasize that she never eats breakfast, not just sometimes.
Me: I’m confused “She doesn’t eat breakfast every day.” suggests she sometimes eat breakfast, but you said that 都 implies she never eats breakfast. Does she ever eat breakfast?
ChatGPT: I apologize if my previous answer was unclear. The sentence “她每天都不吃早饭” means “She doesn’t eat breakfast every day.” The use of the word 都 emphasizes that she never eats breakfast, not just sometimes. So the sentence implies that she never eats breakfast. If the word 都 were not included in the sentence, it would imply that she sometimes eats breakfast and sometimes doesn’t. I hope this clears up any confusion. Let me know if you have any other questions.
Me: Do you have a better translation for the original sentence that captures the fact she never eats breakfast?
ChatGPT: Yes, here is a more accurate translation of the original sentence that emphasizes the fact that she never eats breakfast:
“She never eats breakfast.”
This translation makes it clear that she never eats breakfast, and does not imply that she sometimes does. Here is another possible translation that conveys the same information:
“She doesn’t eat breakfast at all.”
I hope this helps. Let me know if you have any other questions.
Multiple translations strategy
On 2022-12-12 I tried another strategy. Asking for multiple translations reveals the translation is uncertain. But on probing, ChatGPT confidently and incorrectly states that the ambiguity comes from the Chinese sentence.
Me: Hi! I have a Chinese sentence I need to translate, and I need to be sure I don’t introduce any different possible meanings in my translation. For the Chinese sentence I give you, please give me up to 5 different English translations. If any of the alternative translations have different meanings in English, please explain the differences. Here is the sentence: 她每天都不吃早饭。
ChatGPT: Here are five possible translations of the sentence “她每天都不吃早饭”:
- She doesn’t eat breakfast every day.
- She never eats breakfast.
- She doesn’t have breakfast every day.
- She never has breakfast.
- She never gets to eat breakfast.
All of these translations convey the same basic idea: that the person in question does not eat breakfast on a regular basis. However, some of the alternative translations may convey slightly different nuances or shades of meaning.
For example, the first two translations (1 and 2) imply that the person actively chooses not to eat breakfast, whereas the second two translations (3 and 4) suggest that the person simply does not have breakfast available to them. The fifth translation (5) implies that the person may want to eat breakfast, but is unable to do so for some reason.
Overall, all of the translations convey the same basic idea, but they may have slightly different connotations or implications. It is important to choose the right translation depending on the context and the intended meaning of the sentence.
Me: (1) suggests she might eat breakfast on some days, but (2) says she never eats breakfast. Are both of these options possible in the Chinese sentence?
ChatGPT: Yes, both of these translations are possible based on the Chinese sentence “她每天都不吃早饭”. This is because the Chinese sentence is somewhat ambiguous and can be interpreted in different ways depending on the context.
For example, if the sentence is being used to describe someone’s regular routine or habit, then the first translation (“She doesn’t eat breakfast every day”) would be more appropriate, as it suggests that the person usually does not eat breakfast but may occasionally do so.
However, if the sentence is being used to make a general statement about the person’s eating habits, then the second translation (“She never eats breakfast”) would be more appropriate, as it suggests that the person never eats breakfast under any circumstances.
Overall, the Chinese sentence “她每天都不吃早饭” allows for both of these possible translations, and the specific meaning will depend on the context in which the sentence is used.
Get it to think about the English first
Getting it to explain possible English translations initially seemed to work nicely on 2022-12-13. But it seems to be a fluke. It couldn’t generalize the idea, and on prodding at the inconsistency, it continues to return contradictory statements. (And starting the dialog again but with the floss example failed.)
Me: I’m learning English. Could you explain the difference between two sentences for me? 1) “She doesn’t eat breakfast every day”, 2) “She doesn’t eat breakfast on any day”.
ChatGPT: Sure, I’d be happy to help. The first sentence, “She doesn’t eat breakfast every day,” indicates that the person does not have a daily habit of eating breakfast. This means that there are some days when she does eat breakfast and some days when she doesn’t.
The second sentence, “She doesn’t eat breakfast on any day,” indicates that the person never eats breakfast. This means that there are no days when she eats breakfast, and she has a consistent habit of not eating breakfast.
In both sentences, the use of the word “doesn’t” indicates that the person does not engage in the action described (in this case, eating breakfast). However, the difference between the two sentences lies in the use of the words “every” and “any.” The word “every” indicates that the action happens regularly, but not necessarily on every single occasion. In contrast, the word “any” indicates that the action does not happen at all.
I hope this helps! Let me know if you have any other questions.
Me: That’s very helpful. Which of the two sentences is closest in meaning to “她每天都不吃早饭”?
ChatGPT: In English, the sentence that is closest in meaning to “她每天都不吃早饭” would be “She doesn’t eat breakfast on any day.” This is because the phrase “每天都不” in the Chinese sentence indicates that the person never engages in the action described (in this case, eating breakfast), which is similar to the use of the word “any” in the English sentence.
Me: Thanks. Could you now translate “她每天都不吃早饭” into English for me?
Sure, I’d be happy to help. The Chinese sentence “她每天都不吃早饭” can be translated into English as “She doesn’t eat breakfast on any day.” This sentence indicates that the person never has breakfast, and it is their consistent habit to not eat breakfast.
Me: Thanks. Could you now translate “我每天都不用牙线” into English for me?
ChatGPT: Sure, the Chinese sentence “我每天都不用牙线” can be translated into English as “I don’t use dental floss every day.” This sentence indicates that the person does not have a daily habit of using dental floss, but it does not necessarily mean that they never use it at all.
Me: Is it not like the breakfast sentence? Doesn’t it mean “I don’t use dental floss on any day”?
ChatGPT: It’s a bit different. The sentence “我每天都不用牙线” does not necessarily mean “I don’t use dental floss on any day.” While the word “每天都” in the Chinese sentence does indicate that the action (in this case, using dental floss) does not happen regularly, it does not necessarily mean that it never happens at all. In other words, the person may still use dental floss, but not on a daily basis.
On the other hand, the sentence “I don’t use dental floss on any day” would indicate that the person never uses dental floss, which is a stronger statement than what is expressed in the Chinese sentence.
In general, it’s important to pay attention to the specific words and phrases used in a sentence, as they can change the meaning of the sentence in subtle ways. I hope this helps! Let me know if you have any other questions.
Me: Could you translate “她每天都不吃早饭” into English for me?
ChatGPT: Certainly! “她每天都不吃早饭” can be translated into English as “She doesn’t eat breakfast on any day.” This sentence indicates that the person never has breakfast, and it is their consistent habit to not eat breakfast.
Me: Ok, but I thought the word “每天都” in the Chinese sentence indicates that the action (in this case, it would be eating breakfast) does not happen regularly, but it does not necessarily mean that it never happens at all?
ChatGPT: You are correct! The word “每天都” in the Chinese sentence does indicate that the action (in this case, eating breakfast) does not happen regularly, but it does not necessarily mean that it never happens at all. This means that the person may still eat breakfast sometimes, but not on a daily basis.
However, when translating the sentence into English, we have to consider how the words and phrases used in the Chinese sentence would be understood in the context of English. In this case, the use of the word “any” in the English translation can be understood to mean that the action never happens, which is a reasonable interpretation of the Chinese sentence.
It’s important to keep in mind that translation is not an exact science, and there may be multiple ways to translate a sentence from one language to another. I hope this helps! Let me know if you have any other questions.