Stay On the Cutting Edge: Get the Tom's Hardware Newsletter
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
You are now subscribed
Your newsletter sign-up was successful
Join the club
Get full access to premium articles, exclusive features and a growing list of member rewards.
An account already exists for this email address, please log in.
ChatGPT transcripts are reportedly read by humans to improve responses, including those with personal information — 'Project Lilly' has seen OpenAI hire hundreds of contractors to manually review logs
AI companies don't have a great track record in areas like copyright or user privacy — unless they're the ones on the short end of the stick, that is — but it's generally known that the chat logs from platforms like ChatGPT are used for improving models. The mechanism as to how this happens was still a mystery until today. 404 Media just published a report about OpenAI's process of human review for chat transcripts, explaining how the review process works, and how it involves other humans sometimes reading private information.
The rating project's name at OpenAI is Project Lily. The publication got information on the project's instruction guides, Slack channels, real ChatGPT conversations, and, of course, the rating system to classify conversations. The operators are called "prompt reviewers," and their job is fairly simple: look at anonymized real-world chats, and judge the quality of ChatGPT's responses to assess whether they actually answer the question, and that the text doesn't overuse "AI-speak," patronizing tones, emojis, or sycophancy, among other parameters. Anthropomorphizing and stating "personal" experiences are both off the table, meaning that while it's OK for ChatGPT to say "I found some information," it's not OK for it to say "as a chef, I like to..." or "I know what that's like."
The work is "very rote," according to a reviewer, but at reportedly over $50 an hour, it's a high rate for what looks like reasonably simple work. The reviewer also said that their guidelines keep changing and are often self-contradictory, a feeling most software developers should easily identify with.
The person doesn't think that most users are aware their chats are being read by others, though, something that's particularly troubling when many use ChatGPT as an impromptu friend or therapist and put deep secrets in words for the bot to read.
While the chats allegedly go through an anonymization pass and reviewers don't see usernames, OpenAI admitted to 404 Media that the filtering may let some personal data through, especially in shorter chats. The site notes that in many conversations, the user asks ChatGPT to keep the contents secret, as well. The version of the chat handed to reviewers also reportedly includes a "user memories summary," containing a summary of the users' questions and interests, context, and potentially even location.
Crucially, Project Lily does not grade the chats' actual factual accuracy other than flagging obvious mistakes, implying that there's likely at least one more team (or several) doing separate evaluations. Likewise, this reviewing is separate from manual safety checks that ascertain if someone might be looking to hurt someone else (or, presumably, themselves).
The existence of the project also indicates that contrary to these image AI companies try to cultivate, the models don't improve just with technological advancement and better training sets — it appears you still need more than a few competent humans in the mix.
Stay On the Cutting Edge: Get the Tom's Hardware Newsletter
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
By now you may be wondering about the "allow us to use your chats to improve our product" (paraphrased) setting present in most consumer-facing chat bots. That setting is turned on by default in every bot we can think of, even with many paid plans. In ChatGPT's case, it does default to off in Enterprise, Business, and Educational customers.
That toggle switch does not work retroactively, though, so any chats already in ChatGPT's database will remain there unless the user requests deletion. Also, said deletion is also not retroactive, meaning that deleted chats may have already been hoovered and anonymized, and possibly reside in a dataset somewhere.
Although OpenAI initially had no answer to 404 Media's inquiry on whether users were explicitly informed that their chats could be read by humans, the company eventually offered a link to one of its FAQ pages that discusses human review for the purpose of model improvement. We verified ourselves that said notice is at least two years old, and likely older. After the publication of the exposé, the firm changed its help page explaining how people can opt out of data collection, but there's no mention of human operators in that text.
This type of data collection and review is a running theme across most providers. Google Gemini clearly states that "humans may review some saved chats" in its Privacy Hub. Anthropic's stance is similar, with a page dedicated to this topic. Perplexity's stance, meanwhile, is unclear, as its Privacy Notice doesn't confirm or deny human access to chat logs.
Bruno Ferreira is a contributing writer for Tom's Hardware. He has decades of experience with PC hardware and assorted sundries, alongside a career as a developer. He's obsessed with detail and has a tendency to ramble on the topics he loves. When not doing that, he's usually playing games, or at live music shows and festivals.
If you work in software development you expect this, sort of a no-brainer that you'd analyze weak points to improve them and - at this point - that would mean a human reading the messages.
LOL, we were just using chatGPT to discuss an election. An incumbent is not rerunning this year, chatGPT said they are "term limited", they are not term limited, there is no limit for the position in question. We literally spend the next 30mins, 2 of us, trying to get chatGPT to tell us the truth and it couldnt, every prompt was something about
If you work in software development you expect this, sort of a no-brainer that you'd analyze weak points to improve them and - at this point - that would mean a human reading the messages.
Even more importantly, who would believe their use of a public cloud service like this would be private, especially when they tell you right in their terms of service that they do this. This is "news" only to the clickbait-driven perpetually offended.
The data generated by skilled engineers and professionals using inference as a service is so valuable all the major AI companies provide free chatbots. There is no need for industrial espionage when a company will tell you their trade secrets voluntarily.
On the other hand, the personal information people tell their AI girlfriends and AI medical consultants is much more difficult to legally monetise.
I kind of always assumed this was the case, but now that its confirmed, I fully intend on sending chatgpt pictures of my 'bean bags so someone has to go through that chat 🤣
ChatGPT logs are not the real problem. The problem, and you'd better pay attention if go to see a doctor, is the following:
The last few months I have been to at least 3 doctors whose patients' paperwork (those countless pages about billing, obligations for payments, Emergency Contact, "Patients' rights, Privacy and HIPPA" laws etc, now suddenly also include a few nuggets that essentially say: Your conversation with the doctor will be be recorded by an AI system"
And if you forget to opt out or just skip over all those pages because you saw them a 1000 times before, your medical conversation will most certainly land at our AI overlords.
Now, you think your doctor's office (these days most likely part of a very large corporation owned by a Wall Street hedge fund)....is not going to sell these medical conversation (which is allegedly protected by HIPPA laws) to the AI parent company, then you really are fooling yourself, bi time
PS: Even if you opt out and if you don't ask to make sure it's not recording, some doctors "forget" to turn it off....
PPS: This is my own experience, I live in SWFL in an affluent town, full of retirees with tons of doctors with fancy practices that cater to them. Your neck of the woods may not have doctors who have similar equipment. Yet.
....And if you forget to opt out or just skip over all those pages...your medical conversation will most certainly land at our AI overlords.
Rather dramatic, don't you think?
PEnns said:
... you think your doctor's office (these days most likely part of a very large corporation owned by a Wall Street hedge fund)....is not going to sell these medical conversation (which is allegedly protected by HIPPA laws) to the AI parent company, then you really are fooling yourself, bi time
Have any evidence whatsoever of a conspiracy by the entire medical profession to violate federal law?
I shouldn't have to point this out, but I'll note that -- AI transcription or not -- your doctor's office already has all your medical records. And, to further debunk this, I'll also add that, as of last year, only about 30% of medical practices are owned by any corporations at all, much less "very large Wall Street Hedge Funds".
