GPT-4 trained on YouTube transcripts

Based on the information provided in the search results, OpenAI reportedly used transcriptions of over a million hours of YouTube videos to train GPT-4, its most advanced large language model. This was part of their effort to gather high-quality training data, which is crucial for the development and improvement of AI models like GPT-4. The company developed its Whisper audio transcription model to assist in this process, which allowed them to transcribe the YouTube content[17].

The use of YouTube videos for training data was considered legally questionable by OpenAI, but they believed it to be fair use. OpenAI president Greg Brockman was personally involved in collecting videos that were used for this purpose. The company's spokesperson, Lindsay Held, stated that OpenAI curates unique datasets for each of its models to help their understanding of the world and uses numerous sources, including publicly available data and partnerships for non-public data[17].

Google, which owns YouTube, has 'robots.txt files and Terms of Service that prohibit unauthorized scraping or downloading of YouTube content. Google spokesperson Matt Bryant mentioned that the company takes technical and legal measures to prevent such unauthorized use when they have a clear legal or policy basis to do so[17].

The search results indicate that the training of GPT-4 on YouTube transcripts was part of a broader strategy by AI companies to overcome the challenge of finding sufficient and diverse data to train their models effectively. This strategy also included using data from other sources such as Github, chess move databases, and schoolwork content from Quizlet[17].

Citations:
[1] https://www.psychotactics.com/flawless-transcript/
[2] https://dzone.com/articles/the-top-6-use-cases-of-gpt-4
[3] https://siliconangle.com/2023/07/20/research-paper-suggests-openais-gpt-4-may-become-less-accurate/
[4] https://www.businessinsider.com/openai-gpt4-ai-model-got-lazier-dumber-chatgpt-2023-7
[5] https://community.openai.com/t/how-can-i-efficiently-train-a-gpt-on-hundreds-of-thousands-words-of-video-transcripts/595943
[6] https://litslink.com/blog/app-ideas-for-gpt-4-turbo-with-vision
[7] https://www.reddit.com/r/ChatGPT/comments/17o8qei/are_gpt_4_answers_better_and_more_accurate_than/
[8] https://arxiv.org/html/2401.06920v1
[9] https://www.reddit.com/r/voxscript/comments/1443i11/understanding_voxscripts_approach_to_large/
[10] https://openai.com/gpt-4
[11] https://www.zdnet.com/article/gpt-4-is-getting-significantly-dumber-over-time-according-to-a-study/
[12] https://openai.com/research/gpt-4
[13] https://www.youtube.com/watch?v=QbExALqJ8CY
[14] https://github.com/dk-liang/Awesome-GPT4-with-Applications
[15] https://pubs.rsna.org/doi/10.1148/radiol.232411
[16] https://www.nature.com/articles/s41598-024-55568-7
[17] https://www.theverge.com/2024/4/6/24122915/openai-youtube-transcripts-gpt-4-training-data-google
[18] https://www.digitaltrends.com/computing/best-apps-to-use-gpt-4/
[19] https://www.plusdocs.com/blog/the-best-gpt-4-apps
[20] https://lablab.ai/apps/tech/openai/gpt4