Why get the html ?

[LoyalPotato]

Hello,
As I've been researching ways of getting youtube captions I've come across several ways of getting them.
One of them was through https://youtube.com/get_video_info?video_id=videoId.
So I was wondering what is the advantage (if there is any) to using the endpoint of the video itself and getting the information in the html over using this endpoint.

[jdepoix]

Hi @LoyalPotato, the answer is quite simple: I did not know of this endpoint! 😄
Maybe it was not available when I initially implemented this module or I simply didn't find out about it. How did you find it and what are the other options that you mentioned? Maybe there are more options I don't know about. The main problem with all the options I was able to find before creating this module was that they weren't able to provide me with automatically generated subtitles (which was the main reason I created this module). But looks like your endpoint returns the exact JSON which is currently being extracted from the HTML.

I agree with you, this definitely seems like a more reliable way than getting the HTML. However, I will have to investigate further how this behaves in certain scenarios:

• What does this endpoint return when rate limits are exceeded?
• What does it return when video/transcripts are not available?
• Can we retrieve transcripts for age restricted videos, or do we still need cookies for that?

There are a few edge cases I added to this module as I found out about them in the last years. Changing such a central component would require checking if those edge cases are still relevant and what they look like on this endpoint.

Nonetheless, this is a great find and I will definitely look into this and consider integrating it! 😊 👍

[LoyalPotato]

I've seen it now in some places (stackoverflow like this & this for example) but here seems to have it somewhat more organized.
I have been trying to organize all the ways that I can obtain the captions (user submitted ones, auto-generated and only through reading your module was I able to figure out how to get translated ones).

Here's what I have written down (other than the one I previously mentioned):

"http://video.google.com/timedtext?type=list&v={videoId}" // You can use this to view a list of all the user 
// submitted captions available

"http://video.google.com/timedtext?type=track&v={videoId}&lang={langCode}" // With this you can get the same 
// XML structure used in other endpoints 

// Here's an example link

"http://video.google.com/timedtext?type=list&v=zenMEj0cAC4"
// and the track
"http://video.google.com/timedtext?type=track&v=zenMEj0cAC4&lang=pl"

The other one I have jotted down is just through the regular documented youtube api, but without auth, and with an api key, I can only get the listing to know which are available.

As for the scenarios that you have noted I haven't tested those cases yet.

[jdepoix]

Yeah, the timedtext endpoint has been around for a while, that's the one which doesn't allow you to get ASR transcripts.

[danielcliu]

@LoyalPotato I think there maybe problems with grabbing the Generated Transcripts via that method. I looked into that like years ago though and I could be wrong

[jdepoix]

Yeah, that was my first thought as well. Back in the day when I initially created this I couldn't find a way to get automatically created transcripts, which is why I created this module. Therefore, this was first thing I checked and it actually works on generated transcripts as well! In fact, this contains the exact JSON which is currently being extracted from the HTML to get the timedtext URLs, which is pretty neat!

[jdepoix]

For some reason the data is encoded like URL params. You can check out what's in there by running:

import requests
import json
from urllib.parse import urlparse

response = requests.get('https://youtube.com/get_video_info?video_id=Fx_N4GlwdEM')
data = parse_qs(response.content.decode())
print(data)

json_data = json.loads(data['player_response'][0])
print(json_data)

[jdepoix]

There also is data['status'] and data['errorcode'] (when you're trying to retrieve an invalid video id) which could improve error handling.

[LoyalPotato]

Just writing an update on the endpoint that I mentioned get_video_info
It appears that it appears that it has been disabled, or at least made harder to use ( I haven't found a solution as of today for continuing to use that endpoint ).
So what I changed to was another endpoint that I see mentioned as InnerTube https://www.youtube.com/youtubei/v1/player?videoId={videoId}&key={api_key}&contentCheckOk=true&racyCheckOk=true ( ApiKey = AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 ).
Some more info about this endpoint and how to use I found in this package: pytube.

[jdepoix]

Thanks for the update! I am currently quite busy, so I don't see myself implementing this in the coming months (at least as long as the current approach is working). However, it is good to know alternative approaches, in case YouTube introduces some changes which prevent the current approach from working. So any kind of information on different approaches still is highly appreciated, even though I am currently not able to implement them! 👍