[FFmpeg-user] How to extract m3u8 stream URL from webpage with 2 players?

[Bo Berglund]

On Sat, 8 Jan 2022 16:51:07 +0100, Reino Wijnsma <rwijnsma at xs4all.nl> wrote:

>On 2022-01-08T10:25:48+0100, Bo Berglund <bo.berglund at gmail.com> wrote:
>> On Fri, 07 Jan 2022 14:05:02 +0100, Bo Berglund <bo.berglund at gmail.com> wrote:
>>> If I open the source page and use the F12 function key to activate browser debug
>>> mode I can see the stream URL for both players.
>>> Right now they are:
>>> #1 = https://hlss.goodgame.ru/hls/188429.m3u8
>>> #2 = https://1111296894.rsc.cdn77.org/LS-ATL-54548-10/tracks-v1a1/mono.m3u8
>>>
>>> What I would like to have is a way to extract the xxx/mono.m3u8 URL #2
>>> automatically in a script.
>>>
>>> The URL:s seem to change irregularly so I will use any method found to extract
>>> the URL minutes before the download starts by launching the extractor script via
>>> cron.
>> It turns out that shortly after I posted my question the second player stopped
>> working altogether on the webpage, so there is nothing to test anymore...
>> :(
>Both work for me. They're actually one and the same.
>
>$ xidel -s "https://msnbcdailyshows.com/" -e '//iframe' --output-node-format=html
><iframe src="https://goodgame.ru/player?188429" width="800" height="480" frameborder="0" allowfullscreen></iframe>
><iframe src="//2checksitea.blogspot.com/2012/07/099msnbc1.html" width="100%" height="430" frameborder="0" scrolling="no" allowfullscreen></iframe>
>
>The second url refers to the exact same "goodgame.ru"-url:
>
>$ xidel -s "https://msnbcdailyshows.com/" -f '(//iframe)[2]' -e '//div[@class="post-body entry-content"]/iframe' --ignore-namespaces --output-node-format=html
><iframe src="https://goodgame.ru/player?188429" width="600" height="400" frameborder="0" allowfullscreen></iframe>
>
>The HLS-manifest-url:
>
>$ xidel -s "https://msnbcdailyshows.com/" -e 'x"https://hls.goodgame.ru/manifest/{substring-after((//iframe)[1]/@src,"?")}_master.m3u8"'
>#or
>$ xidel -s "https://msnbcdailyshows.com/" -f '(//iframe)[2]' -e 'x"https://hls.goodgame.ru/manifest/{//div[@class="post-body entry-content"]/substring-after(iframe/@src,"?")}_master.m3u8"'
>https://hls.goodgame.ru/manifest/188429_master.m3u8
>
>$ ffmpeg -hide_banner -user_agent "Mozilla Firefox" -i "$(xidel -s "https://msnbcdailyshows.com/" -e 'x"https://hls.goodgame.ru/manifest/{substring-after((//iframe)[1]/@src,"?")}_master.m3u8"')"
>[hls @ 03105100] Skip ('#EXT-X-VERSION:3')
>[hls @ 03105100] Opening 'https://hls.goodgame.ru/hls/188429.m3u8' for reading
>[hls @ 03105100] Skip ('#EXT-X-VERSION:3')
>[hls @ 03105100] Opening 'https://hls.goodgame.ru/hls/188429-279700.ts' for reading
>[hls @ 03105100] Opening 'https://hls.goodgame.ru/hls/188429-279701.ts' for reading
>Input #0, hls, from 'https://hls.goodgame.ru/manifest/188429_master.m3u8':
>  Duration: N/A, start: 436.194944, bitrate: N/A
>  Program 0
>    Metadata:
>      variant_bitrate : 2330000
>  Stream #0:0: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720, 30 tbr, 90k tbn
>    Metadata:
>      variant_bitrate : 2330000
>  Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp
>    Metadata:
>      variant_bitrate : 2330000
>At least one output file must be specified
>
>-user_agent is required!

This tool xidel you use, from where is that?
How can it be installed?


-- 
Bo Berglund
Developer in Sweden