[FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

Reino Wijnsma rwijnsma at xs4all.nl
Sat Apr 16 16:04:02 EEST 2022 

On 2022-04-13T01:42:39+0200, Reino Wijnsma <rwijnsma at xs4all.nl> wrote:
> ...and I don't understand how something like "http://188.17.143.39/hls/CH_R04_DOMASHNY/variant.m3u8?version=2" would translate to...
>
> On 2022-04-12T17:54:32+0200, Bo Berglund <bo.berglund at gmail.com> wrote:
>> https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/hdntl=exp=1649865178~acl=%2f*~data=hdntl~hmac=5480897632ddf98d23bf870098333af5055752c54d28764b8c894711086d1266/VIDEO_1_4596000.m3u8

Well,...

On 2022-04-13T23:49:26+0200, Reino Wijnsma <rwijnsma at xs4all.nl> wrote:
> Alternatively I could have a look on a much newer system of one of my family members

...I've found out that "http://188.17.143.39/hls/CH_R04_DOMASHNY/variant.m3u8?version=2" (from the iframe's HTML-source) has nothing to do with the akamaized-url. Very strange.

In the network-traffic I've found 2 different JSON api endpoints that house the bare m3u8-url:

$ xidel -s "https://api-leap.nbcsports.com/feeds/assets/2007524?application=NBCNews&format=nbc-player&platform=desktop" \
  -e '$json/(videoSources)()/cdnSources/(primary)()/sourceUrl'
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8

$ xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json" \
  -e '$json/(videoSources)()/sourceUrl' \
  -e '$json/(videoSources)()/cdnSources/(primary)()/url'
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8

I have no idea where the identifier 2007524 is coming from though! It's not in the HTML-source of "http://www.freeintertv.com/view/id-2308" or its iframe. I couldn't find it anywhere else. It must be the program-id of the NBC livestream.

This HLS manifest doesn't work just yet:

$ xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json" \
  -f '{"url":$json/(videoSources)()/sourceUrl,"error-handling":"xxx=accept"}' \
  -e '$headers[1]'
HTTP/1.1 403 Forbidden

The m3u8-url needs to be sent to "https://tokens.playmakerservices.com" in a POST-request:

$ xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json" -e '$json/(videoSources)()/{"requestorId":"nbcnews","pid":pid,"application":"NBCSports","version":"v1","platform":"desktop","token":"","resourceId":"","inPath":"false","authenticationType":"unauth","cdn":"akamai","url":sourceUrl}'
{
  "requestorId": "nbcnews",
  "pid": "2007524",
  "application": "NBCSports",
  "version": "v1",
  "platform": "desktop",
  "token": "",
  "resourceId": "",
  "inPath": "false",
  "authenticationType": "unauth",
  "cdn": "akamai",
  "url": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8"
}

$ xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json" -e 'serialize($json/(videoSources)()/{"requestorId":"nbcnews","pid":pid,"application":"NBCSports","version":"v1","platform":"desktop","token":"","resourceId":"","inPath":"false","authenticationType":"unauth","cdn":"akamai","url":sourceUrl},{"method":"json"})'
{"requestorId":"nbcnews","pid":"2007524","application":"NBCSports","version":"v1","platform":"desktop","token":"","resourceId":"","inPath":"false","authenticationType":"unauth","cdn":"akamai","url":"https:\/\/nbcnews3.akamaized.net\/hls\/live\/723429\/NBCNewsNowa418c506\/clear\/master.m3u8"}

$ xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json" -e 'x:request({"post":serialize($json/(videoSources)()/{"requestorId":"nbcnews","pid":pid,"application":"NBCSports","version":"v1","platform":"desktop","token":"","resourceId":"","inPath":"false","authenticationType":"unauth","cdn":"akamai","url":sourceUrl},{"method":"json"}),"url":"https://tokens.playmakerservices.com"})/json'
{
  "akamai": [
    {
      "token": "hdnts=st=1650111897~exp=1650112017~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=bcab5244534ea41b8b5c585c67805765f582cc5386ab89665833c9e8651c16d0",
      "tokenizedUrl": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1650111897~exp=1650112017~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=bcab5244534ea41b8b5c585c67805765f582cc5386ab89665833c9e8651c16d0",
      "sourceUrl": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8"
    }
  ]
}

$ ffmpeg -hide_banner -i "$(xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json" -e 'x:request({"post":serialize($json/(videoSources)()/{"requestorId":"nbcnews","pid":pid,"application":"NBCSports","version":"v1","platform":"desktop","token":"","resourceId":"","inPath":"false","authenticationType":"unauth","cdn":"akamai","url":sourceUrl},{"method":"json"}),"url":"https://tokens.playmakerservices.com"})/json//tokenizedUrl')"
[...]
Input #0, hls, from 'https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1650112296~exp=1650112416~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=4f01d8d10fecf43c9419c9bd6923d64bc4c8df1885d30ba038dbd4205f555ac4':
  Duration: N/A, start: 81275.964722, bitrate: N/A
  Program 0
    Metadata:
      variant_bitrate : 8238624
  Stream #0:0: Video: h264 ([27][0][0][0] / 0x001B), none, 90k tbr, 90k tbn
    Metadata:
      variant_bitrate : 8238624
  Stream #0:1: Audio: aac ([15][0][0][0] / 0x000F), 0 channels, fltp
    Metadata:
      variant_bitrate : 8238624
  Program 1
    Metadata:
      variant_bitrate : 8238624
  Stream #0:2: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1920x1080, Closed Captions, 29.97 fps, 29.97 tbr, 90k tbn
    Metadata:
      variant_bitrate : 8238624
  Stream #0:3: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 8238624
[...]

Prettified query to extract the m3u8-url:

$ xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json" -e '
  x:request({
    "post":serialize(
      $json/(videoSources)()/{
        "requestorId":"nbcnews",
        "pid":pid,
        "application":"NBCSports",
        "version":"v1",
        "platform":"desktop",
        "token":"",
        "resourceId":"",
        "inPath":"false",
        "authenticationType":"unauth",
        "cdn":"akamai",
        "url":sourceUrl
      },
      {"method":"json"}
    ),
    "url":"https://tokens.playmakerservices.com"
  })/json//tokenizedUrl
'

On 2022-04-14T10:26:56+0200, Bo Berglund <bo.berglund at gmail.com> wrote:
> Could it be that it stays working as long as someone (me) keeps using it at
> regular intervals?

I doubt it.
With the help of my hobby-project's m3u8-to-json() function <https://github.com/Reino17/xivid/blob/master/xivid.xqm#L34-L106>...

$ xidel -s --module=xivid.xqm "https://stream.nbcnews.com/data/live_sources_2007524.json" -e 'x:request({"post":serialize($json/(videoSources)()/{"requestorId":"nbcnews","pid":pid,"application":"NBCSports","version":"v1","platform":"desktop","token":"","resourceId":"","inPath":"false","authenticationType":"unauth","cdn":"akamai","url":sourceUrl},{"method":"json"}),"url":"https://tokens.playmakerservices.com"})/xivid:m3u8-to-json(json//tokenizedUrl)'
[
  {
    "id": "hls-0",
    "format": "m3u8[manifest]",
    "url": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1650112928~exp=1650113048~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=be7299794a2ff43e94c14cce1729f28193715b47ce5b57016cee51a3ddd4238d"
  },
  [...]
  {
    "id": "hls-15",
    "format": "m3u8[h264+aac]",
    "resolution": "1920x1080 at 29.97fps",
    "bitrate": "8239kbps",
    "url": "https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/hdntl=exp=1650199328~acl=%2f*~data=hdntl~hmac=f363e78953e6afddd2e3f75613c2a2aa5cc9e7dbca846aab9f9167f5501dca3e/VIDEO_0_6596000.m3u8"
  },
  {
    "id": "hls-16",
    "format": "m3u8[h264+aac]",
    "resolution": "1920x1080 at 29.97fps",
    "bitrate": "8239kbps",
    "url": "https://nbcnews3.akamaized.net/hls/live/723429-b/NBCNewsNowa418c506/clear/hdntl=exp=1650199328~acl=%2f*~data=hdntl~hmac=f363e78953e6afddd2e3f75613c2a2aa5cc9e7dbca846aab9f9167f5501dca3e/VIDEO_0_6596000.m3u8"
  }
]

...you can see that the m3u8-url for the HLS manifest is only valid for 2 minutes (1650113048 - 1650112928 = 120).
The m3u8-url for the individual videostreams on the other hand is valid for 24 hours (1650113048 - 1650199328 = 86400 / 60 / 60 = 24).

-- 
Reino

More information about the ffmpeg-user mailing list