[FFmpeg-devel] [PATCH] ipfsgateway: Remove default gateway

[Timo Rothenpieler]

On 11.08.2022 22:18, Michael Niedermayer wrote:
> On Thu, Aug 11, 2022 at 07:56:04PM +0200, Mark Gaiser wrote:
>> On Thu, Aug 11, 2022 at 7:35 PM Timo Rothenpieler <timo at rothenpieler.org>
>> wrote:
>>
>>> On 11.08.2022 19:21, Mark Gaiser wrote:
>>>> On Thu, Aug 11, 2022 at 6:49 PM Timo Rothenpieler <timo at rothenpieler.org
>>>>
>>>> wrote:
>>>>
>>>>> On 11.08.2022 18:26, Mark Gaiser wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> On the IPFS side we do have a solution for that with CAR files, you can
>>>>>> read more about that here [1].
>>>>>> Within the scope of this ipfs gateway protocol handler there isn't a
>>>>>> solution yet to use CAR files, it is on our radar but still in the
>>>>>> discussion phase.
>>>>>>
>>>>>> On the cURL side we had this same discussion with 2 possible solutions
>>>>> [2].
>>>>>> For completeness, i'll list them here in full too:
>>>>>>
>>>>>> 1. An error message that gives no example but instead points the user
>>> to
>>>>>> documentation on how to get it working.
>>>>>> === cURL example
>>>>>> $ curl
>>> ipfs://bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am
>>>>>> Error: local gateway not found and/or IPFS_GATEWAY is not set
>>>>>> Learn how to run one: https://docs.ipfs.tech/install/command-line/
>>>>>> ===
>>>>>>
>>>>>> 2. An error message that makes the user aware of IPFS and provides a
>>>>>> solution to get it working immediately.
>>>>>> === cURL example
>>>>>> $ curl
>>> ipfs://bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am
>>>>>> Error: local gateway not found and/or IPFS_GATEWAY is not set.
>>>>>> Try: IPFS_GATEWAY=https://ipfs.io
>>>>>> or run your own: https://docs.ipfs.tech/install/command-line/
>>>>>> ===
>>>>>>
>>>>>> Within the cURL implementation we're going for point 1.
>>>>>> The same idea can very well apply to ffmpeg too. Different texts that
>>>>> match
>>>>>> the different context, but in the same spirit.
>>>>>>
>>>>>> Now ffmpeg is a bit different here. First and foremost because it
>>>>> predates
>>>>>> the curl.
>>>>>> But also because the default fallback gateway was an explicitly
>>> requested
>>>>>> feature from the ffmpeg side to give an "it always works" feeling.
>>>>>> ffmpeg therefore has a fourth option: Do nothing and keep it as-is.
>>>>>
>>>>> I'm not sure who requested that, but I doubt "tunnel all user traffic
>>>>> through some random third parties server" was the idea there.
>>>>>
>>>>
>>>> Here's the conversation requesting this very feature:
>>>> https://ffmpeg.org/pipermail/ffmpeg-devel/2022-March/293835.html
>>>
>>> I generally agree with the points brought up there.
>>> But my conclusion very much is not "just put a somewhat random default
>>> into the code".
>>> Even a list of defaults is not Okay.
>>> We can't hardcode "magic servers".
> 
> I think we really should be looking at first principles here, and not
> say what to do and what not to do in isolation.
> Especially as some mails in this thread are a bit more emotional than
> what ive seen normally.
> That said the concern is very real and valid.
> 
> So lets see why things where done as they are first.
> * we added IPFS support to (obviously supprt IPFS which is a increasingly
>    relevant thing)
> * a full self contained IPFS implementation was not available and may or
>    may not be practical (this should be revisited and reconsidered with people
>    knowing the protocol well)
> * The first goal is if possible support it out of the box and on all platforms
> * Asking the user to setup a IPFS gateway or even to point to one per ENV variable
>    appeared not really possible on locked down platforms like phones (maybe there
>    is a way that was missed ?)
> * so That left the choice to either add a default or to drop IPFS support for
>    some platforms.
> * The patch was on the ML for a long time and noone objected to the simple
>    default
>    
> Now what is the problem with a single hardcoded default ?
> (please correct me if iam missing something)
> 1 It can log you
> 2 It can man in the middle you
> 3 It can stop working
> 
> If we tell the user to find their own gateway this does not actually protect
> them from these, it rather makes it "their problem" not ours.
> Also a user setting up a IPFS_GATEWAY pointer will not maintain its security
> a year later, 5 years later that will still be there and that will be a big
> security issue too if a random choice is a big security issue.
> So as much as a hardcoded default is bad, this is also bad
> 
> a full IPFS implementation (if this is possible which iam not sure about)
> may be a solution. Running a local IPFS node which receives security updates
> should work too. Again i suspect the later may be hard on locked down devices
> like phones. (again someone who knows this should comment here)
> 
> So which options are there now
> * full IPFS implementation (gold standard but maybe impossible)
> * user setup IPFS node (probable not possible on some platforms)
> * "its the users problem" (manually maintaining a link to a secure
>    gateway sounds insecure to me with average users)
> * Maintain a list of believed to be secure gateways outside the source
>    maybe on https: git.ffmpeg.org. (this was not discussed previously)
>    the code could if no local node/gateway and no IPFS_GATEWAY environment
>    fetch a random entry from that gateway list and print info to the
>    user notifying of the use of the default
> 
> It is quite possible iam missing something but this last option seems
> an improvment over a single default. Also it seems more secure to me
> to the average user than setting a IPFS_GATEWAY and then forgeting
> that it was set for years.
> 
> We could also limit such a external fetched (updatable) list to
> platforms where all other options are impossible
> I dont know if thats a good idea or not, iam just throwing that out here
> 

I'm aware that it's harsh, but with such limitations to run the protocol 
securely, it probably shouldn't have been merged in the first place.

Any kind of built in hardcoded server is not acceptable imo.
Even with it pointing to our own infrastructure, we can't really 
guarantee its availability, specially should the protocol gain traction 
and heavy use.

I'm not sure what the correct way forward is.
But the proposed patch here still seems like the best option to me.

>>
>> This is just your - valued! -  opinion, but still just 1. I insist on
>> waiting to hear from Michael to hear a decision on this, mainly because he
>> was quite persistent in asking for this feature to begin with.
> 
> Iam quite happy to leave this discussion to others, last time it was
> just that noone seemed to care over a really long time to comment
> now it seems everyone really cares.
> I think its very good that people are thinking about it now, it is a
> rather annoying situation as each option is a tradeoff which sucks in
> some form
> Maybe the ultimate best would be a change at the IPFS protocol level
> so that lean light clients could securely use the protocol easily


The patch wasn't on my radar at all. I had assumed it was actually 
implementing IPFS in some fashion.
Not via an entire external http gateway. I'm a bit confused that it's 
its whole own protocol.