[FFmpeg-devel] [PATCH] ffmpeg-web/robots.txt: attempt to keep spiders out of dynamically generated git content

[Michael Niedermayer]

On Wed, Jul 14, 2021 at 10:40:53PM +0200, Michael Niedermayer wrote:
> On Wed, Jul 14, 2021 at 04:00:53PM -0400, ffmpegandmahanstreamer at lolcow.email wrote:
> > On 2021-07-14 14:51, Michael Niedermayer wrote:
> > > Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> > > ---
> > >  htdocs/robots.txt | 13 ++++++++++++-
> > >  1 file changed, 12 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/htdocs/robots.txt b/htdocs/robots.txt
> > > index eb05362..4bbc395 100644
> > > --- a/htdocs/robots.txt
> > > +++ b/htdocs/robots.txt
> > > @@ -1,2 +1,13 @@
> > >  User-agent: *
> > > -Disallow:
> > > +Crawl-delay: 10
> > > +Disallow: /gitweb/
> > > +Disallow: /*a=search*
> > > +Disallow: /*/search/*
> > > +Disallow: /*a=blobdiff*
> > > +Disallow: /*/blobdiff/*
> > > +Disallow: /*a=commitdiff*
> > > +Disallow: /*/commitdiff/*
> > > +Disallow: /*a=snapshot*
> > > +Disallow: /*/snapshot/*
> > > +Disallow: /*a=blame*
> > > +Disallow: /*/blame/*
> > LGTM based on my own personal experiences. But the robots.txt has to be
> 
> will apply
> 
> 
> > applied for git.ffmpeg.org as well, and not just ffmpeg.org. Or else they
> > will just do the same for git.ffmpeg since there are treated separately.
> 
> was expecting this a bit ...
> i will look into that tomorrow or so unless someone else does before me

done

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

I am the wisest man alive, for I know one thing, and that is that I know
nothing. -- Socrates
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20210715/456873f8/attachment.sig>