[FFmpeg-devel] [PATCH] ffmpeg-web/robots.txt: attempt to keep spiders out of dynamically generated git content

[Michael Niedermayer]

On Wed, Jul 14, 2021 at 08:51:21PM +0200, Michael Niedermayer wrote:
> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> ---
>  htdocs/robots.txt | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/htdocs/robots.txt b/htdocs/robots.txt
> index eb05362..4bbc395 100644
> --- a/htdocs/robots.txt
> +++ b/htdocs/robots.txt
> @@ -1,2 +1,13 @@
>  User-agent: *
> -Disallow:
> +Crawl-delay: 10
> +Disallow: /gitweb/
> +Disallow: /*a=search*
> +Disallow: /*/search/*
> +Disallow: /*a=blobdiff*
> +Disallow: /*/blobdiff/*
> +Disallow: /*a=commitdiff*
> +Disallow: /*/commitdiff/*
> +Disallow: /*a=snapshot*
> +Disallow: /*/snapshot/*
> +Disallow: /*a=blame*
> +Disallow: /*/blame/*

This is based on https://serverfault.com/questions/506613/ideal-robots-txt-for-a-gitweb-installation
i will add this link to robots.txt


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Avoid a single point of failure, be that a person or equipment.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20210714/6be472a3/attachment.sig>