e726fb add default robots.txt for all wikis

Authored and Committed by Christian Boltz 6 years ago
    add default robots.txt for all wikis
    
    Indexing /index.php doesn't make much sense - it provides page history
    etc. which isn't really needed in search engines. Besides keeping search
    engines "clean", it also avoids that search engines like YandexImages go
    wild (260589 requests by YandexImages to index.php vs. 6300 requests
    with other user agents on tr.o.o today!)
    
    Our "real content" is available with speaking URLs (without index.php)
    in the URL.
    
        
file modified
+1 -1