perl-WWW-RobotRules - Database of robots.txt-derived permissions
| Website: | https://metacpan.org/release/WWW-RobotRules |
|---|---|
| License: | GPL+ or Artistic |
| Vendor: | Virtuozzo |
- Description:
This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <https://www.robotstxt.org/robotstxt.html>. Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site.
Packages
| perl-WWW-RobotRules-6.02-30.vl9.1.src [19 KiB] |
Changelog
by Mohan Boddu (2021-08-09):
- Rebuilt for IMA sigs, glibc 2.34, aarch64 flags Related: rhbz#1991688 |